

Why you should use curl and firebug to scrape - nachivpn
http://nachivpn.blogspot.com/2015/05/why-you-should-use-curl-firebug-to.html

======
marvel_boy
Newbie here. cURL is a command line tool, firebug is a browser tool. So how
can I use the two tools at the same time?

~~~
nachivpn
HTML of a page is received in response to a HTTP request. A page load might
involve multiple HTTP requests. Often parts of a web page that interest you
might be loaded specifically by a request. By using firebug, you could examine
which HTTP request fetched you the content you need and what parameters
(headers, cookies, etc) where associated with it. Using this information, you
could simulate a HTTP request of your own using cURL (in fact, firebug
provides a "copy as cURL command" feature which can be very useful). Firebug
is used only in the initial stages to understand. Post which, cURL can be used
in your shell script (or a suitable language of choice) to implement your bot.

