On: Ask HN: I want to scrape a web page...

[chrisa]

  require 'net/http'
  require 'uri'
  output = Net::HTTP.get(URI.parse("http://www.google.com"))

[seanmccann → chrisa]

curb is a lot faster than net/http

require 'curb' c = Curl::Easy.perform("http://www.google.com) puts c.body_str

[bdclimber14 → seanmccann]

I wanted to thank you for your help. I wrote a script to scrape over 30,000 courses and textbooks from Arizona State University successfully in a couple days.

[bdclimber14 → seanmccann]

So apparently accessing a secure URL isn't straightforward... No login is required, its just an HTTPS but it won't go in curb.

[bdclimber14 → bdclimber14]

Wasn't an SSL problem but a cookie problem...

[nc → seanmccann]

Premature optimization much?