

Python Requests: HTTP for Humans - frzn
http://python-requests.org/
Requests is an ISC Licensed HTTP library, written in Python, for human beings.<p>Requests allow you to send HEAD, GET, POST, PUT, PATCH, and DELETE HTTP requests. You can add headers, form data, multipart files, and parameters with simple Python dictionaries, and access the response data in the same way. It’s powered by urllib2, but it does all the hard work and crazy hacks for you.
======
old-gregg
Absolutely my favourite recent development in the Python world. I won't be
mistaken if I say that many of us have built half-arse versions of this on top
of urllib(2) multiple times in the past. :-)

The support for keep-alive is particularly welcome, huge kudos to the author.

Question to users of other languages: do equivalents exist for PHP, Perl, C#
or Java? Ruby folks rave about restclient, but is there anything else?

~~~
davnola
For Ruby there's httparty, <https://github.com/jnunemaker/httparty>, curb, and
a couple of others that escape me right now.

~~~
bascule
Here's the new one I'm working on:

<https://github.com/tarcieri/http>

httparty does a few of the same tricks, but I still find its API a bit
obtrusive. At least it automatically parses JSON! (my library does this too,
if an appropriate library is loaded)

In addition to a humane interface, it will soon be backed by a Ragel-generated
HTTP parser:

<https://github.com/tarcieri/http/tree/master/parser>

~~~
samstokes
Looks very promising. I see it currently uses Net::HTTP under the hood - do
you plan to keep it that way, like Httparty and Faraday, or are you going to
do away with Net::HTTP altogether? (The comment saying "this is temporary"
next to requiring 'net/https' is encouraging... :))

~~~
bascule
I plan on eliminating Net::HTTP entirely and writing my own client using Carl
Lerche's parser for his Picard library which is written in Ragel:

<https://github.com/tarcieri/http/tree/master/parser>

------
DanielRibeiro
Discussion from the two month old submission:
<http://news.ycombinator.com/item?id=2882301>

------
megaman821
This looks so nice. Is there any upcoming support for HTTP caching? I know it
is not the easiest thing to do because you will probably need to make several
cache backends (file, db, memcached, redis, etc.) to keep everyone happy.

Also I couldn't tell if there is keep-alive support, is there?

~~~
kenneth_reitz
Keep-alive is 100% supported in the upcoming v0.7.0 release, along with async
i/o :)

Caching: I think that will be best be served by a hook/helper module, but
we'll see.

~~~
samps
Cool! Is there anything I can look at to see how async I/O will work? Does all
the async support depend on gevent monkey patching (as requests/async.py would
suggest)?

~~~
kenneth_reitz
It's unreleased, and therefore completely undocumented at the moment. But
here's the API: <https://github.com/kennethreitz/requests/issues/98>

~~~
st3fan
Mmm async.

Does that also mean that it will have better support for timeouts? I'm not
interested in socket timeouts, more in a 'finish this in N seconds or just
give up' thing.

For that reason I am using pycurl now for some stuff where I had previously
used the requests library. Which has a really really horrible API :-/

------
vidar
What is the timeline for v0.7? Desperately need the asynchronous stuff!

~~~
kenneth_reitz
As soon as I can push it out. Aiming for Friday (if not sooner)!

~~~
vidar
That is great news! I really appreciate your work, thanks!

------
bascule
I just started working on a similar library for Ruby:

<https://github.com/tarcieri/http>

------
mp3tricord
Looks like Python got something similar to Perl's LWP
<https://metacpan.org/module/LWP>

~~~
draegtun
I think LWP is more equivalent to Python's urllib2 (ie. both big beasts!). So
this i believe is closer to what LWP::UserAgent
(<https://metacpan.org/module/LWP::UserAgent>) does.

Also check out Mojo::UserAgent (<https://metacpan.org/module/Mojo::UserAgent>)
for a nice(er) alternative. And don't forget that Perl now comes a nice
lightweight alternative as standard: <https://metacpan.org/module/HTTP::Tiny>

~~~
mp3tricord
Yes gp, LWP::UserAgent specifically. Never knew about HTTP::Tiny thanks.. I'll
check it out.

------
efsavage
I don't mean to badmouth Requests or Python in general, it certainly looks
like an improvement, but can anyone enlighten me as to why an API that wraps
an HTTP request with concise methods is #1 on HN, twice? This looks more like
a good answer to SO question than "news".

This looks like all the other wrapper methods you spend 5 minutes doing, once,
when you start a new codebase?

~~~
lukeschlather
The philosophy of Python is that those wrapper methods shouldn't be necessary.

I'm glad when language quirks and libraries get linked, it's good for
discussion. (And more informative than a good third of the stuff that gets
linked.)

~~~
drgath
I came here with the same question. Considering I come from PHP & JavaScript
where the native APIs are awful, wrapping is a necessary step for sanity.
Interesting that Python didn't have this until now.

~~~
lukeschlather
Python has always had it to some degree, but merging these sorts of things
into the core language is what major releases are for. No language is perfect.
I think the reason you see this upvoted so much is there are so many Python
programmers who want to see this on the roadmap for future versions of Python
(and it probably will be.)

------
pchivers
Link to previous discussion on HN:

<http://news.ycombinator.com/item?id=2882301>

------
d0vs
Unfortunately, no Python 3 support

~~~
sneak
Why is that unfortunate? Why would anyone run python > 2.7?

~~~
briancurtin
Do you have a good reason why they shouldn't?

Apparently 113,798 people found a reason to download the 3.2.2 Windows
installers (32 and 64 bit) from python.org through only the first 10 days of
the month of October. <http://www.python.org/webstats/usage_201110.html>

(doesn't include mention of the 3.2 source tarball downloads, which are under
5,000, most likely since people get that from their distro anyway)

------
jacoblyles
Finally, an easy way to make POST requests from Python

------
artlogic
Forgive me if I've missed this, but does this library have support for
multipart/form-data POSTing? That's a capability in the python library that's
sorely missing - especially in 2.X.

~~~
d0mine
yes, it does

------
leonderijke
I'm using it in a project I'm working on. It's really easy to use!

------
objectified
Although Requests looks very nice, I've wondered why most people wouldn't
simply use pycurl - <http://pycurl.sourceforge.net/> ? It's stable and you get
all the power that cURL has (which is a lot), and the API is fairly simple -
although maybe not as pythonic as the Requests API is.

On a side note: last time I checked urllib2 (which actually drives Requests)
could not do SSL requests over a proxy, which gave me the feeling that this
implementation still isn't very mature.

~~~
kenneth_reitz
I'm moving off of urllib2 as we speak, to sit directly on top of httplib.

One thing at a time :)

------
josephturnip
I love Requests, but it would be really nice to have some Python libraries
with cert-verified SSL support. I've hacked SSL support on top of urllib2 for
my projects at work
(<https://github.com/josephturnerjr/urllib2.VerifiedHTTPS>), but it'd be
really nice if libraries started prioritizing secure communications as a
first-order requirement.

~~~
kenneth_reitz
This will be in requests _very_ soon. :)

~~~
josephturnip
Then I will be porting my projects over to Requests very soon :)

It's pretty easy to implement, as it already works at the lower (socket) level
with the ssl module, it just hasn't made it up the stack into urllib2 (or
Requests..yet!).

------
seanmcq
I note that you call gevent.monkey.patch_all() in async.

I'm still fairly new to gevent and I'm curious if others consider this good
style for a library?

~~~
MostAwesomeDude
If you're using Twisted (or Tornado, ugh) then there are builtin HTTP clients
which are async in all the right ways and provide similar high-level APIs for
making requests. If you're using gevent, then this is what you want. If you're
using eventlet, you should switch to gevent or learn Twisted.

So yeah, this is suboptimal, but not the worst thing in the world. It should
really be documented though.

------
jsilence
Trying to use the requests module yields:

    
    
      File "/usr/lib/python2.6/cookielib.py", line 38, in <module>
        from calendar import timegm

ImportError: cannot import name timegm

for me.

On two different machines, one OSX SL with Python 2.6 from MacPorts and the
other Ubuntu 10.04.3 LTS with Python 2.6 via aptitude.

Can not find any solution for this via Google and its siblings.

I'm stuck :(

-jsl

------
voyvf
I love this API, and it looks like the dev version is getting even better.
r.iter_content is exactly what I needed! :D

------
nemoniac
Excellent library but they need to fix this piece of misinformation: "A
project that is released as GPL cannot be used in any commercial product
without the product itself also being offered as open source."

This is false. A project released as GPL can also be released on another
(commercial) license.

~~~
kenneth_reitz
Yes, it has to also be offered as open source.

------
pbreit
If Python is going to include urllib or urllib2 in the standard distro, why
not make it good? Http is foundational to (web) programming. I am shocked at
how poor the standard httpclient implementations are in most environments.

------
dannyobrien
Oh, what I would give for a "mechanize for humans" built on this! I love
mechanize, but I dream of not having to go back to its one page documentation
and the source code every time I want to do something.

------
abraham
There is also an addon for Requests that lets you make OAuth requests:
<https://github.com/maraujop/requests-oauth-hook>

------
16s
How does this compare (feature wise) to the poster library? Are they about the
same? I've used poster some, but not requests... I'll have to try it now. Just
curious how it compares to poster.

~~~
kenneth_reitz
The current stable release actually has poster inside (and has for a while),
but it's being taken out soon.

------
cjus
Given that there are a great deal of service APIs in the wild this is going to
lower the barrier of entry for many less experienced developers looking to
prototype their ideas.

------
dreamdu5t
What's with the "for human beings" phrase? Is that a way of saying the
software is easy?

~~~
scott_s
Some APIs are appropriate for programmers to target. Others are more
appropriate for compilers or higher-abstraction libraries to target.

------
jrydberg
It would be nice with something similar for Twisted.

