On a side note, Kenneth is a standup guy who helped propel me into the world of open source. I made a few slight documentation updates on Requests as part of my first pull request ever, and he made me feel like my contribution was so important.
First, it will be extremely hard to get Django to replace a large chunk of it's core with an external library. Not only because they don't rely on an external dependencies, but because there's no obvious benefit.
For example, why would you replace the Django test client with something resembling requests? There's no need to go through the entire request process cycle just to test a small chunk of code. It makes them less efficient, less accurate, and more difficult to understand when something goes wrong.
Second, things like "cachecore" are not synonymous with "http caching", and are extremely confusing names. I think that "uricore" is a good idea, but it's another non-obvious name. It's not a "core" at all, but rather an improved version of urlparse.
I think there are a lot of valuable uses that can come from separating common functionality out so it can be reused in various places, but I dont think Python users are concerned about HTTP not mapping 1:1 with WSGI day-to-day. I can definitely say I'm not, and I'd like to think I have some knowledge as to what matters in [Python] web development.
Luckily, all of these changes will be purely non-destructive refactoring and won't have any negative effects.
That sounds like what economists like to call a "pareto improvement", always a good thing :)
There's no need to test the HTTP lifecycle when you're performing simple "heres the data for my view, execute it, and return the response to me". The http client itself should be tested, as well as Django's internals. It should be clear and obvious separation of concerns.
In the longer term, I'm sure this interoperability will pay bigger dividends, which I'm sure is the reason behind all this anyway.
I agree with the assertion that HTTP is the most common denominator.
# done from memory
In : from webob import Request
In : print Request.blank("http://google.com/index.html")
GET /index.html HTTP/1.0
In : from webob import Request
In : from paste.proxy import TransparentProxy
In : print Request.blank("http://google.com").call_application(TransparentProxy())
I think there's still a place for requests, but I don't see it as becoming the standard way to build HTTP Requests and handle HTTP Responses. I view it more along the lines of a library like mechanize, only nicer. It has it's place.
Requests certainly adds a lot of stuff, some that is below WSGI (e.g., Keep-Alive – WSGI specifically avoids that level of transport), a bunch of stuff that is more stateful (the session stuff), and a bunch of stuff that just felt like it was moving around too much to put in WebOb (e.g., OAuth).
That said, there's nothing keeping a client library from adding that. TransparentProxy for instance could instead be something that handles pipelining and statefulness, and perhaps also acts as a request factory. WSGI is rather helpful here because it does not try to touch those parts – it leaves things open for other tools to take control there. Also to make a better client library you could subclass Request and add functionality that is handy but hasn't been added to WebOb (WebTest does this for functional testing, as an example). Auth probably fits in there. Redirect handling could go in there somewhere too. But if you always redirect transparently then you've made permanent redirects useless.
WebOb itself certainly isn't a client library, its scope is largely limited to representing HTTP requests and responses. But if you are looking for HTTP-related tools, it has a lot of them.
There is a substantial work done to process all the HTTP overhead. In addition to that there is no load-balancing included and it would likely be delegated to a whole new service or machine, when ZMQ could just provide this.
Instead of relying on Werkzeug and HTTP, I can use DictShield models, serialized to JSON or Python, and send those across ZMQ sockets. The ZMQ sockets don't time out, like HTTP, they are instead removed when a host goes down. PUB/SUB messaging is possible, round-robin routing. Lots of patterns instantly available.
Ease in using HTTP itself is welcome, but I don't want to use it in my infrastructure. It's a band-aid for WSGI instead of a solution.
It's not even a matter of preference; it's a matter of a need for HTTP and they are addressing this need, while apparently accounting for 0MQ (he mentions it in the post).
Also, you mention down-thread, "HTTP is the weaker of the two transports"; it's only "weaker" if you ignore any use case that involves communicating with a web browser.
Are you just trying to make an ideological argument that HTTP is used in cases when 0MQ might be a better option? If so, I agree but I just don't understand why you seem dismissive of this project when HTTP has its place and, therefore, this project has its place. The fact that 0MQ is mentioned in this post seems to indicate that the author understands 0MQ's place.
In cases where you can only use HTTP, this will be great, but if I have a choice I'd use ZMQ instead.
I don't mean to sound dismissive, but I am opinionated. I think Kenneth and Armin do great work. I'm sure this project will be excellent.
Requests will have a set of request-level adapters which will let you define the protocol you're speaking, whereas urllib3 aspires to have connection-level adapters which let you define the transport you're using.
So, hypothetically once we make this a reality, you could have a ZMQ connection transport which has a JSON request adapter and happily use Requests with whatever made up scenario you like. :)
For example, zerorpc (http://github.com/dotcloud/zerorpc-python) supports a request-response pattern (using ZMQ REQ/REP), and can return a stream as a response. That maps nicely to http, and I could see that "mounted" as a mock http endpoint which is very interesting.
But there are lots of features in zerorpc that I'm not sure how to map to an http library. For example:
* zerorpc methods must expose positional arguments; how would that map to http query arguments?
* zerorpc arguments can be arbitrary data structures.
* zerorpc has no notion of headers.
I'm excited at the idea of making cross-transport interop easier. At the same time I wonder if the mold of "HTTP-ness" might stifle the ability to think outside the box when designing a transport?
Either way, it'd be great if we had a big list of possible use cases we'd want to support to keep in mind as we're designing this. Anyone want to start one? :D
EDIT: This is incorrect, read below.
Appreciate the heads-up!
gevent has an wsgi module, but it is very low level. I am still looking for a clean/nice way of doing it in python.
$ gunicorn -k gevent
Would Django have to drop their own Request/Response objects and adopt the Requests/httpcore provided ones? Was there any discussion with the Django core devs at pycon regarding it?
And by the way, I love Kenneth's work, especially Requests. We use it in production and it's a joy to build on top of it.
A rewrite of many parts of Werkzeug is required to support Python 3. Might as well kill two birds with one stone :)
Edit: As a bonus link to make this post more interesting, the other day I caused the Python 3 version of requests to be packaged for Fedora 16 and 17: https://bugzilla.redhat.com/show_bug.cgi?id=807525
Composability is one of the most powerful concepts we have. WSGI wasn't mistake-proofing middle-ware by making the protocols different, he was breaking composability that is now being fixed by Kenneth et al.
Thank You! But let us not allow these mistakes to be made again.
Check these slides for an overview of all the things ZMQ can do: http://j2labs.tumblr.com/post/5036176531/zeromq-super-socket...
Even load balancing is included, which means you run less services too.