

WSGI and the Pluggable Pipe Dream - DasIch
http://lucumr.pocoo.org/2011/7/27/the-pluggable-pipedream/

======
endlessvoid94
That was quite a rabbit hole, but a good read nonetheless.

I was pretty surprised at all the criticisms that Pump received yesterday
(<http://news.ycombinator.com/item?id=2810373>). The general response was
defensive and critical just because it was trying to replace WSGI. However, it
would seem that WSGI isn't perfect for all applications and indeed there is a
place for Pump. My question is: how can we resume encouraging people to
contribute? Quite honestly, if I had posted that thread and received the
response it got, I would be utterly discouraged from contributing to the
community in the future.

~~~
adamtj
WSGI is _not_ a framework. It's a _gateway interface_ , like CGI. Django is a
web application _framework_. You don't (usually) write apps using a raw
gateway interface like WSGI or CGI. You write web applications in a framework
like Django or Pump.

The framework then needs to work with the web server -- but how? To be
portable and useful, a framework it should support CGI, SCGI, Fast-CGI, ISAPI,
HTTP, etc. That lots of work to do for every single framework. Big ones and
little ones all need to support all those interfaces, and you know they won't.
That's the whole point of WSGI: to be one single abstraction to bind all web
servers to all frameworks.

If you want to write a framework, WSGI sets the bar pretty dang low. It
supports pretty much everything you need to do HTTP without too many awkward
quirks. You don't need to worry about how to implement a fast-cgi->nginx
adapter. You just make your framework talk WSGI and plug it into existing
nginx->fast-cgi and fast-cgi->wsgi libraries. Done.

WSGI was carefully designed to be as simple as possible, but no simpler. It is
simple, but powerful and flexible. It's possible to write powerful composable
middleware that works with _any_ webserver and _any_ framework, because
middleware is just a WSGI app that wraps a WSGI app. Middleware can
transparently add sessions, caching, compression, chunking, etc, that don't
need to be tightly coupled to the framework or the webserver. So, just like
WSGI can bind all webservers to all frameworks, any middleware can insert
itself between any framework/webserver pair (I'm sure there are exceptions, of
course). The framework shouldn't have to screw around with chunked encodings
or compression. WSGI's design allows one good library to easily do that work
regardless of the rest of your stack.

This article spoke about how awkward send_request() is. What it does is make
it trivial for middleware to read, add, and modify headers without having to
parse HTTP. Suppose you want to write a javascript obfuscator. One way to do
that is to write some middleware that detects Content-Type="text/javascript"
and obfuscates the response. Because of the way send_request() works, you
detect javascript by looking at a variable, not writing an HTTP parser. Also,
if you write that middleware for Django, it'll work unchanged when you get fed
up and switch frameworks.

The whole write() vs yield vs return thing can is awkward, but it makes it
easier to write a wsgi adapter for older cgi-based frameworks that respond via
sys.stdout.write().

Only framework, middleware and gateway authors have to care about WSGI. If a
framework author is complaining about WSGI, he or she almost definitely don't
understand where WSGI came from or what problem it's really solving.

WSGI is minimal, composable, and elegant, even considering that it's backwards
compatible with older frameworks. If everything supports WSGI, then everything
can just be plugged into everything else without much screwing around.
Composability and loose coupling are very powerful, especially at the levels
of the stack where WSGI lives.

~~~
pyninja
I agree with you, but Pump is not a web framework. You are not supposed to
write your application with Pump. Picasso is a framework I built on top of
Pump (<https://github.com/adeel/picasso>) that's more suitable for that.

(I can see how you could get a mistaken impression from reading the discussion
on HN, though.)

------
afhof
"But before you blindly throw it away you have to understand why it was
created in the first place."

This is an excellent piece of advice that I wish more people would use. If you
find something that you disagree with (a law for instance) it is a good idea
to first see how it came to be. The decisions people usually have some
reasoning behind them that can't just be discarded when being evaluated.

------
mclin
The last part was the most insightful for me.

 _Personally I am pretty damn sure that WSGI no longer carries the importance
it had a few years ago. I think it no longer makes sense to merge different
applications on the WSGI level together, it should be done on a higher level
and JavaScript is a nice way to do that._

 _Just think about Google's gray bar. You can totally throw such a bar on top
of different independent parts of the application by emitting a tiny piece of
JavaScript that generates that bar and handles your user session._

I started thinking about it and was like "Right, you can make multiple async
requests to hit multiple apps!"

------
7952
I like the conclusion about using http. This goes against the approach
advocated in monolithic frameworks such as django. It would be interesting to
see a web framework that separates components and connects them with http.

~~~
adamtj
That's _exactly_ what WSGI does. Well, except for one brilliant improvement:
It doesn't pass around raw HTTP, it passes around pre-parsed HTTP data
structures. There are three structures: a response code ("200 Ok"), a list of
headers, and an iterable that yields the body of the response (or older
frameworks write to a filelike object that you can reimplement if necessary).
Dealing in pre-parsed (or rather, not-yet-serialized) HTTP makes it much
easier to write middleware.

If you _really_ want to plug stuff together with HTTP, there exist HTTP
servers that serve wsgi apps, and I'm sure there is a wsgi app somewhere that
is an HTTP client. So you plug your framework's wsgi-app (which you didn't
write and don't have to worry about) into an HTTP server. Then you point a
WsgiHttpProxy at your HTTP server. Plug that WsgiHttpProxy app to your
middleware and plug your middleware into another instance of your HTTP server.
Rinse and repeat for as many middleware layers as you want. Now everything is
HTTP, like you want.

Or, you know, just plug all your middleware together, avoiding parsing,
serialization, IPC, context switching and all that crap.

~~~
Wilduck
The author isn't arguing that this is a bad idea per se, just that it doesn't
actually work that easily.

>> But what I do not believe in is that magical plug that is called “framework
independent pluggable application”. I don't know where this idea came from
that it might work, but it does not. The idea that you can reused code on top
of WSGI to work with Framework 1 and Framework 2 is not working out.

~~~
adamtj
If it's not working out because wsgi.input can't be read twice, then write a
middleware that turns it into a StringIO, and write another middleware that
rewinds it.

Nothing is going to save you from the fact that you either need to buffer your
response, or don't throw an unhandled exception after it's started. Yes it's a
problem, but buffering is the only magic that'll help.

If the problem is that middleware authors can't be bothered to implement the
whole spec because it's too hard, then submit a patch. I promise you that WSGI
is easier to implement than HTTP/1.1 In fact, I would argue that it's as easy
as possible. If it were easier, we probably wouldn't be having this
discussion, because it wouldn't have been powerful enough to have been
adopted.

Or write a middleware container that transforms the WSGI api into something
you think is more sane, then implement middleware in thate.

Build the next great thing on top of WSGI, because in the end, the answer is
either WSGI, something that's architecturally equivalent, or else each
framework has to have its own compression, session, and other libraries. And
no answer is going to save you from having to do a little bit of work to put
things put together correctly.

~~~
7952
I just don't understand why we need a single monolithic product that is at the
core of the system. When you use any language specific API you are locking out
possibly useful tools.

The overhead of asking each component to use HTTP is an acceptable price to
pay for native compatibility with the Internet.

------
perfunctory
There are a few things I don't understand in this post

> neat nod to async systems that can use this neat trick to yield empty
> strings to signal that they are not ready yet.

Can't the system just do nothing to signal it's not ready yet?

~~~
the_mitsuhiko
> Can't the system just do nothing to signal it's not ready yet?

In an async system that means you either have to suspect execution (for
instance by yielding something) or to block. And blocking is bad.

