

Ask HN: Python framework advice - sounddust

This has been asked on HN before, but it was a couple of years ago and the development of Python frameworks is moving rapidly.<p>I'm looking for advice on which Python-based frameworks we should use on a new project.<p>This project will have a high level of traffic  (~50 million page views/day), and will be ported from another platform from scratch, but we will need to use an existing PostgreSQL database (therefore it might be preferable not to use ORMs).  We'll also need memcached support, as well as templating.<p>We want something lightweight, flexible, that doesn't try to do too much for us or at least makes it easy to replace functionality (for example, we have our own session management and rate-limiting systems that we want to preserve).<p>Also important is an active community, good documentation, and proof that it can scale (that is, it's already being used by large sites).<p>I've looked at many frameworks, but none I've found so far seems to meet all these requirements.   (web.py seems to be the most appealing to me, but doesn't seem to have any large clients;  Django seems to be the most developed, but I'm concerned about the "doesn't try to do too much for you" requirement I've listed above).<p>Any advice would be appreciated!
======
zeemonkee
You'll probably get a lot of replies from people saying "just use Django", but
that's just because it's what they are using.

First of, going by your requirements, I'd pick SQLAlchemy for handling your
database. Although SQLAlchemy _includes_ an ORM, it's not just an ORM. It has
an excellent data mapper layer you can use with legacy databases, without
having to drop down to raw SQL:

<http://www.sqlalchemy.org/docs/sqlexpression.html>

If you go with SQLAlchemy the next step is the choice of framework. There are
a few around, but I can only provide advice on 3: Django, Pylons and Werkzeug.

Django + SQLAlchemy is frankly not worth the bother. The real benefit to
Django is the apps that go with it - for example authentication, the admin
site, 3rd party apps such as Pinax and so on. These are great, but if you are
building on top of a legacy DB then you won't be able to make much benefit of
them, and unfortunately the Django ORM is tightly coupled with these apps.

Django without these apps is not IMHO that compelling, as other components -
such as the template engine - are done better elsewhere.

Pylons is the next in popularity - the biggest site I know of running on
Pylons is Reddit. It has more flexibility in choice of components, but lacks
Django's app ecosystem and documentation is sorely lacking compared to Django.
That said, you can use Mako or Jinja2, both excellent template engines, and
you can integrate SQLAlchemy very easily. This lack of good, up to date docs
is a pain point, as is the auto-generated modules - there is a lot of
generated boilerplate code and it takes a while to learn where everything goes
(routing, middleware, etc).

Finally Werkzeug - not really a full framework as such, more a toolkit for
building WSGI applications. This is my personal favourite, as it's well
documented and designed, and lets you structure your application the way you
want to. The tutorial explains how to integrate Jinja2 and SQLAlchemy.

That said, you will spend some time laying out a Werkzeug application before
you get going, unlike Django and Pylons, and I wouldn't recommend it to Python
newbies. There is a new microframework - Flask (<http://flask.pocoo.org>)
built on top of Werkzeug which makes things much easier, but it's a very new
development and is probably not production-ready quite yet.

~~~
alnayyir
>It has an excellent data mapper layer you can use with legacy databases,
without having to drop down to raw SQL:

[http://docs.djangoproject.com/en/dev/ref/django-
admin/#inspe...](http://docs.djangoproject.com/en/dev/ref/django-
admin/#inspectdb)

>These are great, but if you are building on top of a legacy DB then you won't
be able to make much benefit of them, and unfortunately the Django ORM is
tightly coupled with these apps.

InspectDB > migration > populate new models

My current job is converting old systems and old databases for use with
Django. It's pretty easy to convert old schemas and migrate data in Django.

>Django without these apps is not IMHO that compelling, as other components -
such as the template engine - are done better elsewhere.

Nobody said you have to give up those benefits.

Please don't give advice on decisions when you haven't actually dealt with
what's under discussion. It doesn't sound like you've ever actually migrated
anything to Django.

I'd say pylons or Django, fiddle with both, see which one catches your fancy
better. Don't take the advice of others outright.

The more time I've spent with Django, the more my complaints have faded away.
The more I learn about it, the more tricks and tools I find to deal with
anything I encounter.

There's really no point in dismissing it just because you don't understand
inspectdb and migrations.

Edit: For the actual migration process, if you're dealing with a rather large
table (millions of rows) you might want to drop down to SQL for
convenience/speed, but that's still relatively optional. Most of our
migrations are in python + django ORM, not in SQL. (All but 2, out of ~15)

~~~
zeemonkee
I'm well aware of inspectdb and migrations. The OP however stated that he
didn't want to use an ORM.

And what if you need to support composite primary keys ? The ticket on that is
5 years old with no fix in sight:

<http://code.djangoproject.com/ticket/373>

Frankly, the more I've used Django the more I've run up against it's limits.
One of these limits is that the apps such as auth and admin are tightly
coupled to the ORM and template engine.

I've given my experiences with different frameworks, the pluses and minuses of
each. It's up to the original poster to decide naturally.

However, yet again any criticism of the One Holy Web Framework brings ad
hominem arguments and angry rebuttals.

~~~
alnayyir
>I'm well aware of inspectdb and migrations. The OP however stated that he
didn't want to use an ORM.

That part I missed, my bad.

>And what if you need to support composite primary keys ? The ticket on that
is 5 years old with no fix in sight:

If it were me? I'd patch it myself. I've patched django for local use only
multiple times because when I looked at the relevant ticket, they refused a
fix. I've no love for the politics of the project itself.

>Frankly, the more I've used Django the more I've run up against it's limits.
One of these limits is that the apps such as auth and admin are tightly
coupled to the ORM and template engine.

If you're straining against it that much, I wouldn't even necessarily bother
with using SQLAlchemy, it can be rather painful at times. I'd just use nodejs
or <http://wiki.github.com/defnull/bottle/> with an amalgamation of preferred
tools.

>I've given my experiences with different frameworks, the pluses and minuses
of each. It's up to the original poster to decide naturally.

That's pretty much what I said.

>However, yet again any criticism of the One Holy Web Framework brings ad
hominem arguments and angry rebuttals.

Cut the martyr bullshit. It's a technical decision, not a chance for you to
air your ego.

~~~
zeemonkee
Why patch Django when SQLAlchemy supports it out of the box ?

Out of interest, what do you find painful about SQLAlchemy ? It's admittedly
not as easy to use "out of the box" as Django ORM, but the payback is the
flexibility.

Why would I want to use node.js ? I'm quite comfortable in Python, and don't
feel the need to abandon it just because I don't want to use Django. And I did
recommend Werkzeug and its amalgamation of tools.

> Cut the martyr bullshit. It's a technical decision, not a chance for you to
> air your ego.

Air my ego ? WTF ? I've just noticed how sensitive Django fanboys are to even
the slightest criticism, and will go out of their way to justify its
shortcomings. It just gets annoying. It's just a framework FFS with its
advantages and disadvantages.

Don't get me wrong - Django's not that bad; it just that some projects suit it
better than others.

EDIT: not calling you a fanboy personally, but there is a noticeable rise of
fanboyism in Django that reminds me of Rails a few years ago. Let's both try
and keep the discussion both technical and friendly.

~~~
alnayyir
>I've just noticed how sensitive Django fanboys are to even the slightest
criticism

The generalizations are the problem here. _Stop it_

>It's admittedly not as easy to use "out of the box" as Django ORM, but the
payback is the flexibility.

You could say that of assembler. I'm of the opinion that if you're going to
suffer SQLAlchemy, you might as well use something else entirely. Without the
kind of integration Django offers, you might as well deal with the database on
its own terms.

I can't even be bothered to respond to the rest of this. I'm not in the mood
to feed the troll today, not on hackerne.ws

~~~
zeemonkee
Of course you go on about how painful SQLAlchemy is without any specifics,
throw in a statement about "Without the kind of integration Django offers, you
might as well deal with the database on its own terms" and accuse me of
trolling....

------
stse
I'm also in the hunt for a python framework. I've had a look at several, most
have already been mentioned here, like Flask and web.py. Personally I'm
probably going to go with custom framework built on Werkzeug or modifying
Flask.

I think Tornado Web (<http://www.tornadoweb.org/>) might be an interesting
option for you. Should be highly scalable and is being used by FriendFeed.
There were some things that I didn't like about it, but I've seemed to have
forgotten what exactly.

------
khill
Pylons would be my tool of choice for this type of project. I prefer both
SQLAlchmeny and Mako to the Django equivalents anyway but, personal preference
aside, the fact that you have an existing data model and don't want something
that "tries to do too much" make Django less desirable for this project.

------
towndrunk
I have recently started looking into Bottle for a personal project. So far I
really like it.

<http://bottle.paws.de/>

------
scorpioxy
Well, I like Django too but you pretty much removed it from the equation when
you said you want a framework that doesn't do too much for you. That pretty
much means you want a minimal framework. So perhaps you'd want to go with
web.py which if i understand correctly is what reddit uses. Or maybe you'd
just want to go with something like Cherrypy and build things on top of that.

~~~
zeemonkee
Reddit uses Pylons, not web.py (it was originally written in Lisp, then moved
to web.py, then Pylons).

~~~
scorpioxy
Ah ok. I missed that last part then.

But I'd still recommend web.py or something similar because it seems to me
that he doesn't really need a framework, he wants to put one together from
parts he prefers. Doing that with something like Django is a bit painful.

~~~
zeemonkee
It would be good to know how experienced he is with Python. I recommended
Werkzeug elsewhere, which would fit that requirement quite well, but I
wouldn't necessarily recommend it to someone quite new to Python.

------
davisp
WebOb, Routes, SQLAlchemy, Mako. My favorite framework is no framework.

The bottom line is that its a sliding scale. Django will give you a large
amount of functionality for free but requires that you do things the Django
way. As you move along the scale towards projects like Werkzeug you'll
inevitably lose some of the free stuff but you'll be able to make a lot more
choices on how things work.

You'll want to check Ian Bicking's blog for some good reading on the no-
framework approach. [1] is a great place to start.

[1] <http://blog.ianbicking.org/2010/03/12/a-webob-app-example/>

------
sounddust
Quick note: The "50 million page views/day" is a typo. I meant 50 million per
month!

