
Lessons learned defying Joel Spolsky with Django - benregn
https://speakerdeck.com/nickbruun/lessons-learned-defying-joel-spolsky-with-django
======
nnq
Bumping into ORM limitations + moving to Jinja for templates --> one word:
Flask

...really, what advantage does Django provide at this point in this project
anymore?

~~~
Alex3917
"what advantage does Django provide at this point in this project anymore?"

Documentation. I use Django but without any ORM and with Jinja2, so it's
basically just Flask but with more stack overflow threads and more third-party
software.

Is there any actual advantage that I would be getting by using Flask instead?

~~~
hcarvalhoalves
The advantages aren't really in using Flask, but being able to use a better
ORM (or no ORM at all, just SQL abstraction) and template engine.

After working with Django ORM for a large project, I started to rethink the
usefulness of models. It turns out just treating data as sets lends to a
functional style and is simpler down the road.

~~~
tomsthumb
> It turns out just treating data as sets lends to a functional style and is
> simpler down the road.

Could you elaborate on this or point in the direction of something that does?
I'm pretty sure I get the gist but having some more meat to chew to make sure
the perspective is fleshed out fully would be awesome.

~~~
hcarvalhoalves
The bottom line is, tying data to objects seems like an elegant idea, but in
practice it sucks. ORMs introduce various problems, like statefulness (when
the whole point of dealing with a database is atomicity) and mismatches (your
data doesn't necessarly map to one row or one object, it doesn't have to).
I'll try to give two examples.

Consider how people often write update code on Django:

    
    
        instance = Model.objects.get(id=some_id)
        instance.foo = bar
        instance.save()
    

This sucks a lot. It's doing two queries, and you have concurrency issues.
This would be optimal, since it's atomic:

    
    
        Model.objects.filter(id=some_id).update(foo=bar)
    

But in this case, it's not instantiating any object nor triggering any
signals, so what's the point of the ORM after all? We might as well just use a
sane DB API.

Here's another place where Django's ORM fails:

    
    
        Model.objects.annotate(Count('foos')).filter(foos__lt=10)
    

This won't work. Django will complain 'foos' is not a field on the model -
even though 'foos' is a column in the result set, and it's perfect valid to
SELECT on it. I reported this as a bug, but because the ORM works with model
definitions and the result set can contain any column, what the ORM is really
supposed to do or not is murky, so it's WONTFIX. This is one instance where
data doesn't map to an object and the ORM concept crumbles.

There are many other pain points with ORMs, and Django's in particular, but
these are the highlights for me. For an elegant querying API, in my opinion,
check out RethinkDB. It doesn't depend on schemas (therefore, ORMs) and it
supports map/reduce semantics, which solves 100% of what you need to do with
data.

~~~
subleq
You are mistaking about filtering an annotation.

    
    
       > Model.objects.annotate(Count('foos')).filter(foos__lt=10)
    

The only reason this doesn't work is because the property annotate creates
isn't called `foos` by default. You just need to do this:

    
    
        Model.objects.annotate(foos=Count('foos')).filter(foos__lt=10)
    

and Django now knows to add a `HAVING COUNT(foos) < 10` clause.

------
zzzeek
Don't use just one ORM and then declare "ORM's are stupid". The "object =
None" / "object_id = None" issue illustrated here is certainly not a mistake
every ORM makes.

~~~
Aqueous
Why not just write the SQL?

~~~
drdaeman
The only problem I see, is we can't rewrite SQL.

I.e., in pseudocode:

    
    
        query = SQL("SELECT * FROM entities WHERE owner = ?", owner=me)
        ...
        if some_condition:
            query = query + SQL.WHERE("OR public = TRUE")
        ...
        if other_condition:
            query = query + SQL("LEFT JOIN things AS t"
                                " ON t.entity_id = entities.id") \
                          + SQL.WHERE("things.value > 0")
        ...
        my_nice_list_of_results = run(query + SQL("LIMIT ?", count))
    

This should be technically possible, but I haven't seen any library that does
it.

~~~
memset
I have always wondered why people have an aversion to stored procedures. They
are a programming language in themselves, so you could pass all of your
parameters (owner, some_condition, other_condition) into a proc and, depending
the logic, return a different cursor to a different query.

~~~
zzzeek
it's because the stored procedure development model lacks the tools in order
to make integrating with an application-level domain model simple. You end up
needing to write not just one persistence layer, that of marshaling your
object model to and from SQL statements, but two - all the SQL statements
behind your stored procedure layer, and a second to move all the data between
the SPs and your object model. To make matters worse, the stored procedures
must be written completely by hand without the benefit of any in-application
schemas to help compose statements.

One reason for the variety of opinion on this is that different developers
make more or less use of domain models in the first place. Those who are
accustomed to writing all SQL completely by hand with no helpers at all, and
not working with a domain model tend to view the stored procedure approach as
equivalent. Those who are accustomed to having at least some simple marshaling
layers like a construct that generates an INSERT statement given a list of
column names see the SP approach as more tedious since simple techniques like
that are usually not easily available, at least in more old school SP
languages like TRANSACT-SQL and PL/SQL.

All of that said, I do think this is a problem that can possibly be solved.
Postgresql allows SPs to be written in many languages, including Python. I
have an ongoing curiousity about the potential to integrate a Python-based
object relational system into a stored procedure system. But it might end up
looking like EJBs.

------
jroseattle
I was expecting this slidedeck to be a bit more focused on defying Spolsky,
and whether or not that was a good decision.

FWIW, I've never bought into Spolsky's vision that re-writing code is poor
strategy. Steve Jobs never thought twice about ripping something apart and
starting over. If anything, code re-write can be an advantageous position --
you often have a greater understanding of the problems you're intending to
solve. When well-executed, it can take the form of heavy refactoring, even
when switching languages/platforms.

~~~
LockeWatts
I'm not really sure why Steve Jobs is your model for code design.

~~~
widdershins
Because he oversaw the creation of 3 of the most successful operating systems
of all time? I know he didn't design them, but he was involved in managing the
projects. Just playing devil's advocate.

~~~
zerr
Well, it doesn't mean he evaluated at the code level and decided that
rewriting is better than refactoring...

------
lstamour
I found I could follow along with the slides, though some of the icons and
messages around third-party tools were lost on me. I'd definitely appreciate a
video.

Edit: 9 minutes ago, Nick posted to Twitter: "we have the recording - just
need the sound cleaned up. Expect it early next week ;)"

~~~
lstamour
Video now live: <http://vimeo.com/65057265> [Slides in sync with audio]

------
emperorcezar
While I love seeing slides, this deck obviously could use the audio or
transcript along with it.

~~~
rattray
Agreed, I for one was felt feeling rather in the dark for the last few slides
especially...

~~~
iconfinder
We are posting a version with audio on our blog on Monday:
<http://blog.iconfinder.com>

~~~
abs_farah
I look forward to it!

Only audio, no video?

~~~
iconfinder
We will combine audio with the slides :-)

~~~
lstamour
Video now live: <http://vimeo.com/65057265>

------
gingerlime
_> 20ms with Jinja2 without auto-escaping_

could this performance improvement back-fire if you end up with a security
issue?

I'm not saying that it definitely would. If you know what you're doing / trust
your data sources or sanitize them elsewhere, you should be fine. I'd be
careful turning off such a feature completely though...

~~~
Daishiman
Of course. This is merely a tradeoff between performance and developer time.
99% of projects will never have HTML autoescaping as a performance pain point.
Then again, you're going to need tens of hours to review all templates to make
sure you're escaping everything. If your hardware budget is greated than what
it costs to audit the code, it's the proper decision.

------
tachion
I wonder, is there a video of the talk available? The slides, unfortunately,
alone are rarely very informative.

~~~
iconfinder
We will post video version soon (Monday) on <http://blog.iconfinder.com>

------
level09
I found these tools mentioned by the author really helpful, I compiled a list
of them:

<http://jinja.pocoo.org/> <https://www.getsentry.com/welcome/>
<http://graphite.wikidot.com/start> <https://opbeat.com/>

is there a way though to use/test opbeat ?

~~~
iconfinder
Just send them a message via twitter or email. I'm sure they will help you get
started.

------
hasenj
You should give SQLAlchemy a try:

<http://lucumr.pocoo.org/2011/7/19/sqlachemy-and-you/>

It gives you more control and requires you to be more explicit about your
queries and relationships.

------
speg
Does anyone know what tool was used to profile the django app? Looks cool.

~~~
naithemilkman
If you're talking about the slide that says '87.91% of time spent rendering'
then yes, I would like to know too.

Annoying that it doesnt have slide numbers to refer to.

~~~
cdavid
it looks like kcachegrind or pycallgraph
(<https://github.com/gak/pycallgraph>).

------
claudiusd
I don't think Spolsky's argument applies to startups... His argument basically
boils down to "You think the code in front of you is a mess, but the reality
is you're just having trouble reading somebody else's code which is probably
good enough". But what if you wrote the code yourself? In that case, it's
probably just a mess.

------
scott_meade
ORMs may be "stupid", yet Basecamp manages to handle 400-500 request per
second with Rails and MySQL just fine.
<https://twitter.com/dhh/status/287221705443774465>

------
Uchikoma
Iconfinder. They are profitable? They have business requirements? Joel wrote
about the problems of a rewrite when you need to earn money, implement money
making features, heavy competition while at the same time maintaining two
plattform where one is a moving target. I don't think Iconfinder fits in any
of these constraints.

~~~
nickbruun
Incorrect. We fit in all of those constraints at the point where a rewrite was
decided ;)

~~~
Uchikoma
Thanks for your reply, from looking at Iconfinder it did not look like a large
piece of software with lots of business requirements.

------
iconfinder
We have posted a video version of the talk (slides + audio):
[http://blog.iconfinder.com/staying-sane-while-defying-
joel-s...](http://blog.iconfinder.com/staying-sane-while-defying-joel-spolsky-
with-django/)

------
bsaul
Did someone understand the last slides about the differences between
transactions in tasks with and without celery ? I'm using celery, and i have
been using django in the past, and I really didn't get the point.

~~~
mbell
He's trying to avoid the situation where you enqueue a background job inside a
transaction and the worker gets started on that job before the transaction is
committed. If the job needs to hit the database for any reason your likely to
get errors as the new data hasn't been committed yet.

It looks like the work around he used is to cache the job queue locally and
only flush it to the real job queue after the database commit, so your
guaranteed whatever data may be needed for the job has been committed to the
database.

------
dmishe
Interesting point on celery and transaction, though I've never seen that
happen in real life, interesting.

------
Uchikoma
+100 for using "leaking abstraction" in a slide deck "defying Joel Spolsky"

------
beambot
Is there a (publicly available) video that accompanies this slide deck?

------
Kiro
I didn't understand why ORMs are stupid. Can someone enlighten me?

~~~
macspoofing
ORMs pretend to give you full abstraction, but in reality you have to be aware
of the underlying SQL layer when you build your object model.

~~~
antihero
I find that if you are aware of the SQL underneath, they are good for saving
time.

Do you really want to write SQL to retrieve data and code to populate an
object for every damn thing in your system?

~~~
macspoofing
I agree. Problems happen if you treat ORMs as a total blackbox.

