

Common Pitfalls with Django and South - andrewingram
http://andrewingram.net/2012/dec/common-pitfalls-django-south/

======
andrewgodwin
(Author of South here) Nice article - good set of common pitfalls, too. The
multiple people one is especially tricky for people when they first encounter
it, it seems.

I'm hoping I can resolve some of these issues (like the permissions thing)
when we start moving parts of the code into Django proper.

~~~
arocks
Thanks for the update and for creating such an indispensable tool. Regarding
your last line, is there any plans to integrate South into Django mainline
code in the near future? Is there a place to read the philosophy or roadmap
for this?

~~~
andrewgodwin
There is, hopefully in a couple of releases' time. There are some posts I made
to the mailing list about it a while ago - I can't look them up quickly though
- and a talk I did at DjangoCon this year.

------
hcarvalhoalves
Everything stems from the fact South attempts to freeze your model definitions
in time for each migration. While this works, often you need _more_ than just
the model definition (what effectively generates the DDL), but the surrounding
codebase / Django magic as well (methods, model managers, etc.).

Another pain point with South is that you run your migrations _after_ pushing
a new release. That makes for hard zero-downtime updates, because your
codebase and the database are out of sync for a while before you can run
`django-admin.py migrate`.

After having used South, I feel the best approach still is to write your own
SQL migrations by hand and run it against your database _pre-deploy_ , never
destroy data (erase/update columns/tables), and then make sure your codebase
handles the new/absent columns/tables transparently. This way you can revert a
broken release just pushing the previous version.

~~~
bmelton
I've always had luck just running the migrations before hand -- the only real
caveat there is that you need to do a two-step migration process if you're
removing fields.

If all your changes are additions, and your app is managed by supervisord,
it's fairly trivial to pull the code updates, run the migrations, then bounce
your supervisor instance.

Like I said though, if you're deleting, the process is to stage out the
deletes into a separate migration so that your codebase is using new or
added/replacement columns before you remove tables or you're kind of screwed.

Also bear in mind that this knowledge is somewhat dated as all my recent
Django ORM experience is based on Oracle (ugh) which I haven't gotten South to
work with.

~~~
hcarvalhoalves
> I've always had luck just running the migrations before hand -- the only
> real caveat there is that you need to do a two-step migration process if
> you're removing fields. If all your changes are additions, and your app is
> managed by supervisord, it's fairly trivial to pull the code updates, run
> the migrations, then bounce your supervisor instance.

The problem is that some configurations need workers to restart every X
requests to avoid memory leaks, and that makes the deployment process
completely non-deterministic if you're pushing new code and expecting workers
to still serve the old one. There are troubles with dynamic imports too, which
happen on large codebases.

~~~
bmelton
That's a good point that I didn't consider. I'm using Gunicorn and now that
you mention it, I dunno how much is or isn't dynamic, but my process is
dynamic so it takes seconds to happen and may or may not be throwing errors.

I've never _noticed_ any errors deploying as described, but that doesn't
necessarily mean anything in light of this information.

------
jwpeddle
I've seen a couple of articles that have suggested both "don't use the orm"
and "load fixtures using loaddata in your datamigration". These things
contradict each other. loaddata uses your models, which will be potentially
out of date. We have had to manually load up the fixture using json.loads,
then create the model instances using orm.Foo and setattr.

Also, not a great solution, but we monkeypatched south to simply use the
number in the file name, ie 0001.py. This way if two developers try to commit
a migration, they cause a conflict, and we can fix it then instead of finding
out later. It makes the filename less friendly, but we usually use grep/ack to
find a migration anyways.

~~~
andrewingram
This is actually a really good point and a glaring oversight on my part, I'll
have to edit that part of the post.

[Update: I've changed the last paragraph of the article to reference your
comment and this solution]

I think the solution is to create a version of loaddata that knows how to work
with South's frozen ORM. A quick search turned this up:
[http://stackoverflow.com/questions/5472925/django-loading-
da...](http://stackoverflow.com/questions/5472925/django-loading-data-from-
fixture-after-backward-migration-loaddata-is-using-mo/5906258#5906258)

------
bokonist
I use South on a fairly substantial django project (35 models, 85+
migrations). After around a few dozen migrations we got really frustrated with
south. We still use south, but now run all migrations via db.execute(""CREATE
... """) with manual SQL rather than using the south models and the south
wrappers. There were too many instances when there would be some little thing
wrong, and we would get long, very hard to debug error messages. Also a number
of times I would get my local install into a weird state where I could not run
migrations, and I'd have to manually patch up. By doing straight SQL I always
know exactly what is running, and so it is both easier to debug locally and
feels safer to run against the prod data.

When we switched to db.execute, we lost the ability to do backwards
migrations, but that's ok. Backwards migrations are really, really dangerous.
In fact, deleting columns, ever, is really dangerous. If you are deleting data
permanently it is always better to hand craft precisely what you want to do,
and have everyone look it over, rather than running a backwards migration.

~~~
stevejalim
I've used South on at least 20 projects, some of which were very large (1yr+
of development, 100+ migrations, incl data migrations and raw SQL execution)
and generally had Good Times throughout (aside from the coordination of
migration on multiple developers' branches that is less of a South problem and
more of a workflow one).

Caveat/qualifier: every project was on Postgres.

------
ivix
Here's another:

If you look at your history and think 'oh, migration 0005 hasn't been run',
don't do 'migrate 0005'. That will of course take you _back_ to 0005, deleting
any modifications made after 0005.

------
rheide
We've just started using South on Heroku, and one thing I've been confused by
is that I have to deploy the code to the live server in order to be able to
run the migration commands, thus causing the site to run on the new code that
depends on the post-migration database structure for a short time.

Does anyone know an easy way to run the migration on a Heroku Postgres
database without pushing the code to a live server?

~~~
codeinthehole
One way is to use symlinks to link in your live build. Then your deployment
process can upload your new codebase and run the migrations before switching
the symlink so that you new codebase is being served.

Where I work, we have a templated Django project that has a fabfile to do
this: <https://github.com/tangentlabs/tangent-django-boilerplate>

If you look in the deploy function ([https://github.com/tangentlabs/tangent-
django-boilerplate/bl...](https://github.com/tangentlabs/tangent-django-
boilerplate/blob/master/fabfile.py#L123)), you can see the flow is something
like:

    
    
        def deploy():
            deploy_codebase()
            ...
            migrate()
            ...
            switch_symlink()

~~~
rheide
This is a very creative solution. I like it.

------
kmfrk
I would love to see some code examples, especially for the cases where
something breaks, until you have your wits about you.

------
mikle
I've been using south for a few months now and I didn't even know half those
pitfalls could exist. I'm Using south just for migrating simple changes to my
DB and it is wonderful for that, but it seems like there is more depth to what
it can do.

Great article.

------
peregrine
I wish this article was around year or two ago. I had an awful experience with
south so much so that I switched to ruby and sequel. Sequel totally decouples
your migrations from your model. It's pretty nice actually and ends up saving
quite a bit of headache.

~~~
andybak
Wow. Must have been a pretty bad experience to warrant switching languages and
frameworks!

Care to elaborate? I've always found South pretty straightforward.

~~~
peregrine
Lot of things described here, circular references, conflicts amongst the team,
trouble with simple data import. These things are a result of the coupling of
models to migrations. The method of models describing and generating the
migration which generates the db is troublesome.

~~~
andrewgodwin
However, in the reverse way, the migrations defining the models (as in Rails)
often leads to mismatches as well. There's no easy solution to this problem,
alas.

When model-based migrations work, they work really well, and help reinforce
code schema/database schema being entirely in sync.

------
tocomment
I recently started doing professional Django training. Migrations haven't come
up in training sessions yet but I'm not sure what I would recommend. I feel
South might be too intimidating to thiose just learning Django. Are there any
alternatives?

~~~
famousactress
When I first started doing Django, I found South intimidating. I think it's
mostly just some vocabulary and concepts that end up feeling more complicated
than they have to be (I remember the orm freezing being totally confusing, and
things like my team members always 'faking' migrations baffling me).

That said, I think introduced properly/carefully South is the easiest way to
do migrations by a landslide. If anything, I'd expect people to eventually
grow out of it, not into it.. And not most people either. I think a number of
the commenters talking about hitting bumps in the road once they got to
(gasp!) dozens of models or migrations, probably bailed prematurely. Our
project is 300+ models, and well over 500 migrations now and South still suits
us pretty nicely.

If you'd like any help reviewing tutorial materials, I'd be happy to lend a
hand. I haven't looked lately, but I'd love to see a better intro blog post
for South show up. A couple years ago when I was learning I found what's out
there just a little bit obtuse (not to knock Andrew's very solid tutorial at
all, I just could have used something even more stripped down and careful
about introducing concepts to get started).

------
batgaijin
I will defend ORMs until my dying day, but south is entirely new beast. Took
me a few days of screwing around to be able to safely dig myself out of my own
holes.

------
camus
just started using python south ( and ruby taps ) ,so great tips ! tx

