

Schema migrations merged into Django master - Spiritus
https://github.com/django/django/commit/9aa358cedd1ad93c0f4c20700db7016651dc0598
The Kickstarter:<p>http:&#x2F;&#x2F;www.kickstarter.com&#x2F;projects&#x2F;andrewgodwin&#x2F;schema-migrations-for-django
======
andrewgodwin
I'd like to point out that this is just a first working version, and there's
still some work left to do, but I'm very relieved this is finally merged!

~~~
mattdeboard
Is this pluggable? Can users stick with South (or whatever migration backend)
and still use the same API?

~~~
andrewgodwin
No, this is a complete rewrite and a different way of doing migrations than
South currently does. There should also be a South 2 eventually that is mostly
compatible with this new migration design, for those stuck on older Django.

------
inglesp
This is exciting for Django, and it's exciting for OSS in general, as it
demonstrates the viability of crowdfunding (certain) OSS development.

Congratulations to Andrew and to all involved.

~~~
vanni
For reference:

[http://www.kickstarter.com/projects/andrewgodwin/schema-
migr...](http://www.kickstarter.com/projects/andrewgodwin/schema-migrations-
for-django)

[http://www.aeracode.org/2013/8/23/plan-comes-
together/](http://www.aeracode.org/2013/8/23/plan-comes-together/)

------
j4mie
And here are the docs:
[https://docs.djangoproject.com/en/dev/topics/migrations/](https://docs.djangoproject.com/en/dev/topics/migrations/)

~~~
japhyr
Migrations are per-app, but the command is shown as

    
    
      python manage.py makemigrations
    

Using South, it was

    
    
      python manage.py schemamigration app_name
    

How does django know which app to build migrations for?

~~~
andrewgodwin
It does them all at once. The new system knows about interdependencies, so if
you have ForeignKeys between apps it'll make all of them in the right order.

~~~
arocks
So if I understand correctly, "makemigrations" will convert all apps to use
migrations except apps which were created using previous versions of Django?

~~~
andrewgodwin
It will prompt you if an app doesn't yet have migrations and ask if you want
it to. That prompt is currently a bit aggressive - asking you every time - but
that will be fixed soon.

------
falcolas
If any devs read this, for MySQL, you may want to consider using the update
method adopted by many larger companies - something like pt-online-schema-
change (or make the alter method pluggable so someone else can).

Basically, create a table as a copy of the old table, set up triggers to
update the new table as data pours into the old table, alter the new table,
and then do a rename.

This buys you some rollback capabilities, but more importantly it limits the
impact on production traffic as the alters run. Of course, this is only really
an issue on tables with hundreds of thousands of rows, but it's better than
the naïve approach.

~~~
famousactress
I've come to think this is something that I'll do earlier rather than later on
projects moving forward. The last few projects I've worked on have put this
off "until we get bigger", which inevitably ends up affecting agility and the
types of migrations you're willing to do and when you're willing to release
them. As complex as the approach is, I think I'd rather bite it off early in
future projects.

Or, use postgres!

~~~
falcolas
Personally, I don't start worrying about it until it actually becomes a
problem (i.e. when an alter actually takes more than a few seconds to run). At
which point it's easy to migrate to using pt-online-schema-change.

Then again, I do most of my DB maintenance manually (I have DBA experience),
so while I do use Django in production, I have less need to rely on generic
automation tools.

Or use PostgreSQL, Oracle, MSSQL, or... lots of options there.

[EDIT] As a final thought on the topic, never be afraid of making schema
changes, even in MySQL. It may require a bit more work to implement, but there
are a bevy of solutions (most of which can be automated) which can limit or
eliminate the "pain" caused by such alters.

------
gamegoblin
I started coding Django a couple of months ago and was blown away when I
attempted to add a field to an existing model only to get yelled at.
StackOverflow comments told me that South was the only way to go. I guess I
just couldn't believe it wasn't an already existing feature. It seems that
adding fields is common enough to be needed early on...

Happy about the news!

~~~
glesica
In the past (not sure about now) the Django docs (and thus, I have to assume,
the devs) were somewhat against automated migrations. The rationale was that
the developer should understand his or her data models and do migrations
manually to avoid problems caused by mistakes made by the migration machinery.
Obviously this attitude has changed, perhaps because of the success of South
and the fact that most people would rather have the process automated, even if
it causes occasional weirdness.

~~~
chrismorgan
It was documented as something that was _desired_ , but not something that
would go into Django core until the dust settled as to what the best approach
was; it's not something to hurry into.

What's been implemented now is significantly better than what South had. If
they'd just decided to adopt South, that wouldn't have happened. Thus, it's
good that it happened. :-)

------
jsmeaton
Fantastic work! I've been following the blog posts and it was interesting to
understand the decisions you made along the way.

Also, thanks for supporting Oracle straight away. When you first mentioned the
kickstarter I noted the possibility that oracle support would either lag or be
missed entirely, effectively making oracle support in django a second-class
citizen.

I'm starting a new project in a few weeks (on Oracle RAC), so I'll try to test
as much as possible.

~~~
bmelton
I'm also using Oracle, and have long lamented the lack of South (or anything
like it). I will be a happy test user for this as well.

------
caioariede
A bit OT, but how do you guys handle database changes between multiple
branches? Say I added some fields in branch A through multiple apps and then I
need to go back to the master to do a fix. How do you revert the changes
before checkout master?

~~~
sdfjkl
If you write backwards migrations as well as forwards, South will do that for
you, so presumably the Django migration system will do as well.

~~~
caioariede
I know this. But I'm wondering if there is an easy way to do that, like a
checkpoint that I can rollback all migrations to a specific point.

~~~
kanzure
Some people just choose to use separate database names for each branch.

------
defrex
"Migrations specify which other migrations they depend on - including earlier
migrations in the same app - in the file, so it’s possible to detect when
there’s two new migrations for the same app that aren’t ordered."

This is a fantastic feature. I've worked on projects that use a South fork
that only uses a migration number, eg 0003.py, specifically to cause version
control to trigger a merge conflict.

------
bb0wn
Finally, but too little too late in my opinion.

I love python, but I would rather use Rails for web stuff at this point. So
many 3rd party libraries are needed to do what Rails can do right out of the
box.

~~~
nkuttler
That's fine. Some people consider Django bloated. Some prefer flask. Others
will prefer something with more stuff included than Django. It sounds like you
never were really the target audience for Django?

~~~
bb0wn
What exactly is the target audience for Django? I actually love using Python
and the Python design philosophy. If I'm not part of the target audience for
Django, who is?

~~~
d0m
Well django is already very big.. so the target audience is people using
Django ;-) It's a very stable, python-based, web framework with a huge
community and widely supported. Some python hackers find it too big with too
much bloated features. As far as I'm concerned, anytime I start with a smaller
web-framework I need to add mostly all django functionaly the next few days so
meh.

IMHO, for new projects, DJango is clearly one of the top contestant.

~~~
andybak
Yep. If half the world complains it does too much and the other half that it
does too little then maybe it's got it about right ;-)

------
tocomment
Dumb question here. Instead of keeping all these migration files why can the
migration just compare the models to the database directly and make he
appropriate changes?

~~~
jsmeaton
For a couple of reasons that I can think of. You might want to do a rollback.
You may have multiple copies of the database (various dev versions, staging,
production), and multiple changesets might build up in one environment before
progressing to another which must be specifically ordered. There may be
changes that need to be manually coded or tweaked, and you need some
intermediate structure for that to happen.

------
gitaarik
So what about data migrations? In the docs I couldn't find something about it.
And it's pretty important for a complete migration tool, isn't it?

~~~
teilo
Not sure what you mean. South has something called a "data migration".
However, the only difference between "data migrations" and "schema migrations"
is that data migrations do not include a dry run. Dry runs require transaction
support. Not all databases have it (MySQL).

Other than that, what is your definition of a "data migration"?

~~~
metaphorm
data migration involves mutating data in the table. for example, consider a
timestamp column that was originally recorded in some local timezone but that
you need to convert to store everthing in UTC + offset format. No changes to
the table itself, but a fairly significant change to the data.

you could manage that operation using a Python script, but it would
potentially be slow and might make it hard to take advantage of database
specific features. A data migration tool allows you to describe this operation
in succinct declarative code and then the migration tool will figure out how
to get the database to do that with maximum safety and efficiency.

or you could do it in raw SQL as well, but thats a bit uncomfortable if your
project has used the ORM interface for all database operations. you'd also
have to write different SQL for each backend. a data migration tool can
generate the correct SQL for any backend you plug in.

~~~
teilo
Understood. But I cannot imagine a data migration framework that would be
anything more than a scaffold for custom functions.

Imagine the difficulty involving forward and backward data migrations. I
cannot imagine a way to automate this.

In any case, you can't have it both ways. Either you do it in SQL, or you do
it in Python using the ORM. There isn't a third option.

------
DrJ
People still argue over Flask vs Django.

Django is a working car, the comes with a lot of features, and is more
difficult to customize (and you eventually have to start hacking at it for
custom features).

Flask is the assembled chassis, engine, steering (no frame, body). It's up to
you to build it into a car of your choice.

Eventually though you will have to look at tuning the base system for your
special special system.

------
Demiurge
So is this slated for 1.7 now? Or is it still on track for 1.6?

~~~
tomchristie
It's for 1.7, always has been.

~~~
Demiurge
Ok. But it hasn't always been, according kickstarter
([http://www.kickstarter.com/projects/andrewgodwin/schema-
migr...](http://www.kickstarter.com/projects/andrewgodwin/schema-migrations-
for-django)) 1.6 is mentioned, and "1.7 at latest". Considering 1.6 isn't out,
I was hoping for it!

~~~
tomchristie
Noted, apologies. 1.6 is in beta and feature frozen, and it's certainly been
the plan for a while to target 1.7.

I wouldn't sweat it tho - the 1.5-1.6 cycle has been pretty quick, and I'm
sure as soon as the migrations work is ready and tested the Django core team
will be pretty eager to push a 1.7 release.

~~~
Demiurge
I'll work on tempering my excitement :)

------
slig
I briefly read the doc in the changeset, but couldn't find if the migration
files from South are compatible with this new bultin migrations. Does anyone
know that?

~~~
kingkilr
No, south migrations are not compatible, you'll have to collapse your south
migrations down and start with these.

------
Ensorceled
Yes! My £50.00 was well spent :-)

------
nilved
Awesome, Django might be usable out of the box now. :)

~~~
nkuttler
Yes, for people who couldn't type

    
    
        pip install south

~~~
nilved
That's why I specified "out of the box." It doesn't make sense for there to be
completely mandatory extensions like South -- they should be bundled in,
fitting with the "batteries included" culture of Python.

~~~
nkuttler
I don't really feel out of the box when I type

    
    
        pip install django south

~~~
nilved
Regrettably, how you feel doesn't change the well-understood and long-standing
definition of "out of the box."

------
lectrick
Welcome to Rails circa 5 years ago! :)

~~~
moe
Except no, Rails doesn't have this. Not even remotely close.

Rails still makes you write your migrations by hand. It also forces you to
distribute your critical constraints, validations and dependencies over a
usually incomplete model class, and incremental migration files.

In Rails/AR there is _no place_ in the filesystem to find out about the
currently declared schema.

It's a fundamentally broken design.

~~~
bmckim
db/schema.rb

~~~
moe
schema.rb contains an _incomplete_ reflection on the current SQL schema. No
references, no relationships.

If you don't know the difference between AR and a declarative ORM you may want
to refrain from dropping smarty oneliners.

~~~
lectrick
At absolutely no point in our dev team's existence did we find a huge need to
have more than the model files and/or schema.rb as a schema reference. If you
can't infer what you need to from there, then maybe web dev's not your bag,
baby

~~~
moe
_then maybe web dev 's not your bag, baby_

Over the past 15yrs I've written web-apps in 7 different languages and a wide
range of frameworks.

From your comment I can infer that you know Ruby on Rails, and that's pretty
much it. Maybe you should reconsider your tone?

------
workhere-io
Somewhat related: Are there any plans to integrate Alembic into SQLAlchemy?

~~~
lucian1900
It's about as integrated as it could get while still being a separate package,
since it's written by the author of Alchemy.

Most users of SQLAlchemy tend to use several libraries anyway, as opposed to
one framework (the Django style).

------
groundCode
This is great news for Django - congratulations to all involved!

------
Daviey
Neat. Once this hits a stable release, does it mean that South should be
considered deprecated for new projects?

~~~
andrewgodwin
For anything on Django 1.7 (the version it will come out in), yes, South will
no longer be needed. Older versions of Django should get a South 2 in the next
few months with a lot of this code backported.

------
ris
But does it work with custom ("initial") sql?

This was one of the failing points of south in my opinion.

------
kdazzle
Why wouldn't Django just use South instead of creating ever more dependencies
on itself?

~~~
simonw
The migrations support for Django is being written by the author of South,
based on everything he learnt on that project. It's essentially South 2 but
baked in to Django.

------
est
Now let's get rid of WSGI and make async signals possible!

~~~
jgroszko
Are you trying to combine WebSockets or realtime communication with Django?
I'd look at Cody Soyland's excellent work on this.
[http://codysoyland.com/2011/feb/6/evented-django-part-one-
so...](http://codysoyland.com/2011/feb/6/evented-django-part-one-socketio-and-
gevent/) or look at the Django example in gevent-socketio
[https://github.com/abourget/gevent-
socketio](https://github.com/abourget/gevent-socketio)

------
ciokan
I absolutely love python and I'm using it daily but Django seems
so....outdated?! Biggest web framework needs someone to raise funding via
kickstarter to build a schema migration?

2013 and you still can't build a REST application without installing 3rd party
code? I mean c'moon there's angularjs, emberjs, knockout, extjs etc etc etc
out for years don't you see a light at the end of the tunnel or you're the
type of team that build more tunnel?

Sorry to say but news like this just make me sad. Still happy I chose flask +
sqlalchemy for my website. Django is somewhere in 2004-2005 still. I lose the
admin of course but I hate general things anyway (one size fits all type of
thing).

~~~
wbond
I recently built (more or less) a REST API and a Backbone front-end using
Python. Why would I want to use Django to do so when there are so many other
(more appropriate) options? Does Django really need to be a framework that
does everything?

I started before Flask supported Python 3, so I used bottle, psycopg2 and a
bunch of other python packages. Knowing what I was interested in
accomplishing, I was pretty easily able to stitch together these libraries to
create a solid, to-the-point codebase.

Having spent a bunch of time working on various backbone-heavy apps with Rails
I don't have much interest in using a big, full-stack, opinionated framework
to act as a REST API. There is so much overhead and so many opinions to work
against.

~~~
taude
Couldn't agree more, especially when you're basically speaking JSON with some
authentication/authorization. I even question myself when using Flask-Restful
vs just making my views return straight up JSON....but...that's a different
discussion.

