
Schema Migrations for Django - pykello
http://www.kickstarter.com/projects/andrewgodwin/schema-migrations-for-django
======
nicpottier
Just threw $100 your way.

I'm a big believer in this type of model and having you be a Django core
contributor and South developer makes it a no brainer. Paying for quality open
source is always a win in my book.

Godspeed!

~~~
easytiger
I am, on the other hand, not a fan of this investment model. Indeed I would go
so far as to say I don't think it works nor does it make much sense.
Especially in the open source world.

I've been using django since 2006. This has been a suggestion since then. Why
hasn't it been implemented if it is so necessary? It doesn't require a massive
core shift, it doesn't even require that much work it seems? So why hasn't it
been done with so many eyes on so well documented a platform on so accessible
a language with so many commercial deployments? If no one has scratched this
itch then paying for someone to scratch an itch you may think you are go

Will the person be deprived of their normal source of income for exactly the
time period required to do the work? How is that justified as a cost? Maybe
they are a contractor who will take the time out. If that is the case how can
there be expandable requirements (longer goal objectives for more money? It
could be this whole thing sits very well with their day to day job, but in the
western world that is unlikely. I'm sure all intentions are good but it also
strikes me as a bit questionable to do these things when lots of other people
have committed much more centrally to the project with little immediate
remuneration.

~~~
flexie
I don't understand the concern. It's a real problem and if someone wanted to
do it for free, they could have and would have. Here is a guy who offers to do
it if he get's paid a minimum amount. Fine. No moral issues.

~~~
ProblemFactory
> Here is a guy who offers to do it if he get's paid a minimum amount.

I'm sure Andrew would work on this even if the kickstarter had failed - he
built South without one after all :)

But having a funded project helps in many ways. It frees him from the
distraction of contract work, proves that a lot of people really care about
the issue, and finally is a thank-you-donation for the work on Django in the
past and the future.

~~~
adamc
It's a no-brainer if only as a thank-you for something I've used and certainly
appreciated.

------
andrewgodwin
Worth mentioning that if someone wants to ask me (I'm running this) questions
about this I'll be checking here!

~~~
kmfrk
What are the problems with South, as you point to? I think those are a big
part of the selling point for your project. :)

Additionally, could you explain how the project will not be obviated by
Django's plans to implement built-in schema migration in, I think, 1.6 or 1.7?

You taken on one of the most important aspects of managing Django services, so
I'm genuinely interested, but I just want to know what the thinking is behind
this.

~~~
andrewgodwin
I try to address the key points in the Kickstarter, but essentially South is
too fragile (it relies on exactly linear histories) and too verbose (bakes the
entire ORM state into every migration) - the main point is to fix those.

It's also worth pointing out that this IS the built-in schema migration that's
going in in 1.6 or 1.7.

~~~
mladenkovacevic
You should reiterate in this thread that you are the originator of South as
well so if anyone knows its limitations it should be you :)

------
ch0wn
I haven't used the Django ORM in over a year and I'm pretty happy with Flask +
SQLAlchemy for the moment. That said, I instantly backed the campaign when I
saw it. South has been a tremendously helpful tool I used in virtually all my
Django projects since its inception and I'm glad I can finally give something
back.

------
gregsadetsky
Hi Andrew, congrats on the campaign and reaching your goal so quickly! And
thank you for South! I've been using South for some years and it would be
impossible to imagine working on a Django project wihout it. Two quick
questions:

\- Are there any South "Good practices" books which you would recommend?
South's documentation is great, but it might be interesting to look at some
real-world project examples to learn a few new tricks.

\- I'm interested in particular in data migration development practices. How
do other developers go about developing and testing them? I've discovered that
by raising an Exception during the data migration, South will rollback the
specific migration, allowing me to iteratively test/debug the migration until
it seems bug-free and ready. Are there any other ways to do it (would running
the data migration with the "fake" option allow me full read access to the
database, but would rollback / skip any writes?).

Thank you for your time!

~~~
shabble
There was an interesting chat-log[1] someone posted on /r/Django on reddit
talking about dealing with South migration merging/conflicts, and other issues
that crop up in a multi-person project.

I'm curious about the best practices for data-migrations as well; I've seen a
few places[2][3] suggest that wrapping your fixture loading "./manage.py
loaddata <fixture>" in datamigrations is better than leaving it up to
initial_data.{yaml,json}

Edit: here's the one I was thinking of: South common pitfalls:
[http://andrewingram.net/2012/dec/common-pitfalls-django-
sout...](http://andrewingram.net/2012/dec/common-pitfalls-django-south/)

[1] <https://gist.github.com/sjl/4438002>

[2] <http://djangosnippets.org/snippets/2897/>

[3] [http://stackoverflow.com/questions/5472925/django-loading-
da...](http://stackoverflow.com/questions/5472925/django-loading-data-from-
fixture-after-backward-migration-loaddata-is-using-mo/5906258#5906258)

------
Ensorceled
When I saw this at the top of HN I was thinking "Damn it! The South guy
already has a plan for this!" ... then I see it IS the South guy asking for
funding to implement his plan!

Done!

Oh, and thanks for the email support now and then :-)

------
pjungwir
I don't know whether you'll find it useful, but when I wrote a migrations add-
on for Rails, I spent a lot of time learning how to query Postgres and MySQL
for the status of current foreign keys, indexes, and "check" constraints,
including special extensions like "where" clauses on indexes. If you need to
do something similar, perhaps you can borrow the queries from here:

    
    
        https://github.com/pjungwir/db_leftovers
    

If that seems useful and you have any questions, please let me know!

~~~
andrewgodwin
I already have written code for this several times in the past for the basic
constraint types, but not the where clauses - I'll check it out. Thanks!

------
philipkimmey
Having South take that leap into Django core will be awesome.

I'll bet one natural result of it entering core will be better shared
understanding of best practices & foibles of different backends.

Managing migrations on increasingly large tables is stressful for me primarily
due to my lack of knowledge, moreso than fragility in any piece of the puzzle.

In any case, I've thrown in my pledge - best of luck!

(If anyone has any good recommended reading for DBA-type knowledge in a devops
world, I'd love an Amazon link you think is worthwhile.)

~~~
zalew
> Managing migrations on increasingly large tables is stressful for me

large like lots of fields or large like millions of rows? if the latter, than
migrations are a PITA and you'll often find inserting a select into a
recreated new table is faster than altering the existing one. but that's
probably not something a migration tool like this should/could support without
lots of your work anyway.

~~~
philipkimmey
Rows, not columns. But more so than really large tables, I find MySQL will tip
over even on smaller tables, not very hot tables, if I want to do the work
without downtime.

(It may well just be my environment, but it is fairly standard RDS on AWS.)

~~~
jcoby
It's a limitation of MySQL unfortunately. MySQL does schema changes by
creating a new table with the new schema and copying the old data into it and
doing an atomic switch over. The table gets locked for obvious reasons.

Percona has pt-online-schema-change [1] but I have no experience with it or if
it works on RDS.

There are other solutions. The most simple is doing the table rewrite yourself
(but risking losing data). The more complex is to have a buffer/queue that can
hold requests until the migration completes but this only works in certain
specialized situations.

I'd love to find a better solution to migrations in MySQL. I've had migrations
take 15 minutes on a relatively small table (2M rows). I'm a little scared to
try pt-online-schema-change for fear of corrupting the db.

I would switch to PostgreSQL (nothing but love for that database from me) but
the ORM I'm using has issues with identifier quoting and I can't use my "user"
table with it.

1: [http://www.percona.com/doc/percona-toolkit/2.1/pt-online-
sch...](http://www.percona.com/doc/percona-toolkit/2.1/pt-online-schema-
change.html)

~~~
philipkimmey
We were using PGSql at the beginning of this project, but I was lured by the
automatic backups / restore capabilities that RDS provides and I migrated.

One of my co-workers used to pound on PGSql pretty hard in a very high traffic
environment and has nothing but good things to say, but I'm not crazy about
the idea of having to learn all of PGSql's foibles like we have MySQL.

My impression is that MySQL 5.6.x (we're on 5.5.x) should help with the schema
migration pain, but I don't have any hard evidence for that claim.

My hope is that MYSQL 5.6.x will remove a lot of the pain, but I don't have a

------
dkoch
I was hesitant to choose Python 3.3 for a new project with Django 1.5 because
of compatibility issues with add-on libraries. I've been able to get South
working from the tip of the hg repo, which was a big one for me.

Thanks for working on this, it makes me much more comfortable with my Python
3.3 decision knowing migrations will have first class support in Django. I
just sent in my contribution.

------
msandford
I'm really glad to see someone tackling this; even better that it's someone
with plenty of experience and insight. I wish more people would give me
opportunities to help fund big features like this in open source.

------
tbrock
I found myself talking to a co-worker about this being a problem last week.

We found that having this functionality as part of the core is something that
makes a framework feel significantly more agile. Virtually nobody gets their
schema exactly right the first time. Asking developers to to learn about and
use 3rd party software (if it even exists) adds a very real friction to
migrations. Altering your data model from time to time is a natural operation
that occurs during application development and should not seem out of place.

I'm very excited for this, thanks Andrew!

------
jkeesh
Andrew--

We use Django and South in our projects, and really appreciate the work you
have done. I just contributed, and am excited about this project.

My one suggestion/request: Please make it an emphasis to create awesome
documentation. I am familiar with South now at this point but often find
myself wishing more use cases and more examples were documented. I can usually
"figure it out," but great documentation saves everyone time. This seems like
a great project, so make the documentation great too!

------
justin_vanw
EDIT: By 'not possible' I mean it's not possible to delegate responsibility
for it to a tool that derives what the differences in schema are.

Schema migrations are not possible in general.

Yes, you can diff two schemas and generate statements to turn one into the
other. Adding columns works fine this way. Removing columns works fine.

Do you really need a library to add and remove columns? Not really. But
rollback, someone will say. Schema migration libraries let you undo things.
Nope. You can't rollback a drop column once it is committed, that data is
gone.

What happens when you run into issues in production that you didn't have in
your test/development environment?

Basically, automated schema migrations don't work. However, I guess if you
haven't been through it you'll just have to learn the hard way. :)

SQL gives you a very flexible environment. You never have to break existing
code to push out new features, if you are just a tiny bit careful.

~~~
haeric
Well, do you EVER actually need a library? No, mostly you can always do stuff
manually. But South makes a lot of stuff easier/less brittle, especially when
you have a) multiple servers, b) multiple developers, c) multiple
environments, and/or d) volatile data structures. It is also useful to see
what database changes a pull request introduces. And don't get me started on
data migration, which is TRULY easier in South, if you are working with
multiple branches/developers and/or have schema changes as well.

South certainly has it's warts, but that's why I'm happy to contribute to
Andrew's project.

Oh, and rollback: Depending on what the data is, it might actually be possible
to rollback. It might be an aggregate column of some sort, or moving of data
from one table another. But it's mostly not superuseful, and it's not a big
part of the awesomeness of South.

------
Kiro
Off-topic but how is money "earned" from Kickstarted taxed?

------
kevinflo
As someone who just spent a lot of time learning Django, with it being the
first web framework I've ever used, thank you! The current system resulted in
some serious headaches while learning, and these changes will definitely
remove a huge barrier to entry for people like me.

------
eyepulp
Thanks for doing this Andrew - our company uses South daily, and I've griped
about a lack of a core Django migration capability from day one. It's a
pleasure to support such a specific and useful project. Money well spent.

~~~
andrewgodwin
You're more than welcome, thanks for contributing! It's companies like yours
that help such a good ecosystem exist.

------
kawsper
How is this different from the Ruby on Rails way?

I have been away from the Django project for a long time, and I really like
the Rails way. I have supported anyway, everyone deserves a proper migration
system.

~~~
andrewgodwin
South is actually based on ActiveRecord's migrations a little bit (or rather,
it was 4 years ago) but the key difference is autodetection of changes - Rails
models are essentially declared via migrations, while Django ones are created
in a separate models.py file, so there's a need to detect when changes are
made.

~~~
damncabbage
Is it anything like DataMapper's[1] auto_upgrade!() and model definition
approach?

1\. <http://datamapper.org/why.html>

------
damon_c
This is a no brainer donation in so many ways... from the obvious of helping
you complete this which will benefit so many people directly, to helping
establish this very fine precedence!

Thanks and good luck!

------
niggler
For stuff like this, why don't people just set up their own donation site? The
target audience isn't necessarily off-put by other steps, and you can get past
the kickstarter fees.

~~~
shardling
Kickstarter fees are something like 5%, right? (And a further 5% for payment
fees, but you'd have to deal with that anyway.)

Out of the goal, that's 125 pounds. How many man-hours do you think it would
take to set up the site, test it, secure it, etc? I can't see that ever being
worth it for relatively small goals like this! And that's before you even
consider that folk are probably more likely to feel comfortable donating to a
kickstarter, since they already have an account etc.

~~~
camus
a html web page and a paypal / whatever / ... account are enough come on ...

------
d0m
First thing I thought while reading this kickstarter was "Why re-invent
South.. it's already perfect!". I changed my mind when I understood who the
author was. To be honest, I don't fully understand why the new version is
needed.. we should just include South in the Django core, problem solved. But
hey, I'm no expert in schema migration and I'm happy to help the author moves
forward with his project. The only complain I had with south was that it
wasn't installed by default ;-)

~~~
nulluk
Andrew spoke about the current situation at DjangoCon last year (2012) not
sure how much still stands out of this video but its good for a general
overview of the current state and his plans for the core integration -
<http://www.youtube.com/watch?v=gwP7zLDDdPA>

------
dbla
Great idea! First time pledging to kickstarter. I think there are a lot of
improvements to be made related to database migrations. I built DevJoist
(devjoist.com) with a similar goal in mind, but primarily for PHP developers.
Multiple people have pointed to Django south as a great solution, so I'm happy
to hear you're going to make it even better. If you ever want to chat hit me
up! Email in profile.

------
davidw
Speaking of migrations, I just got a super-simple migration system merged into
Chicago Boss!

[https://github.com/evanmiller/ChicagoBoss/blob/master/README...](https://github.com/evanmiller/ChicagoBoss/blob/master/README_DATABASE)

Well, almost...it's still missing one pull. Even if it's very, very basic, I'm
pretty happy to have _something_ rather than nothing, though.

------
d4vlx
Mybatis migrations is also a great alternative for schema migrations

<http://www.mybatis.org/migrations/index.html>

It is made by the Mybatis team but it not dependent on the Mybatis object
mapping framework or on Java. It runs from the command line.

------
maciejgryka
Wow, I haven't been following Kickstarter projects as closely before, but this
seems to be rising quite quickly!

I guess all the right ingredients are here: someone competent providing a
useful service with an achievable goal. Nicely done and good luck!

------
fvox13
So, it looks like this is going insanely well, as a kickstarter.

What do you plan to do with the excess money, if you raise more than the
highest, £7000 amount? Donate the remainder to the Django Software Foundation,
hopefully?

~~~
andrewgodwin
If it gets that high, I'll likely give some back to the DSF, yes, or put it
towards other core feature projects. Or perhaps just have the next DjangoCon
sponsored by "Schema Migrations".

~~~
shabble
Looks like it's already almost at £10k.

Can/Do you have plans to add any additional features in addition to the
existing 'stretch goals'?

I'm sure everyone has their own pet wishlist for Things I wish South Did
Better, and I wonder what an appropriate fora for suggestions might be.

Mine would be some sort of 'keyframe'/checkpointing, a bit like the full
history reset hack[1], but with more control over exactly where you split
things.

[1] [http://stackoverflow.com/questions/4625712/whats-the-
recomme...](http://stackoverflow.com/questions/4625712/whats-the-recommended-
approach-to-resetting-migration-history-using-django-sout)

------
dkhenry
I love kickstarter as much as the next guy, but this is stupid. You just threw
10% of your donations away. I don't think Kickstarter is necessarily the best
tool for this kind of fundraising.

~~~
erydo
These days when I see a project with the word "Kickstarter", it gives me an
immediate understanding of the basic funding model and the mechanics of it.
I'd be less likely to donate if it were just a "donate with paypal" link on
some blogpost.

I don't know whether that sentiment scales to a broader audience, but I
strongly suspect it does.

------
JustARandomGuy
I just backed this Kickstarter, good luck on hitting your stretch goals. While
South is decent enough, I've always thought it needed replacement or a lot of
renovation.

------
kevinastone
This looks awesome. One question, why not release it as a third party module
first and then merge it into the core later ala staticfiles?

------
neves
wow! More than 10.000 pounds. Are you creating a new way to support Open
Source software? Hope this catch on for other projects.

------
est
I hope someone can invent an SQL diff engine, e.g. input two files of creat
table sql and output the alter table sql.

------
metaphorm
this is great. i'm planning on contributing to this and will try and convince
some of my colleagues to do so as well. South is very handy and much
appreciated but undoubtedly it can be improved alot and should be part of
django core. This project will be wonderfully useful to the whole community.

------
616c
What is bizarre is I saw it, thought it was great, came to post it on this
site, and boom, was far too late.

------
Siecje
What about using SQLAlchemy Migrations?

~~~
zalew
Django ORM isn't based on SQLAlchemy's core (there were some proposals
concerning this idea [https://groups.google.com/d/msg/django-
developers/0IuJssTt8t...](https://groups.google.com/d/msg/django-
developers/0IuJssTt8tc/lL9wWuoKobUJ)), so I guess using Alembic or other SQLA
migration tool would require some frankensteining to make it work (so it
listens to the Django models.py definitions) and bringing in external
dependencies, while Andrew's proposed solution is to make it native and more
automatic.

~~~
andrewgodwin
Exactly, the value here is in linking it directly to the models and apps
paradigm of Django. South already does this, and people love it, so it seems
reasonable to continue down that path and make it even better.

------
rozap
Just wanted to say thanks for South, it makes my projects a whole lot easier.
Looking forward to this.

------
rohitn
Thank you, this was absolutely necessary. Can't wait for it to release.

------
patrickk
Some stats on the traffic sources to this page. Looks like Twitter was
important to getting this project funded:

<https://bitly.com/ZhmXuw+>

------
raverbashing
How about this:

\- I use South which is pretty good

\- I use a NoSQL db. I will lose some things, but gain a lot as well (and in
my point of view, the gains are bigger)

I'm not sure 'buying features' of software is a good way to go. (I'm not
saying it's bad)

Edit: Why the hate? It certainly is better to have an integrated solution, I'm
not saying this proposal is bad, only that there are other alternatives.

And the point is moot, donations went over the target already, hopefully this
is going to be something big!

