

Moving 6 Billion Messages Without Being Noticed - kemayo
http://dt.deviantart.com/journal/Moving-6-Billion-Messages-Without-Being-Noticed-285571516

======
matc
Moving databases between clusters is no easy feat. Can you share more details
on the man-hours it took to oversee and verify the migration?

I see you mention several days for the data to be copied, but how about the
time to put your solution together and verify it? Also, did recomputing
checksums take less time between passes, proportional to the size of the
remaining changes?

 _We didn't find much published material about the best way to do this,_

You may want to have a look at our live migration solution:
<http://chronicdb.com>

It goes beyond checksums: it guarantees the data move is atomic and data
consistent.

~~~
drwxrwxrwt
It took a while. cant put a number cause a lot of it was done in spurts
between other projects. altogether the project took about four months, but
that included refactoring a lot of old code to play nice with mirrored writes
and developing some framework level components. and waiting for hardware. and
waiting for innodb conversion. and building setting up proper monitoring and
alerting. future projects like that will take fraction of the time. Migration
and verification itself took about 2 weeks. Yes, passes got proportionally
quicker. first took almost a week, last few hours. we did not crank the speed
to the max though as we wanted to keep the load low and we had time.

thanks for the link, will check it out

------
kemayo
The first comment on the article expresses surprise that nothing broke when we
did this. Normally I might think that's a vote against our expected
competence, but honestly I was a bit surprised that nothing went wrong as
well.

