Hacker News new | comments | show | ask | jobs | submit login

We've taken a very similar approach when migrating data from one DB to another (MySql to Redis in our case, but the principle should apply to any databases). We split it into 4 phases:

* Off - Data written only to MySql (starting state)

* Secondary - Data written to both MySql and Redis, and MySql shall be the source of truth.

* Primary - Data written to both MySql and Redis, and Redis shall be the source of truth.

* Exclusive - Data written exclusively to Redis.

As mentioned in the article, the Secondary phase allowed some time for the new database to be populated. And the distinction between Primary and Secondary phases gave us a rollback option if something went wrong.




The difficulty here is what happens when one of the databases is not available temporarily (network error, etc.) You cannot have a "transaction" cover writes to both systems so you either have to manually undo one of the writes or risk the two systems getting out of sync.


This is typically solved (in my experience) using a reconciliation process using transaction GUIDs along with backfill from non-source of truth in the event the data isn't found in the source of truth. As long as a transaction made it into one of your data stores, consistency isn't lost (and if writes failed to both data stores, alarms should go off).




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: