Hacker News new | past | comments | ask | show | jobs | submit login

At Discourse we maintain a gem for automatic read-only when the main PostgreSQL/Redis is down: https://github.com/discourse/rails_failover



This looks interesting, thanks for sharing! Most of the apps I work with run on Heroku, so I'd likely end up reaching for their [High Availability Postgres] feature, but nice to know of an alternative if I'm not on heroku.

That said, I don't think this would serve the intended purpose here. My specific use case was upgrading the postgres version I was using. To do that we introduced a follower, prevented writes to the primary (with the approach summarized in the post), upgraded the follower, and finally promoted the follower to be the new primary. Automatic failover to the follower during this process would likely confuse things.

[High Availability Postgres]: https://devcenter.heroku.com/articles/heroku-postgres-ha


PostgreSQL upgrades are indeed one of the use cases we have with the gem, the way we do it is:

1. Have app configured to connect to both main and replica.

2. Connect to the rails console and tell the app to stay in read only mode until told otherwise.

3. Disable replication

4. Upgrade main to new PostgreSQL version

5. Tell the app to move back to read-write mode

6. Re-create the replica

This flow helped us do hundreds of PostgreSQL major version upgrades in AWS RDS this quarter when we moved from PG 10 to 12.

And this is just a plus, using the gem during normal operations means that if a Redis or PostgreSQL main explodes for any reason the app keeps serving traffic, albeit in read-only.

> Automatic failover to the follower during this process would likely confuse things.

I believe here the problem is mostly naming. The gem "failover" to read-only mode to a replica, it doesn't promote replicas to main ever. Naming is hard.


Oh that's interesting, thanks for the additional detail! I'm intrigued by what you mean when you say: "tell the app to stay in read only mode until told otherwise". Does that mean use the read-only replica connection during that time? If so, did you have to configure some error handling for that timeframe?


> Does that mean use the read-only replica connection during that time?

Yes!

https://github.com/discourse/discourse/blob/721ee3642558d960...

> If so, did you have to configure some error handling for that timeframe?

Since this was an early goal in the project, the controllers are mostly away of the read-only mode already and know how to deal with it is most places.


I've added a note at the bottom of the post as your gem seems like a great option for this sort of functionality. Thanks again for sharing!




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: