Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

> There's crazy things like this https://blogs.oracle.com/mysql/post/circular-replication-in-... that would be fairly difficult to run in a StatefulSet.

This isn't a great example to cite, since traditional circular replication in MySQL is a massive anti-pattern... it's incredibly fragile and pretty much has no valid use-case.

That's especially true today when other options like Galera or Group Replication are available. But even 10-15+ years ago, mentioning circular replication was a great way to give a DBA an aneurysm. (Well, I suppose that's arguably a use-case, if you profoundly dislike your company's DBAs...)



Probably a bad example, but the sentiment was: encapsulating years of DBA knowledge and complex enterprise architectures in generic k8s abstractions can be incredibly difficult.

For instance, you put a RDBMS in a StatefulSet. You figure out how to replicate between pods. Now a piece of hardware fails and a replica needs rebuilt--what does that? So you add an operator into the mix. What happens if bin logs have been purged and a replica can't just "rejoin"? Now you have to figure out how to transfer data between pods (PVs) to bootstrap a new replica. Now you probably need some sort of init container solution to keep the pod running and volume mounted while you're copying in data. Now that you have all that, how do you handle master/primary failures and replica promotion? How do you handle backups? How do you handle restores?

Once you've solved all that, how do you performance tune your database? At some point, you're probably going to start looking at kernel tuning which you'll need some additional customizations to enable (probably dedicated node per database pod, then you can either skip k8s and configure the host directly or wire up allowed sysctls through k8s).

Or, you can skip all that and require human intervention for everything, but now humans need to wade through the k8s abstractions and fix things. With that route, you break basic k8s functionality like restarting pods since it can take down your database topology


Oh yes to be clear I definitely agree with the overall sentiment. I just wouldn't ever cite circular replication as an example. Basically saying "it's hard to do [inherently flawed terrible thing] on k8s!" just detracts from the argument :)

fwiw my team eventually automated pretty much "everything" for Facebook's database fleet, and it was hundreds of thousands of lines of custom automation code (non-k8s), many years of work for 10+ extremely experienced engineers. In a k8s environment I suspect it would have been even more work.

The open source mysql k8s operators are certainly getting better with time, but there's still a lot of stuff they don't handle.


MariaDB uses a closed source operator for their SkySQL product (it's all on Kubernetes). Here's the most recent info I could find http://www.odbms.org/2019/06/on-mariadb-kubernetes-operator-...

(I used to work there)




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: