Hacker News new | comments | show | ask | jobs | submit login

We've been running large ES clusters here at Revinate for about four years now. I've found the sweet spot to be about 14-16 data nodes, plus three master-only nodes.

Right now, we're running them under OpenStack on top of our own bare metal with SAS disks. It works well but I have been working on a plan to migrate them to live under Kubernetes like the rest of our infrastructure. I think the answer is to put them in StatefulSets with local hostPath volumes on SSD.

We use CoreOS for our Kube cluster and are huge believers in it so I will definitely be leveraging that for ES. One thing I have done for my Cassandra clusters is to write a custom health check running in a container that monitors Cassandra's health and locks the cluster with locksmithctl to prevent reboots when the database is unhealthy. This will be easy to translate to ES. Correspondingly, I will add a curl call to the ES unit file to disable shard reallocation when the service is shut down during a reboot. This prevents costly reallocation of shards when you know that a node will be back online shortly.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact