Ask HN: What is the great (or better) option to replicate storage?

[seungwoolee518]

Hello.

I (or we're) am running a Kubernetes Cluster.

It's total of 11 worker nodes. each node have a 2 HDDs and mostly 2 of NVMe Storage. on Networking side, just 1 10Gbps Ethernet port is attached to each nodes.

However, it's pretty hard to determine which distributed storage solution (or way) is fit to our situation.

We've tried a various ways to achieve.

- Longhorn: Great. Simple to Setup. Sometimes it goes to Read-Only State. (because changed block couldn't synced to other replicated storage, especially Superblock)

- OpenEBS (copied version of Longhorn iSCSI version): Same as Longhorn.

- Ceph: Great. No issues at all. But really stressful when HDDs are jammed with S.M.A.R.T. Error.

How do you manage a storage cluster like ours?

Someone suggests to "don't replicate unless you need to" or "use NFS to Shard disks" but IDK it's the best way to keep the data on failure.

Thank you in advance.

Lee.

[blinded]

What are your requirements? Sounds like you want to do your own storage infra? What stateful apps are you running?

[seungwoolee518 → blinded]

> What stateful apps are you running?

- Databases. (Usually PostgreSQL, seems like it's pretty easy to setup to replicate between PostgreSQL instances.)

- Qdrant, or milvus. (seems like they're support replication between instances, we've *accidentally* running a replicated DB on Single Ceph Cluster)

- Some user generated contents. (usually from Jupyter Environment)

> Sounds like you want to do your own storage infra?

We don't actually have a "storage infra". it's just a bunch (I really don't think 2 HDD is a part of "Bunch") of disks attached to each machines (or node).

[blinded → seungwoolee518]

So it sounds like you have state replication at the application at the service level for everything but Jupyter? I'm not familiar with Jupyter but would it be possible to use object storage? Does it support backups? Can you store those backups off site?