
Ask HN: Glusterfs or ceph on Digitalocean block storage? - jaddison
I&#x27;m curious to hear of experiences running distributed filesystems like glusterfs or ceph (others too?) on Digitalocean&#x27;s block storage.<p>I&#x27;ve got a side project that will be continuously adding many 50kb to 20mb files, so easy expansion is necessary.<p>Any &quot;at-scale&quot; (ie. production, under real load) performance reports&#x2F;benchmarks appreciated.<p>Reliability, recovery from node failure, etc comments welcome.
======
taisgeal
Don't use either. I tried and failed with both. Expensive hardware and a PhD
required to go through all the config options. When I did get both to work,
they were very slow and I kept getting replication issues and bricks that
wouldn't heal.

This cost two months of my life and I was not flavur of the month with my
employer.

I eventually tried lsyncd (
[https://axkibe.github.io/lsyncd/](https://axkibe.github.io/lsyncd/)) and it
works like a charm. It took me about a week to get a decent setup going, but
it has never let me down since, can handle large numbers of small files
(similar to yours) and smaller numbers of large files. I use unison (
[https://www.howtoforge.com/tutorial/unison-file-sync-
between...](https://www.howtoforge.com/tutorial/unison-file-sync-between-two-
servers-on-debian-jessie/)) for web server pairs.

~~~
jaddison
Unless I'm misunderstanding, this isn't what I'm looking for. I'm looking for
expandable storage - the ability to add GB/TB by adding disks. Replication is
important, yes, but if I can't grow the volume, I'll just plain cry.

~~~
taisgeal
You can add storage by expanding the lsyncd droplet(remote) or by using Block
Storage(local) on your web droplets and expanding that.

I have found this to be more cost effective than anything else I have tried.

------
taisgeal
Sorry for Unison, I should point out that I use fcron for running Unison every
5 seconds. This on average syncs 15-20 20Mb files every 5 seconds without any
issue. This is similar to what you describe in terms of load.

------
deadbunny
Don't.

~~~
grzm
Please elaborate. Just "Don't" begs for more information.

