

CopyCat: Protocol-agnostic implementation of the Raft consensus algorithm - kuujo
http://github.com/kuujo/copycat

======
mavelikara
I am writing a program which has to durably store large number of small files.
My current implementation stores these on the local disk as files in a
particular folder structure. I am considering designing some approach to make
this program run on a cluster of machines for high-availability, fault-
tolerance & scalability reasons.

One approach I can think of is to run a distributed document database and use
that for storage. I don't need most features of such database products in the
foreseeable future, so I fear that they will add operational overhead for not
much benefit.

Another approach I can think of is to run my processing nodes against a
network file system, and rely on that to do the replication.

Yet another approach I am considering is to use something like CopyCat to
implement the file replication in my application code. Is this a good use case
for CopyCat?

~~~
ww520
CopyCat looks really good for HA and fault-tolerance. Not so sure about its
scalability since all writes go through the single leader node. It's more
appropriate for the use of maintaining metadata of a distributed system,
rather than maintaining the data themselves.

For your requirement, can you use S3 or something similar?

~~~
mavelikara
Thanks! The software I am writing needs to be (easily) installed at the
customer site. So, S3 won't work for me. Are there any open source S3 clones
that are easy to setup?

~~~
michaelmior
If you mean that you need something that supports the S3 API, you could try
S3Proxy[0]. It basically presents the S3 API over a variety of different
storage backends.

[0]
[https://github.com/andrewgaul/s3proxy](https://github.com/andrewgaul/s3proxy)

------
michaelmior
This looks fantastic! As a DB researcher, I've found in the past I need some
type of consensus algorithm for a system I'm building, but I don't want to
spend a lot of time implementing something myself. I can see this being very
useful in academia for scenarios like this.

