We started to build JuiceFS since 2016, released it as a SaaS solution in 2017. After years of improvements, we released the core of JuiceFS recently, hopefully you will find it useful.
I'm the founder of JuiceFS, would like to answer any questions here.
Hi. I'm pretty interested and excited about this project. Under "Credits" the project states:
>"The design of JuiceFS was inspired by Google File System, HDFS and MooseFS, thanks to their great work."
Would you consider writing up a design doc for JuiceFS. I would be interested to know more about what specific implementation ideas you used for each of those if any, design choices and tradeoffs made, learnings etc. It would make a great blog post. Cheers.
The whole idea was came from GFS: separate the metadata and data, load all the meta into memory, single meta server for simplicity, fixed-size chunk.
The POSIX and FUSE stuff was learned from MooseFS, but changed to use read-only chunk, and merge them together, and do compaction in background. Since most of object storage provide eventual consistency, the model work pretty well, also simplify the burden on cache eviction. In order to access object store in parallel, we divide the chunk into smaller blocks (4MB), which is also a good unit for caching.
The Hadoop SDK (not released yet) was learned from HDFS.
One key thing in the implementation is to use Redis transaction to guarantee atomicy on metadata operations, otherwise we will get into millions of random bugs.
I'm the founder of JuiceFS, would like to answer any questions here.