Hacker News new | comments | show | ask | jobs | submit login
Distributed FS: MogileFS (danga.com)
20 points by astine on Oct 30, 2009 | hide | past | web | favorite | 5 comments



Mogile relies on a central mysql db for metadata (e.g. replica locations) so it doesn't scale well at all, especially not when your workload emphasizes many small files ( because then you exhaust mysql that much faster relative to your disk capacity).


We're running about 90 terabytes of mogile storage containing 20-30 million files and haven't had any problems at all. If we do, the plan is to shard them along the same lines as our database sharding. I don't know of any other products apart from S3 that could have made our file hosting as stress free as it is. What kind of scaling problems have you had?


so your average file size is 4.5MB. that's a pretty good fit.

> I don't know of any other products apart from S3 that could have made our file hosting as stress free as it is.

I think you're right about that part. Better solutions don't come off the shelf. Yet. :)

> What kind of scaling problems have you had?

the system I wrote at Mozy needed to scale to PBs of space and billions of files, so it was obvious that the mysql part would be a problem.

more generally I think mogile is in a pretty narrow niche: typically either you really need a lot of space and files, like flickr or facebook or ..., in which case mogilefs doesn't cut it, or you have a smaller data set but you want random access to your files in which case mogilefs also doesn't cut it.


Ha sorry I didn't realize who I was talking to, yeah if you built mozy then mogile would probably hit the wall fairly early :) I work for one of the larger Japanese web companies, and the CEO has basically declared war on amazon japan (literally - they have anti-amazon tshirts, slogans etc) so S3 was not an option. Otherwise, a lot of people in this space would probably find S3 a perfect fit.


We've been using MogileFS for the last few years to host all our avatars. It's fast and powerful, and has had very few issues.

It's definitely a solid system, though was certainly overkill for our needs (if I could do it again, I would just use s3).




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: