Hacker News new | comments | show | ask | jobs | submit login

Does magic pocket use any non-standard error-correction algorithms, or just parity-style RAID5 or 6?



We use a variant on Reed-Solomon coding that's optimized for lower reconstruction cost. This is similar to Local Reconstruction Codes but a design/implementation of our own.

The data placement isn't RAID. We encode aggregated extents of data in "volumes" that are placed on a random set of storage nodes (with sufficient physical diversity and various other constraints). Each storage node might hold a few thousand volumes, but the placement for each volume is independent of the others on the disk. If one disk fails we can thus reconstruct those volumes from hundreds of other disks simultaneously, unlike in RAID where you'd be limited in IOPS and network bandwidth to a fixed set of disks in the RAID array.

That probably sounded confusing but it's a good topic for a blog post once we get around to it.


That is extremely cool. Please tell the author(s) of that system that a stranger on the internet has appreciation for that feat!


You might be interested in a paper Microsoft published a few years ago, entitled "Erasure Coding in Windows Azure Storage", which describes some similar concepts in greater detail:

http://msr-waypoint.com/en-us/um/people/yekhanin/Papers/Usen...


Hey that is cool. I worked on the first commercial disk array at Thinking Machines Corporation in 1988. We used Reed-Solomon encoding because we didn't trust that the drives would catch all the errors (they did). Good work solving some of those bandwidth constraints in rebuilding the data. John (Smitty) Hayes


We use Reed Solomon variants with local reconstruction codes.




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: