Hacker News new | past | comments | ask | show | jobs | submit login
LizardFS is Pretty Nice (potatofrom.space)
2 points by Pneumaticat 7 months ago | hide | past | web | favorite | 4 comments



"However, it doesn’t write files whole – it breaks each into 64 MB chunks and distributes them across its drives."

And what happens when the file is <= 64 MB?

Does it run on illumos / SmartOS?


According to the LizardFS whitepaper [1], the chunks can be up to 64 MB. So I believe if the file is smaller, the chunk is just as large as necessary.

Looking at some chunk files on my system backs this up:

  root@rem> ls -l | head
  total 11702880
  -rw-r----- 1 mfs mfs 33558528 Dec 12 18:17 chunk_ec_1_of_2_1_000000000000010C_00000001.mfs                                                
  -rw-r----- 1 mfs mfs    69632 Dec 12 18:12 chunk_ec_1_of_2_1_0000000000000179_00000001.mfs                                                
  -rw-r----- 1 mfs mfs    69632 Dec 12 18:11 chunk_ec_1_of_2_1_0000000000000187_00000001.mfs                                                
  -rw-r----- 1 mfs mfs    69632 Dec 12 18:26 chunk_ec_1_of_2_1_0000000000000194_00000001.mfs                                                
  -rw-r----- 1 mfs mfs    69632 Dec 12 18:06 chunk_ec_1_of_2_1_0000000000000197_00000001.mfs                                                
  -rw-r----- 1 mfs mfs    69632 Dec 12 18:36 chunk_ec_1_of_2_1_00000000000001A1_00000001.mfs                                                
  -rw-r----- 1 mfs mfs    69632 Dec 12 18:03 chunk_ec_1_of_2_1_00000000000001CE_00000001.mfs                                                
  -rw-r----- 1 mfs mfs    69632 Dec 12 18:13 chunk_ec_1_of_2_1_00000000000001D4_00000001.mfs                                                
  -rw-r----- 1 mfs mfs   135168 Dec 12 18:05 chunk_ec_1_of_2_1_0000000000000217_00000001.mfs
As for illumos/SmartOS -- maybe? The LizardFS cookbook [2] has a section for illumos, but there's nothing in it. It should probably work (it's C++), but there is probably going to have to be manual effort involved.


Now do a test with a file size of exactly 1 non-NULL byte and report back.

If the 1-byte file gets replicated across multiple disks, corrupt one of the chunks and report back.


Depends entirely on what replication goals you have setup. My simple cluster (not EC or XOR) mandates 2 copies per chunk at a minimum. My 1-byte file was unaffected by a single chunk corruption, even on the node in which the corruption originated.




Applications are open for YC Winter 2020

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: