Looks similar to OpenZL ( https://openzl.org/ )
"OpenZL takes a description of your data and builds from it a specialized compressor optimized for your specific format."
Honestly, Openzl looks even cooler! It would be cool to have it integrated with parquet and avro encoders. If I understand correctly the compressed files should be decompressable with standard tools.
3bit is a bit ridiculous. From that page I am unclear if the current model is 3 or 4bit.
If it’s 4bit… well, NVIDIA showed that a well organized model can perform almost as well as 8bit.
retroactively - create Lightweight Architecture Decision Records (ADRs) by reconstructing key decisions from the available sources, then make it a habit to maintain them for all future changes.
The easiest is to add short info in comments, and longer info in some sort of document and reference the doc in comments.
Lightweight ADRs are a good recommendation. I've put similar practices into place with teams I've worked with. Though I prefer to use the term "Technical Memo", of which some contain Architectural Decisions. Retroactive documentation is a little misaligned with the term ADR, in that it isn't really making any sort of decision. I've found the term ADR sometimes makes some team members hesitant to record the information because of that kind of misalignment.
As for retroactively discovering why, code archeology skills in the form of git blame and log, and general search skills are very helpful.
Looks similar to OpenZL ( https://openzl.org/ ) "OpenZL takes a description of your data and builds from it a specialized compressor optimized for your specific format."
reply