So, it looks like this tries to build a model (or rather, multiple layers of models combined with something called context mixing[0]), potentially analyzing the whole file first? Is it fair to call this a very fancy entropy coder, replacing the last phase of what a traditional audio codec does (the lossless part, after quantization)?
I like the code style, this is pretty nice C. It's refreshing to see human code.
I admit I'm not a fan of the loop macros, or writing constants like 0xd80000 in decimal, or having all of that code inlined with no further explanation or git history.
But I think it's just that this file is lifting parts of the logic from PAQ8 (the inlined num_states/next_state named in a comment sure look exactly like PAQ functions translated into C, which, kudos to PAQ for having extensive comments in their code by the way).
I'm a bit disappointed that PAQ isn't mentioned in the README, and only very obliquely in the code, if that's where those functions come from. PAQ is GPLv3, at least a mention could have been nice (assuming I'm not wrong about the source of that logic).
I use the PAQ terminology as it is likely to be familiar to other compression experts. However, the lineage to PAQ is very, very limited, and many other compressors also re-use its common ideas.
I like the code style, this is pretty nice C. It's refreshing to see human code.
[0]: https://en.wikipedia.org/wiki/Context_mixing