On: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

[Manabu-eo]

You can use a bit for zero or non-zero and then use bits only for providing the sign to non-zero values, for example. The sign part will be variable length but can probably be made very fast with hardware support.