On: In Defense of Pure 16-Bit Floating-Point Neural Networks

[bee_rider]

If you just want the mantissa, why not use integers?

[gumby → bee_rider]

Yes, saturated fixed point or integer math. And I might have been hasty in saying you can’t do this with modern commodity hardware (here’s a paper: https://proceedings.mlsys.org/paper/2022/file/14bfa6bb14875e...)