Ah, of course, very good point. A header-only library doesn’t have separate compile options. This is a great reason for a float-sensitive library to not be header-only, right?
It's not just about being header-only, lots of libraries which aren't header-only still have code in headers. The library may choose to put certain functions in headers for performance reasons (to let compiler inline them), or, in C++, function templates and class templates generally have to be in headers.
But yeah, it's probably a good idea to not put code which breaks under -ffast-math in headers if possible.