No, a NaN not being equal to itself is the correct behavior for an order relation that is a partial order, not a total order.
The bug is neither in the FP standard nor in the implementations.
The bug is in almost all programming languages, which annoyingly provide only the 6 relational operators needed for total order relations.
To deal with partial order relations, which appear not only in operations with floating-point numbers, but also in many other applications, you need 14 relational operators.
Only when they are provided by the programming language you no longer need to test whether a number is a NaN, because you can always use the appropriate relational operator, depending on what the test must really do.
The 2 relational operators "is equal to" and "is either equal to or unordered", are distinct operators, exactly like "is equal to" and "is equal to or greater" are distinct.
If you do not have distinct symbols for"is equal to" and "is either equal to or unordered", the compiler cannot know whether the test must be true or false when a NaN operand is encountered. If you do not know which of the 2 operators must be used, you must think more about it, because your program will be wrong in exceptional cases.
Specifying a stupid rule like a NaN being equal to itself and claiming that the correct behavior is a bug shows a dangerous lack of understanding of mathematics, which is not acceptable for someone designing a new number representation method (even if a single NaN value is used, it can be generated as a result of different, completely unrelated computations, and there is no meaningful way in which those results can be considered equal).
In general almost all programming languages are designed by people without experience in numerical computation, who completely neglect to include in the language the features required for the operations with floating-point numbers, resulting in an usually dreadful support for the IEEE floating-point arithmetic standard, even after almost 40 years since it became widespread.
It is not the standard which must be corrected, but the programming languages, which do not provide good access to what the hardware can do.
Also the education of the programmers is lacking because most of them are familiar only with the 6 relational operators for total orders, instead of being equally familiar with the 14 operators for partial orders.
Partial orders are encountered extremely frequently. If only the 6 relational operators are available in such cases, then they must always be preceded by a test for unordered operands. Forgetting the additional test guarantees errors. To avoid the double tests that are needed in most programming languages, it may be possible to define macros corresponding to all the 14 relational operators, but they cannot be as convenient as proper support in the language.
IMO, this only makes sense if you assume that NaNs can propagate useful information. In the real world, seeing a NaN basically just means that your math is wrong. From my perspective, the only 2 possibly useful semantics for NaN are to either just always throw an error (and not have NaN), or to say that it's equal to itself. Any other option means that == isn't an equivalence relation which pretty much completely breaks math.
On all CPUs it is possible to enable the undefined operation exception.
In that case, no NaN will ever appear, but all undefined operations will throw an exception.
So one of the behaviors that you want is available in any computing device and it should always be selected by all programmers who want to avoid testing whether an operand is a NaN. The only problem appears when you write some function in a library for which there is a policy that it should not require a certain FPU configuration to work, but it should behave correctly regardless of the FPU configuration selected by the user.
However the second behavior desired by you is not good, because that is the one that completely breaks math.
On the set of all floating-point numbers, "==" is a partial equivalence relation. It is not a total equivalence relation. Attempting to forcibly define it as a total relation leads only to inconsistent results.
On any set where there is only a partial equivalence relation and a partial order relation instead of total relations, the result of the comparison of 2 set elements can have 4 values: "==", "<", ">" and "unordered", instead of only 3 values, like for total relations.
To the 3 comparison values of total relations, there are 2^3 - 2 = 6 corresponding relational operators.
To the 4 comparison values of partial relations, there are 2^4 - 2 = 14 corresponding relational operators (e.g. "not less than" is not the same as ">=" but it is "equal, greater or unordered", so to the 6 operators of total orders, their 6 negations are added, plus other 2 operators, "unordered" and "not unordered").
Working correctly with partial relations is not really more complex than working with total relations. They are unfamiliar only because for some reason they are neglected in school, where only total relations are discussed in detail.
You keep saying that == on floating point is a partial relation, but that's only true because of the way NaN currently compares. Floating point exists as a model of the real numbers, on which equality is total, so a system modeling them should absolutely keep that if possible, which it is. It's fine to have a non-reflexive operation that behaves like == does for floats, but making that the primary equivalence relation is absolutely bonkers, since 2 NaN values with the same bit-pattern are completely indistinguishable from each other, yet compare non-equal.
Furthermore, I can't think of any reason that Inf==Inf should be true, while NaN==Nan is false. In both cases, they can be modeling different numbers (although that is true of literally any floating point number).
If there were a major advantage to making == non-reflexive, that would be one thing, but I've been programming for most of my life, (including a lot of floating point), and I've yet to come across a place where I was glad that NaN!=NaN.
I'm not sure what you're talking about by saying that this is the correct behavior for a partial order. In every definition of a partial order that I've ever seen, there is a pre-existing notion of equality (which is an equivalence relation, so reflexive -- that is, we would need NaN = Nan), and secondly the order relation in
the partial order must be reflexive (that is, we would need NaN <= NaN).
Neither equality nor inequality is reflexive, so floats with <= do not form a partial order.
Could you point me to a reference about the 14 relational operators and what this has to do with partial orders? I've never seen anything about this in theoretical mathematics.
> Could you point me to a reference about the 14 relational operators and what this has to do with partial orders? I've never seen anything about this in theoretical mathematics.
In a partial order:
There are four relations that a left side item and a right side item can have. Equal, less, greater, unordered with.
A relational operator could look for any combination of those relations, so 2^4=16 operators, except 'any' and 'none' don't make sense so there's 14.
==, <, >, <=, >=, <> make six.
Six more are "== or unordered with", "< or unordered with", etc. Negating one of the first six operators gives you one of these.
The last two are "unordered with" and its opposite "equal to or less than or greater than".
An equivalence relation can also be either total or partial, like an order relation. So the reflexivity may be true only for a subset of the complete set on which the equivalence relation is defined.
I have mentioned this also in another reply, but the following are the differences between total relations and partial relations.
On a set where total equivalence and order relations are defined, when 2 elements are compared, the result is 1 of 3 possible values: "equal", "less", "greater".
To the 3 values correspond 2^3 - 2 = 8 - 2 = 6 relational operators. In C notation: == != < >= > <=.
On a set where partial equivalence and order relations are defined, when 2 elements are compared, the result is 1 of 4 possible values: "equal", "less", "greater", "undefined" (a.k.a. "unordered").
To the 4 values correspond 2^4 - 2 = 16 - 2 = 14 relational operators.
In both cases, the 2 operators that do not exist, so they are subtracted from 8 and 16, correspond to the always true and always false operators.
For total order relational operators, "!= >= <=" are the negations of "== < >".
For partial order relational operators, that is no longer true, because e.g. the negation of "<" is no longer ">=", but "greater, equal or unordered".
So none of the 6 operators "== != < >= > <=" is the negation of another of these 6. Their 6 negations are distinct relational operators, which results in 6 + 6 = 12 relational operators.
The extra 2 operators until 14 operators are "is unordered" (i.e. isNaN for the case of numbers) and its negation.
I am sorry but I do not remember now references. However everything about partial relations are just trivial consequences of the fact that the 3 possible values of a comparison become 4 possible values.
Thanks for taking the time to explain your terminology. I've found a couple of things about partial equivalence relations (one neat one was on nlab about how the construction of the reals can either be a quotient of a subset of sequences or a subset of a quotient of sequences, and partial equivalences let you reason about the latter), but everything else seems to be in textbooks about CS-specific theory.
When you've talked about "partial orders" and "total orders" I'm surprised that you haven't disambiguated that you're talking about something completely different from what pretty much every mathematician would call a "partial order" or "total order." You could very well know about all this, but when you say "claiming that the correct behavior is a bug shows a dangerous lack of understanding of mathematics" but also don't mention that you're aware of the usual notion of a partial order, it seems a bit odd...
> that is the one that completely breaks math.
> Attempting to forcibly define it as a total relation leads only to inconsistent results.
Where's the inconsistency?
Or, a stronger question, why can't a NaN be the bottom of a total order (in the math sense) for floats? That is, make it be so that for all floats x, NaN <= x?
> an order relation that is a partial order, not a total order.
x == x (aka reflexivity) is part of the definition of a partial order[0][1]
There's case to be made that (bitwise-)distinct NaNs should compare unordered, but that's generally not what people mean when they complain about incorrect floating-point equality for NaNs.
The definition is just saying that "<=" is reflexive. When it says it's "a homogeneous relation on a set P that is reflexive, antisymmetric, and transitive," the meaning of P being a set is that it's an object described by set theory, and sets come with a reflexive, symmetric, and transitive equality relation -- the "native" one that comes from the underlying logical theory. (ZFC is built on a first-order logical theory with two relations: = and ∈.)
