
Why I find IEEE 754 frustrating - plesner
http://h14s.p5r.org/2014/11/why-i-find-ieee-754-frustrating.html
======
StefanKarpinski
Good essay. The dynamic scoping / mutable global state issue, especially with
respect to rounding modes, is a total nightmare. We've been trying our hardest
to provide great support for rounding modes in Julia and it just doesn't work
well. The trouble is that there are functions that need normal rounding
regardless of the current rounding mode while other operations can safely
respect the current rounding mode. There are still other functions for which
neither is appropriate. A software-implemented sin function, for example, may
completely break if used with an unexpected rounding mode, so you don't want
that. But you also don't want to just completely ignore the rounding mode
either. What you really want is a sin function that returns a correct result,
rounded according to the rounding mode. But that means you really need five
variants of the sin function and you need to dynamically dispatch between them
based on the current rounding mode – i.e. global mutable state – which is, of
course, quite bad for performance, not to mention the explosion of complexity.
I've come to believe that the rounding mode concept as designed in IEEE 754
was only ever feasible to use when you were writing your own assembly code.

The essay is not quite right about the motivation for rounding modes – they
are not intended to support interval arithmetic. When we got the chance to ask
him about it, Kahan was quite negative about interval arithmetic, noting that
intervals tend to grow exponentially large as your computation progresses.
Instead, the motivation he suggested was that you could run a computation in
all the rounding modes, and if the results in all of them were reasonably
close, you could be fairly certain that the result was accurate. Of course, at
the higher level the essay is exactly right: the fact that this motivation
isn't explained in the IEEE 754 spec is precisely what allowed this
misunderstanding about the motivation to occur at all.

~~~
phkahler
With all the state and modes and such, is there a mode that just does perhaps:
round to even, clamp to infinity, never produce NaN, and never trigger an
exception? Because that's basically what a lot of people want - do the math
and handle the extremes in reasonable ways without throwing exceptions.

~~~
stephencanon
In your "never produce NaN" arithmetic, what do you want sqrt(-1) to be, and
why is that a better answer than NaN?

~~~
lmm
I want it to throw an exception - a language-level exception - and I'm fine
with sacrificing a bit of performance (i.e. checking the flags after every
operation) to do so. This is a better answer because it means I see the error
where it actually happened, rather than getting a NaN out the end of my
calculation and having no idea which particular computation it came from.

~~~
stephencanon
This is what we had _before_ IEEE-754. Instead of having a closed arithmetic
system, exceptional conditions caused a trap (the "language-level exception"
of its day). It was a terrible situation, and the direct cause of several
"software-caused disasters" that you may have learned about in an engineering
class.

~~~
TheLoneWolfling
Having division by zero being an _unchecked_ exception is a terrible idea, as
you say.

But that's not what I want. I want a paranoid language. I want a language
where potential division by zero is a _checked_ exception. One where "a = b /
c" won't even compile if c might be 0. One that won't compile if it can find
an example of an input to a function where an assertion fires. I want one
where there is no such thing as an unchecked exception. Or rather, one where
you can explicitly override checked exceptions to be the equivalent of (read:
syntactic sugar for) "except: print(readable exception trace); exit(<code>)"
\- but you need to explicitly override it to do so.

Would it be a pain to write in? Yes. But at the same time there's a lot of
software that would be best written in this manner. A language where the
language itself forces you to be paranoid.

~~~
pseudonom-
> One where "a = b / c" won't even compile if c might be 0.

Dependently typed languages can provide this.

~~~
TheLoneWolfling
Can you give an example? I have yet to run into a language that doesn't
require a proof of correctness, but will just attempt to find a
counterexample.

~~~
lmm
You need to provide a proof, but in e.g. Idris the language gives you the
tools to make that proof quite easy.

~~~
TheLoneWolfling
[Citation needed]

Again: I am looking for a language that _doesn 't_ require you to provide a
proof. I'm looking for a language that is a "logical extension" of what
currently is available - that is, I am looking for a language that will
attempt to find a counterexample on compilation and will bail if it can.

~~~
lmm
But non-exhaustively? That exists already - plenty of languages will warn or
error if they can tell you're dividing by zero, but don't catch every possible
case.

Any working program will in some sense be a proof, by Curry-Howard. So I think
asking to not have to provide a proof is backwards; what you want is a
language that makes it easy to express the program and manipulate it as a
proof.

------
jws
When confronted by closed standards with open drafts I generally just
implement the last draft. I might buy the closed spec to check, but my code
comments and documentation will all reference the draft since that is what
people can read.

You might tuck a copy of these into your personal library in case IEEE purges
them.

IEEE754 base document:
[http://www.validlab.com/754R/standards/754.pdf](http://www.validlab.com/754R/standards/754.pdf)

IEEE754r draft:
[http://www.validlab.com/754R/drafts/archive/2006-10-04.pdf](http://www.validlab.com/754R/drafts/archive/2006-10-04.pdf)

Perhaps someone who has seen both versions can comment on how close these are
to the closed versions.

~~~
stephencanon
The 754 (2008) draft you link to is reasonably close to the final standard in
content, but there are definitely a number of significant changes that came in
the two years between that draft and final publication (I'm "STC" from the
change history, for context).

It's not hard to find copies of the final standard online, but the
availability issue is definitely something that the committee is aware of.

------
ekimekim
Python's Decimal module (though not its floats, for some reason) has, IMO, a
pretty good implementation of these features.

[https://docs.python.org/2/library/decimal.html#context-
objec...](https://docs.python.org/2/library/decimal.html#context-objects)

Basically, it encapsulates attributes and status flags into a thread-local
"context" which you get/set through normal function calls. There's also the
helpful "with" syntax which allows you to say "run this code block (and
anything it calls, etc) with this context instead of the current one, then
restore the current one on exit".

A sibling comment talked about a sin() example where you want to use an
explicit rounding mode for your calculations, then apply the global rounding
mode to the result. Under this paradigm it would look something like:

    
    
        with MySpecialContext(settings, etc) as ctx:
          calculations...
          check status flags in ctx
          get result
        round result # this uses the parent context
        return

------
colomon
Looking at the broader issues here, I know I've had the same sort of problems
with the ISO 10303 (STEP) standard [1]. Overall, it consists of dozens of
$100+ books, most of which amount to little more than a long list of
descriptions of the classes that can be used to transmit CAD data. Everything
is in turgid bureaucratese. I've seen nowhere in the standard with any sort of
high level description of how those classes are intended to be used, no
motivation for why things are the way they are. There are some recommended
practices documents, but they mostly seem to cover fringe areas like how to
handle colors rather than core areas like the preferred approach for handling
CAD geometric data.

It just seems so odd to spend so much effort to develop a public standard,
then make it expensive and hard to use. Doesn't that defeat the entire point
of having a standard?

[1]
[http://en.wikipedia.org/wiki/ISO_10303](http://en.wikipedia.org/wiki/ISO_10303)

------
stephencanon
This essay raises a lot of concerns about global state, especially with regard
to rounding and flags (or exceptions). That's a common misconception, but
nothing in IEEE-754 requires that this state be global. In the C language
bindings, for example, dynamic rounding mode and status flags have thread
scope.

In fact, dynamic rounding modes are not required _at all_ by IEEE-754 (2008).
The revised standard requires that languages provide a means to specify static
rounding at "block" scope, which is a language-defined syntactic unit.

> (4.1) the implementation shall provide language-defined means, such as
> compiler directives, to specify a constant value for the attribute parameter
> for all standard operations in a block; the scope of the attribute value is
> the block with which it is associated.

> (2.1.7) block: A language-defined syntactic unit for which a user can
> specify attributes.

You can take "block" to mean whatever makes sense for your language: it could
be a single arithmetic operation[1] or it could be the whole program (though
it's more useful if it isn't). It is recommended, but not required, that
languages provide a means to access "dynamic" rounding modes as well, which
correspond roughly to what most people think of when they think of IEEE-754
rounding modes as widely implemented, but again a huge amount of flexibility
is left to the languages to choose exactly what scope and precedence rules
make sense for their language.

[1] efficient hardware support for such fine-grained static rounding is still
somewhat lacking in the commodity CPU world. On GPUs and some other compute
devices, it is quite natural (and "dynamic" rounding is sometimes quite a
hassle). AVX-512 will bring support for per-instruction static rounding to
mainstream CPUs.

When we look at flags, the situation is much the same. Languages completely
specify the scope of flags. There is no requirement of mutable global state.
For example:

> (7.1) Language standards should specify defaults in the absence of any
> explicit user specification, governing ... whether flags raised in invoked
> functions raise flags in invoking functions.

Like with rounding, current commodity CPUs make it _easier_ to provide flags
with thread scope, but IEEE-754 does not require it. Commodity hardware works
the way it does because mainstream languages work that way. If a different
model makes sense for your language, do that.

Finally, the concern about "exceptions" is entirely misplaced. "Exception" in
IEEE-754 simply means "an event that occurs when an operation on some
particular operands has no outcome suitable for every reasonable application,"
which is a rather different meaning than the way "exception" is understood in
colloquial PL usage. Under default exception handling, which is all that
IEEE-754 requires implementations to support, all that needs to happen in the
case of an exception is for the implementation to raise the corresponding
flag, the scope of which is (as previously discussed) up to the language to
specify.

I would encourage you to direct questions like these about the spec to
committee members. If you work for a big company, a few members probably work
with you. If you don't, most committee members are happy to answer questions,
even from people they don't know.

The concerns about access to the spec itself and to the minutes are well-
placed, and definitely something that the committee is aware of. (But mostly
out of the committee's hands; it's up to IEEE to set pricing. Send them your
comments!)

~~~
plesner
> nothing in IEEE-754 requires that this state be global. In the C language
> bindings, for example, dynamic rounding mode and status flags have thread
> scope.

My example, RegExp captures in JavaScript, also have thread scope. Thread
local global state is still not good. What makes the flags global is that
access to them is provided implicitly and ubiquitously, independent of scope.
All the operations as well as the flag functions like saveAllFlags are given
read and/or write access and none of them take arguments that control the
scope of the flags they're manipulating. They get pulled from thin air. This
is problematic.

I deliberately never say that the rounding flags are global, my strawmen are
that they're either lexically or dynamically scoped, both of which are
problematic.

> You can take "block" to mean whatever makes sense for your language: it
> could be a single arithmetic operation[1] or it could be the whole program
> (though it's more useful if it isn't).

What I'm arguing is that there is no interpretation of "block" that yields a
satisfying result. I didn't consider applying it to individual operations
because the whole motivation for the rounding mode mechanism is to keep the
same mode is in effect across multiple operations. The lack of hardware
support suggests the same thing.

Maybe you can give an example of how you see this working where the attributes
are more tightly scoped than per-thread?

> Finally, the concern about "exceptions" is entirely misplaced. "Exception"
> in IEEE-754 simply means "an event that occurs when an operation on some
> particular operands has no outcome suitable for every reasonable
> application," which is a rather different meaning than the way "exception"
> is understood in colloquial PL usage.

Fair enough, though that introduces a new problem: how do you implement fp-
exceptions if the're not language level exceptions? But maybe if a reasonable
solution can be found for the rounding flags something like that would work
for the exceptions too.

> I would encourage you to direct questions like these about the spec to
> committee members. If you work for a big company, a few members probably
> work with you. If you don't, most committee members are happy to answer
> questions, even from people they don't know.

I did, I raised some of these issues, including the licensing issue, with
David Bindel a year ago.

~~~
Sanddancer
For floating point exceptions, I'd say go with a mechanism like C's fenv.h
[1]. It's out of the way, and handles rounding flags too. For if you're
passing attributes around in different threads, my first thought would be
(ab)using a language's type/object system to make sure that all your threads
are using the same assumptions for floating point behavior; something like
classes named FloatUp, FloatDown, FloatZero, FloatClose.

[http://pubs.opengroup.org/onlinepubs/009695399/basedefs/fenv...](http://pubs.opengroup.org/onlinepubs/009695399/basedefs/fenv.h.html)

~~~
sunfish
fenv.h appears to be implemented as a simple library, but it actually requires
major compiler support. In order to make it work, the C standard added #pragma
STDC FENV_ACCESS to the language itself, which is described in the link you
posted. Compilers such as clang and GCC haven't implemented that pragma or the
features it entails yet, so fenv.h doesn't work reliably in practice.

The underlying problem is that programming languages and compilers want to
model something like "add" as an operation which has two inputs, one output,
and no side effects. The need to support flags conflicts with this. Declaring
that flags don't cross function boundaries or any other boundaries doesn't
make the problem go away.

------
sunfish
The mutable global state problem is hard. And, its implications are woven
throughout the spec. For example, pow(0, nan) is 0, not nan, because that's
slightly more convenient in some cases, and it's assumed that you can always
check the Invalid flag to see if any nans were produced and swallowed.

At the same time though, it's not designed that way accidentally or in
ignorance of the problems it creates. IEEE-754 knew that programming languages
wouldn't be very happy about global state, and chose to keep it because they
believed it was still the best approach. In many other areas, IEEE-754 pushed
against people who said it would be too hard to implement, and in retrospect
they ended up being right in many cases. It's tempting to wonder if global
mutable state really was too much of a tradeoff though, in retrospect.

~~~
dbaupp
> they ended up being right in many cases

For clarity, is 'they' referring to IEEE-754 or the people who said it was too
hard to implement?

~~~
sunfish
I meant that IEEE-754 got a lot of stuff right, from our current perspective.

------
emiliobumachar
Very actionable suggestions. Hopefully some of the ieee folks are reading.

------
dosshell
Can't we make an opensource description of the IEEE754 which resolves the
issues? While the spec doesn't describe the same thing it should be no legal
problems - right?

~~~
jordigh
How are you going to create a free version of the spec without making it
derivative work? If you read the original and you paraphrase it, that's
derivative work, so it would have the same copyright as the original.

------
Sanddancer
This is a rather poor rant. There's pretty much an easy answer to everything
he describes that's been done in the academic world, and in other programming
languages for decades, and almost certainly in his own programming language
for similar problems.

    
    
        Can I copy it into my own spec or would that be 
        copyright infringement? Can I write something myself  
        that means the same? Or do I have to link to the spec 
        and require my users to pay to read it? 
    

Giving a summary is allowed even in the most draconian of interpretations of
copyright. More loosely, pulling a quote of a few lines is standard for any
sort of academic exercise; essays on novels certainly don't have you pull out
the novel when they want to quote a bit of text.

    
    
        What if your language doesn’t have exceptions? What 
        does it mean to provide a status flag? 
    

You certainly have places elsewhere in your language for handling errors. How
do you handle integer divide by zero? How do you handle timeouts when doing
networking? How do you handle disk full errors? I hope you don't go stomping
along after an error in those cases.

    
    
        This may have made sense at the time of the original 
        spec in 1985 but it’s mutable global state – something 
        that has long been recognized as problematic and which 
        languages are increasingly moving away from.
    

This is because statelessness is a leaky abstraction. Your machine's saving to
disk, it's allocating memory for other things, it's pulling in stuff and
shooting stuff over the network. Floating point has lots of knobs that you
need to turn because there are various rounding rules for various problem
domains. You've almost certainly got a type system for your language, perhaps
your answer is to create, or have a way to create, a floatRoundUp type for
when a user needs to go beyond your default rules. In a pure functional
definition, I think one way to consider the problem is to think of a floating
point type as the actual number and the state registers. If you don't care,
then just use the number part, if the state register matters, put those in
too. Saving flags means that you can get repeatability. With a given set of
flags, you'll always get the same answer.

    
    
        Program block? Attribute specification? Nothing else in 
        my language works this way. How do I even apply this in 
        practice? If attributes are lexically scoped then that 
        would mean rounding modes only apply to code textually 
        within the scope – if you call out to a function its 
        operations will be unaffected. If on the other hand 
        attributes are dynamically scoped then it works with 
        the functions you call – but dynamically scoped state 
        of this flavor is really uncommon these days, most 
        likely nothing else in your language behaves this way.
    

There are two ways of handling this problem. Decide for the user, or expose
the functionality needed to provide scoped floating point. The most "pure" way
sounds like using the aforementioned type system to give defaults, but have a
way for the user to tweak the rules as needed. You're the designer, one of
your jobs is to make these sorts of decisions for the user. Things like your
i/o library probably have similar problems, I don't see why floating point is
all that much different.

~~~
wpietri
A quick explanation of my downvote: You seem to be mistaking "I know the
answer" for "everybody should know the answer".

You could have written basically the same reply, but starting out with, "Hey,
those are good questions. Because of my time in the academic world, I happen
to know some of the answers. Let me see if I can help."

But instead you had to be a dick. That's undeniably fun, but contempt for
people asking reasonable questions doesn't make them smarter; it just stops
them from asking questions.

~~~
Sanddancer
I think what set me off was certain statements that he was coming from a
position of authority, like that he was a language design guy. At the same
time, he didn't give any statements to back up how deeply he's researched the
issues. For example, there's no discussion of C's fenv.h, or how fortran's
IEEE intrinsics work. Add to that the jabs on the age of the specification,
and it just ended up giving the feeling of being a much more shallow piece
than it could have been; it just felt like he was saying, "I don't know what
to do. Internet, decide for me."

~~~
wpietri
I thought it was mainly a post about how he personally found this standard
frustrating. And I started getting that impression about the time I read the
title, "Why I find IEEE 754 frustrating".

The guy has a reasonable background as a language guy. He was trying to deal
with the spec; he's allowed to opine on it.

That you felt things? Those are _your_ feelings. That you think he should be
obliged to address your personal concerns preemptively? That's your problem,
not his. If you want to know how deeply he's researched the issues, you could
say, "Hey, have you looked at C's fenv.h?" Rather than just assuming the
answer that lets you be a dick.

------
GFK_of_xmaspast
$88 is not a lot of money for a professional software person, especially if
you can get your employer to pay for it.

