

An Interview with the Old Man of Floating-Point (1998) - Hachiro
http://www.eecs.berkeley.edu/~wkahan/ieee754status/754story.html

======
Someone
For those looking for more details visit dr Kahan home page at
[http://www.cs.berkeley.edu/~wkahan/](http://www.cs.berkeley.edu/~wkahan/)

For example:
[http://www.cs.berkeley.edu/~wkahan/ARITH_17U.pdf](http://www.cs.berkeley.edu/~wkahan/ARITH_17U.pdf)
explains what gradual underflow is.

And for Excel haters:
[http://www.cs.berkeley.edu/~wkahan/Mind1ess.pdf](http://www.cs.berkeley.edu/~wkahan/Mind1ess.pdf)
:-)

~~~
thristian
Don't forget the floating-point walkthrough from the ever-lovable Ridiculous
Fish:

[http://ridiculousfish.com/blog/posts/float.html](http://ridiculousfish.com/blog/posts/float.html)

------
StefanKarpinski
This is always a great read. I think most programmers fail to realize what a
hellish nightmare floating-point arithmetic was and still could be if not for
the amazing work of Kahan and his colleagues and a lot of luck that allowed
this standardization process to be one of the most successful ones ever.

~~~
thristian
I have heard programmers complain about the hellish nightmare of IEEE754
floating-point arithmetic. I'd heard it was a compromise between reliability
and implementability, but I hadn't know until know how far it pushed towards
reliability, or how much more of a hellish nightmare pre-754 floating point
must have been.

~~~
stephencanon
754 arithmetic is not really hellish; it's quite easy to use correctly _if one
can be bothered to learn how it works_. The trouble that most programmers run
into is that they can't be bothered, and assume that it works just like
arithmetic over the reals (which they typically don't really understand
either).

------
cliffbean
This part is interesting:

> "If better precision, directed roundings, and exception handling
> capabilities latent in hardware but linguistically inaccessible remain
> unused, their mathematically provable necessity will not save them from
> atrophy. The new C9X proposal before ANSI X3J11 is a fragile attempt to
> insinuate their support into the C and C++ language standards. It deserves
> the informed consideration of the programmers it tries to serve, not
> indifference."

And it's now 2013 and directed roundings and exception handling largely remain
linguistically inaccessible. C99 did add them, but they remain fragile. And
most other languages haven't added them. It seems that the programming
community has largely settled on indifference.

~~~
fdej
This is sadly true. If you want to do reliable floating-point computations
beyond +, * and / with round-to-nearest (and even that isn't easy, since for
example the use of x87 instructions can result in the wrong precision), the
only portable solution is still to use a software implementation such as MPFR.
Which is a shame because you're letting all that fast silicon go to waste.

~~~
stephencanon
Even if all the basic operations were fully specified and implemented
identically on all architectures, you would still have a noticeable
performance vs. reproducibility tradeoff across architectures for "real
computation". For example, different architectures have different numbers of
registers and different cache hierarchies, which favors different access
patterns in a fast matrix multiplication; thus linear algebra routines that
attempt to be as fast as possible will necessarily group the arithmetic
operations differently, which leads to different results even if individual
arithmetic operations are guaranteed to be identical.

It's not insurmountable, but it's a thornier problem than it looks at first.

------
oofabz
I am familiar with Kahan from his excellent low-error floating point summation
algorithm:

[http://en.wikipedia.org/wiki/Kahan_summation_algorithm](http://en.wikipedia.org/wiki/Kahan_summation_algorithm)

~~~
StefanKarpinski
Compensated summation (aka Kahan summation) is a great algorithm.
Interestingly, pairwise summation [1] is almost as accurate as the compensated
algorithm, while being nearly as fast as naïve linear scanning summation. This
makes it a great default summation algorithm since it does so well in terms of
both accuracy and performance.

[1]
[https://en.wikipedia.org/wiki/Pairwise_summation](https://en.wikipedia.org/wiki/Pairwise_summation)

------
saljam
Almost every year, I reread the famous Goldberg paper, What Every Computer
Scientist Should Know About Floating Point Arithmetic[1]. And with every time
I'm more astonished at how the hell can anyone do anything nontrivial with
floats and _trust_ it.

[1]
[http://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.ht...](http://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html)

~~~
StefanKarpinski
This is the reason that support for changing rounding modes is so incredibly
important for numerical reliability. One of the best ways to assess if your
algorithm is numerically stable is to try it in all the rounding modes. If all
of the answers are similar, you're probably doing ok. If they're wildly
different, you may be computing complete nonsense.

------
jejones3141
Remember the bad old days of IBM big iron base _sixteen_ floating-point, which
could gratuitously cost you up to three significant bits? Thank goodness those
are gone.

------
aidenn0
And to this day, most FPUs I've worked with don't support gradual underflow in
hardware. They can only get 754 compliance with OS assistance.

~~~
nn3
What do you mean? Intel/AMD FPUs suport it in microcode, but it is somewhat
slow. But the OS is not involved.

~~~
aidenn0
Good thing I hardly ever work with x86 (Power and ARM primarily)

~~~
stephencanon
ARM VFP supports gradual underflow. NEON also supports it on arm64. Most Power
implementations that aren't Cell BE do as well.

