

How to defeat the purpose of IEEE floating point (2008) - luu
http://yosefk.com/blog/consistency-how-to-defeat-the-purpose-of-ieee-floating-point.html

======
srean
I have often found myself closer to the opposite end of the spectrum. Many
machine learning algorithms and applications (certainly not all) can tolerate
a fair degree of inaccuracy. This is often justified by the fact that the
training data is typically noisy to begin with and is a random sample from the
space it lives in. If your algorithm can be stable under those conditions it
ought to be stable under floating point jitter.

This usually means you need regularization, or smoothing, or low pass filter,
or averaging in some guise or the other (in some abstract way they are the
same thing). There are some frequently used operations in ML that are very
unstable under inaccuracies, numeric differentiation is one, just dont do it
without smoothening. Inverting matrices is another, but you better have a real
solid reason to ever invert a matrix, yes there are reasons when you do really
want the inverse.

In any case the final point is that I have had to baby talk the compilers
into, "yes, please please dont worry about it, just do these commutative (for
real numbers) operations in any order that you want". This exposes more
parallelism and other compiler optimizations. Often I would gladly take the
hit over some precision in higher order decimal places to gain speed. But this
should not be done blindly, and a smattering of numerical analysis helps
gauging when it would be safe to do so. I doubt whether numerical analysis
figures prominently in the trajectory of a graduating machine learner,
although I think it ought to.

~~~
TheLoneWolfling
One of the advantages of languages with N-ary operators.

* (a,b,c) can take advantage of those tricks, whereas the programmer can still be explicit and go * (a, * (b, c)), for example.

~~~
srean
Indeed. Do you have some specific examples in mind, I would love knowing more.
The usual suspects, for example fold_left, fold_right would typically adhere
to fixed order for floating point operations. A reason Java is a FLOPs killer
is that its standard is too strictly specified, it does not leave much room
for the runtime to take advantage of these features. Even order of argument
evaluation for functions are specified to be left to right.

------
jephir
If you're a game developer, you should look at this Floating Point Determinism
article, which shows some additional techniques for implementing deterministic
physics with floating point.

[http://gafferongames.com/networking-for-game-
programmers/flo...](http://gafferongames.com/networking-for-game-
programmers/floating-point-determinism/)

------
markrages
The quest for consistency seems misguided.

If your code changes behavior based on the timing of the various sections, you
have a bug, a race condition. The solution isn't to lock down your code and
build with only one configuration. The solution is to fix the bug.

In the same way, if the behavior of your code is dependent on the order of
operations in floating point, it is very likely that you are doing something
wrong. The solution is not careful ordering of floating point operations, but
using algorithms appropriate to floating-point (and avoiding floating point
when it isn't applicable.)

Code that has been ported to many platforms tends to be more stable, because
the heterogeneous environments have exposed bugs and flawed assumptions in the
code's construction.

~~~
idlewords
I read the article so you don't have to! The author dreams of consistent
behavior so he can get:

\- Reproducible results across platforms

\- Reproducible results with different build settings (such as debug builds or
optimization levels)

\- Useful test cases for regression testing

\- Test cases that help him find misbehavior in the optimizer

\- An improved chance of finding more subtle bugs, absent 'noise' from small
changes in floating point operations

In particular, he gives examples of how changes to nearby code can change the
output of floating point operations, due to hairiness in how registers are
allocated. You can see how this would make testing quite painful.

These all seem like worthy goals that deserve a better response than "you're
doing it wrong", along with some non-sequitur about race conditions.

~~~
markrages
Regarding race conditions, I was referring to this:

> It's not just floating point that's inconsistent across modes – it's code
> snippets with behavior undefined by the language, buggy dependence on
> timing, optimizer bugs, conditional compilation, etc.

------
rurban
Getting consistently worse results? :) He didn't even mention when matlab 5
started using his 64b only recommendation, the results were getting in-precise
and matlab was getting unusable.
[http://www.cs.berkeley.edu/~wkahan/MxMulEps.pdf](http://www.cs.berkeley.edu/~wkahan/MxMulEps.pdf)

I cannot take a floating point rant seriously without knowing Kahan's
arguments. And a crazy requirement to compare it to slow and false debugging
code does not really help.

------
lmm
This kind of thing makes me happy I work on the JVM. strictfp may be
"cowardly" but it's relatively easy to use.

~~~
Terr_
I understand that avoids some platform-capability differences, but does it
also protect from issues with execution-order or ghostly-coupling between
threads?

~~~
tomlu
I have never heard of those problems, what are they? Hardware bugs?

------
shawnhermans
Maybe I am wrong, but is the author just looking for a decimal type? Those
exist in most major languages, right?

~~~
wolf550e
The performance of decimal arithmetic on hardware not recently made by IBM
means it cannot be used in most places.

~~~
maxlybbert
The original post said he'd be happy to trade performance for consistency.

He could also use fdlibm (
[http://netlib.org/fdlibm/index.html](http://netlib.org/fdlibm/index.html) ),
at least for anything other than primitive operations. But, again, it trades
performance for consistency.

~~~
maxlybbert
This week I learned that C99 and C11 have an FP_CONTRACT pragma, that can be
set "off" for this reason ( [http://www.open-
std.org/jtc1/sc22/wg14/www/docs/n1570.pdf](http://www.open-
std.org/jtc1/sc22/wg14/www/docs/n1570.pdf) , section 6.5, paragraph 8).

