
Do Developers Understand IEEE Floating Point? [pdf] - matt_d
http://pdinda.org/Papers/ipdps18.pdf
======
mysterydip
If you're like me and struggled to understand the traditional textbook
floating point formula explanation, Fabien Sanglard (of doom/wolf3d black book
fame) wrote a page that finally made sense for me:
[http://fabiensanglard.net/floating_point_visually_explained/...](http://fabiensanglard.net/floating_point_visually_explained/index.php)

~~~
Waterluvian
Can I ask an embarassing question to verify I'm not misunderstanding that
page? Does this mean that floats are less precise the larger they are? You
have as many bits to describe the "position of the number" between 0-1 as you
do between 2048-4096

~~~
Sharlin
This is precisely what the term _floating point_ means. The decimal point
"floats" right or left depending on magnitude due to the mantissa-exponent
representation. The opposite is _fixed point_ where you have a fixed number of
bits reserved for the whole and fractional parts. A 32-bit floating point can
represent a vastly greater range of magnitudes than a 32-bit fixed point, with
a vastly greater precision for values near zero, but at the expense of
absolute precision at large magnitudes.

~~~
kps
At the computer floating point was developed, people understood it implicitly,
because it was just an automated version of their current tools: log tables
and slide rules.

And likewise they knew better than to use floating-point where it doesn't
work.

~~~
theoh
There are a couple of good images on this page which make clear the fact that
floating point position 'degrades' in discrete steps as the number gets
larger. It's not quite the same as counting in logarithmic terms, which is
shown for comparison.

[https://www.americanscientist.org/article/the-higher-
arithme...](https://www.americanscientist.org/article/the-higher-arithmetic)

------
kbumsik
Although understanding floating point is important, it is often ignored in
Data Science schools.

The title reminds me a friend of mine who attends a data science master course
without computer background. He asked me for help for his numpy homework on
Bayesian Probability theory. The number he got was so small that he got weird
result. I immedieatly thought it was because float64 couldn't cover that
number so I told him to use longdouble dtype (it's 80 bits in Intel I think)
and the problem solved.

Later he told me even the professor didn't expect such case when announcing
the homework and many students couldn't finish the hw on time because of this.

~~~
mehrdadn
Would the log-sum-exp trick have helped by any chance?
[https://en.wikipedia.org/wiki/LogSumExp](https://en.wikipedia.org/wiki/LogSumExp)

~~~
kbumsik
I didn't know this (I'm not in ML/Data Science) but that looks very helpful. I
looked through his learning materials but I have never seen this one though.

~~~
mehrdadn
Yeah I only happened to know because of ancourse I took that taught some ML. I
mentioned it because, later on, I even came across a situation where it
would've been directly applicable, but for some reason the thought that it
might solve the problem escaped me at that moment, so I asked and someone else
had to remind me of that fact... so I feel it's probably not the easiest thing
to recognize when it's happening.

------
abductee_hg
according to this document the people who implemented java did not:

"How Java’s Floating-Point Hurts Everyone Everywhere"

[https://people.eecs.berkeley.edu/~wkahan/JAVAhurt.pdf](https://people.eecs.berkeley.edu/~wkahan/JAVAhurt.pdf)

p.s.: it's by William Kahan, who the primary architect behind the IEEE
754-1985 standard for floating-point computation (and its radix-independent
follow-on, IEEE 854)

~~~
nraynaud
no language has a good support for IEEE754. The only doable thing is using
some weird macros in C/C++, but the rest of the languages ignore the issue
altogether. And LLVM too.

~~~
ChrisLomont
C/C++ is not required to use ieee754 as it's float format, so your code would
always run under that cloud.

Fortran, D, Factor, and SBCL likely have better support.

But anything where a compiler can reorder things is suspect, if you need exact
behavior, since even simple register spilling causes trouble (Intel), among
other things compilers may do.

~~~
Asooka
AFAIK C and C++ compilers are _not allowed_ to re-order floating-point
arithmetic precisely for that reason. Also, given that all cpus support
SSE/Neon/etc. now and I haven't seen a compiler use the 387 FPU, I suspect
register spilling shouldn't be an issue these days. Of course, legacy/exotic
platforms will deviate from the norm in fun ways.

~~~
ChrisLomont
>AFAIK C and C++ compilers are not allowed to re-order floating-point
arithmetic precisely for that reason.

You don't have to reorder to make bugs. Simply having a compiler replace
(a-b)+b with a is a bug. Simply flushing an internal 80-bit floating register
to a 64 bit memory place, then loading it, causes differences in computations.

Note the C++ spec specifically states it does not require an underlying
representation or even any certain level of accuracy (6.7.1.8). So many of
these bugs are allowed.

The spec does not even require that compile time evaluation of an expression
must match a runtime evaluation of the same expression (8.20.6). When you hit
that in code you're going to be surprised.

C/C++ is not required to round-trip floats/doubles when printed and parsed.
stream formatting rounding modes are allowed (and are) not exact, leading to
different behavior on formatting the same number on different systems.

Dealing with these led to compiler implementation defined behavior added due
to market pressures, making C/C++ numerics less of a mess, not anything in the
spec.

C/C++ has notoriously been bad at all this, which is why the history of it is
littered with such bugs, errors, undefined behaviors, and dozens of compiler
switches to try and mitigate such behavior.

For example, many compilers notoriously replaced high-precision Kahan
summation with low-precision regular summation because optimizations tell the
compiler that such things are allowed.

The state of the art in C/C++ is now passable, but it is still flawed, and has
been historically very bad.

And all this met the C/C++ spec of the time. Because if the underlying
representation does not require IEEE 754, then it cannot require only
transformations for IEEE 754 without making other representations behave
badly.

------
jmiserez
Not developers. This survey included mostly PhD students (73), faculty (49),
but only 23 software engineers. The rest are others. Only 1/4 see software
engineering as their main role according to the paper.

A PhD student is very different from a professional developer with years of
real-world experience in industry. It's a completely different job.

The title is highly misleading.

~~~
crispyambulance
It's OK, not misleading at all, and the abstract spells out what they're
studying and the motivation behind it clearly.

A "developer" is just someone who writes software and not necessarily a
"professional developer" with years of experience in a particular kind of job
like what you have in mind.

Moreover, I would even expect that the vast majority of "professional software
engineers" who make a living strictly writing software without any particular
domain in focus, might actually fare worse in floating point gotchas than
scientific domain workers who are "software engineering dilettantes" but
nonetheless do computation as their job.

~~~
jmiserez
Maybe. The paper even mentions this, sort of contradicting your expectation:

> _Perhaps not surprisingly, the most predictive factor is simply Contributed
> Codebase Size, the effect of which is shown in Figure 16._

> _There is a gain of only about 2 /15 compared to those who reported a
> codebase where floating point was not intrinsic or where they were not
> involved._

The title would be better suited if they included a higher number of
HPC/ML/etc software engineers (post PhD) that work on larger scale projects in
industry/labs.

Specifically, I disagree with this assessment:

> _We believe that the combination of our recruitment process and the
> resulting background of the participants illustrated here suggest that our
> sample is a good representative of software developers who write code for,
> and in support of, science and engineering applications._

That's just outright ignoring all science and engineering applications outside
of academia/academic research.

~~~
crispyambulance

        > The title would be better suited if they included a higher number of HPC/ML/etc software engineers (post PhD) that work on larger scale projects in industry/labs.
    

If someone needs to be a "post PhD" software engineer working on large scale
computation projects in industry to be considered "a developer" whose problems
with floating point are worth investigating and dealing with, then we have a
problem of definition.

Their premise is that if folks in academia are having a hard time with it,
then perhaps the training and tooling around floating point needs some
reconsideration. That seems very reasonable to me.

~~~
ChrisLomont
>Their premise is that if folks in academia are having a hard time with it,
then perhaps the training and tooling around floating point needs some
reconsideration

There's been 50+ years of this reconsideration. Some problems are simply hard,
and perhaps nothing will make it easier than to simply learn the nuances.

For some domains, there are some solutions, like BigInt to avoid overflow, but
each comes with other tradeoffs.

For general floating-point, it's mathematically quite likely there is no
better tradeoff. You cannot stuff infinite values into finite space.

------
caseymarquis
I once encountered an old database using floats as its primary keys. This was
in a fairly popular piece of manufacturing software from the 90s.

~~~
sevensor
I sometimes, in Python, cautiously use floating point numbers as dictionary
keys. I do this when what I mean is "this exact floating-point number" and not
"the result of an equivalent computation." It's _almost_ never a good idea,
because it's really easy to trip yourself up by assuming the latter when only
the former actually holds.

~~~
jmiserez
And even the former apparently isn't guaranteed on all hardware. I found this
gem in the Java documentation, I don't know which processors actually do this:

>" _Arithmetic operations on signaling NaNs turn them into quiet NaNs with a
different, but often similar, bit pattern. However, on some processors merely
copying a signaling NaN also performs that conversion. In particular, copying
a signaling NaN to return it to the calling method may perform this
conversion._ "

Source:
[https://docs.oracle.com/javase/7/docs/api/java/lang/Double.h...](https://docs.oracle.com/javase/7/docs/api/java/lang/Double.html)

~~~
Veedrac
If you're putting floats in a dict, you can't use NaNs anyway, since they
aren't equal to each other.

------
porpoisely
I doubt most developers really understand how floating points work ( as in how
it is represented in memory and why and the mantissa, bias, etc ). Most
understand that it is just an approximation and go on about their day.

~~~
tracker1
When using floating point numbers, that's usually as far as you need to know.
That and the range of safe integers for langauges like JS that only do float.
(Though now there's BigInt at stage 3)

------
known
Yes with [http://0.30000000000000004.com/](http://0.30000000000000004.com/)

------
Traster
This is a pet subject of mine since as an FPGA developer a huge amount of my
time is spent focusing on bit-precision and Fixed-Point and reduced precision
floating point implementations of algorithms. In fact one of the interview
questions I ask focuses on how to do different mathematical operations in
floating point using only logical/integer operations.

On the one hand I find the results unsurprising - even people I know to have
worked a lot on numerics often have only a rudimental understanding of the
intricacies of corner case behaviour for floating point, and yes that
absolutely manifests in hitting a wall when something curious goes wrong.
Mostly this results in head banging until you find the piece of code going
wrong is the numerical piece and then very quickly you start looking at
possible floating point gremlins.

Having said that though this paper seems to have a very academic view of what
HPC is. Even for people designing HPC systems numerical optimizations are
rarely a huge chunk of their job so it's probably not actually that important
- I think the fact few people have a good understanding is a reflection of the
fact it's not often necessary.

Finally while we're on the topic: Does anyone know a good tool that can allow
me to write an equation, specific the precision of inputs/outputs and get the
required precision of all the operators? I know Matlab have the DSP toolbox
but it has some serious limitations, I'm still in search of something
fantastic.

~~~
nestorD
Herbie is not an exact match for your need it might be useful nevertheless :
[https://herbie.uwplse.org/](https://herbie.uwplse.org/)

------
sn41
I understand floating point representation, but beyond that, it is a bit
murky. I find floating point arithmetic to be very hard to really implement.
For example, a basic question I have involves scaling: suppose I want to
perform an operation involving large numbers like 13000.01 + 13000.01 +
13000.01. Do I get better precision results if I do 0.01 + 0.01 + 0.01 in
floating point, then do the 3*13000 in integer arithmetic, and add the
results?

~~~
magicalhippo
A more general way to do that is Kahan summation[1], where you keep track of
the rounding errors made so far.

[1]:
[https://en.wikipedia.org/wiki/Kahan_summation_algorithm](https://en.wikipedia.org/wiki/Kahan_summation_algorithm)

------
man-and-laptop
It's interesting to point out that you _can_ compute with exact real numbers.
But this is of course going to be less efficient than using something crude
and fast like floating point numbers. There's also an argument that
imprecision from your physical sensors is probably going to be more
significant than the rounding errors that come with floating point.

Apart from greater speed, floating point numbers don't really have any
advantages over exact real numbers. Problems like the inability to compute
discontinuous functions over the real numbers manifest as even worse problems
over the floating point numbers.

I'm also a believer that anything beautiful is probably going to end up
useful. I think exact real computation (as described in Type Two Computability
Theory) is one such thing. Automatic differentiation was another thing I found
beautiful, before I found out about its applications to neural networks.

~~~
sn41
Type-two theory has a huge disadvantage: it elevates the type of all
operations by a level. For example, real numbers become functions. Hence
addition becomes a type 2 functional. Integration becomes type 3. It is very
hard to reason about the efficiency (and computability, come to think of it)
of type 3 functionals. There's some esoteric example involving the fan
functional [1].

The complexity theory of type-2 functionals is a theory which is natural to
ask for, but after results by Cook, Bellantoni, Kapron, Seth, Pezzoli and
others, has come to an impasse and now seems abandoned.

[1]
[https://pdfs.semanticscholar.org/3888/cc2d3c44fb051067581e39...](https://pdfs.semanticscholar.org/3888/cc2d3c44fb051067581e3947c3b9f9af1d5c.pdf)

~~~
man-and-laptop
They effectively change the structure of your code. Every time you do an
operation on a real number, it essentially launches a new process that runs
indefinitely in the background. I would file that under speed problems. To
control the speed penalty, you need to make deep changes to your code. I think
Andrej Bauer came to this conclusion.

Regarding computability, it's not so bad. Basically a continuous operator is
usually computable. Example: Integration is continuous, hence (likely, and in
this case actually) computable.

------
cjhanks
I have interviewed a lot of developers, and floating point precision is one of
my common phone screen questions.

My experience is; those who _should_ know about it, do. Of course, a lot of
apps or web developers don't. And I think that is okay.

The bigger issue is code designed for float32 (for performance reasons) being
assumed valid for float64 data. And in these cases the issue is always, what
is the appropriate way to communicate to your user the intended use case of
the function? When the writer is long gone, a user will not intuitively know.
Whenever I have to use a 3rd party library (ahem, PCL). I peruse the code and
grep for float variables... If I see too many, I simply assume I need to
recenter my float64 to be near 0.0.

Not everybody has the experience for this diligence, thus it appears as if
developers don't understand.

------
hackermailman
A lecture on IEEE floating point from CMU including most of the gotchas the
pdf contains
[https://scs.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=...](https://scs.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=7cceb0cb-e07c-4043-a7f3-d4d8c4661512)

This is an active research area, ENS de Lyon has a lot of material available
if interested implementing floating point on FPGAs, plus there is Vol 2 of The
Art of Computer Programming where Knuth shows the reader how Babylonian
mathematicians made extensive use of a sexagesimal (radix sixty) positional
notation that was unique in that it was actually a floating point form of
representation with exponents omitted.

------
jstewartmobile
Even if you understand them... in a polyglot world, you're still going to get
burned.

It is so easy--especially when you're jumping from a NUMBER language like SQL
or Scheme to a floating-point one like C or JavaScript--to backslide into
boners like

    
    
        for(i = 0.0; i < 1.0; i += 0.1)
    

which would be a non-boner with a more advanced number tower.

edit: My first Clojure program was the last time this got me--LISP on the
outside, Java floats/ints/bigints on the inside.

~~~
heavenlyblue
Yeah, and that’s why we have int for counting.

~~~
jstewartmobile
So if I'm marking an axis from 0 to 0.9 in 0.1 increments, I should
_obviously_ do the integer indirection dance?

After all, every damn fool knows--even ones like myself who've implemented FP
in assembly--that 0.1 + 0.1 + 0.1 = 0.30000000000000004.

 _Silly me!_

~~~
heavenlyblue
>> So if I'm marking an axis from 0 to 0.9 in 0.1 increments, I should
obviously do the integer indirection dance?

Yes. Because you clearly don't understand what you're doing at all if you
believe that equivalence relation is defined the same way for floating-points
as it is defined for integers.

Floating-points aren't numbers. They're more like confidence intervals - and
at a certain limit a confidence interval behaves just like an integer. But
it's only a limit.

The fact that most users of programming languages don't want to keep track of
that in their code only reinforces my belief that they must do exactly as the
software guidelines tell them to.

~~~
jstewartmobile
I know what floating point is. I've implemented it in hardware. I've
implemented it in software. But you know what? Floating point is not the only
way people have ever represented real numbers on the computer.

In most SQLs, decimal fixednum types are predominant. Most LISPs have a number
tower that automatically juggles fixnums, bignums, ratios, floats, and complex
numbers. In the 8/16-bit days--before everything had a floating-point unit--it
was not uncommon to store and calculate numbers directly in BCD. Two-digits to
the byte, and all of the operations--including formatting--were very
straightforward to implement. Many of the old business mainframes even have
hardware instructions to pack/unpack/operate on BCD.

 _> that most users of programming languages don't want to keep track of
that..._

Why would I want to keep track of that when the language (or more accurately,
the data type) will do it for me?

I'm sorry man. The whole world isn't C and JavaScript (yet). That we have
forced crud-app developers to resort to garbage like this[0]--in places where
the performance and storage benefits of floating-point are irrelevant--is a
stain on the profession.

[0] [http://www.jacklmoore.com/notes/rounding-in-
javascript/](http://www.jacklmoore.com/notes/rounding-in-javascript/)

------
jdblair
Obligatory link to the classic paper on the subject, from 1991:
[https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.h...](https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html)

------
stallmanifold
How many software engineers had to study IEEE floating point in (any amount
of) detail? Part of my formal education is in computer (hardware) engineering,
and we studied hardware number representations and manipulations quite
extensively in my microprocessors courses, so I developed some facility with
the topic once upon a time.

------
sifoobar
I've met plenty of coders who aren't even aware of the problem, and I spent 13
years maintaining and extending an accounting system written by people who
couldn't tell a floating point issue from their ass on a good day.

It's not going to happen much until we stop pushing floats as the standard
solution for dealing with decimals. I'm pretty sure they are more commonly
abused than used at this point, and most cases would be better off with
fixpoints, rationals or bignums in increasing order of flexibility and
complexity.

Lisp and Perl6 get it more or less right. My own baby, Snigl [0], only
supports integers and fixpoints so far. I might well add floats eventually,
but I intend on hiding them well enough to not encourage anyone who isn't
motivated.

[0] [https://gitlab.com/sifoo/snigl](https://gitlab.com/sifoo/snigl)

~~~
ThePadawan
> (...) until we stop pushing floats as the standard solution for dealing with
> decimals.

I'm curious, in that sentence, who is "we"? I cannot imagine a scenario where
someone would be so technically involved to be aware of the difference between
a floating-point and fixed-point decimal, but still decide to use floating-
point in an accounting environment.

Are you talking about project managers? Architects?

~~~
SpicyLemonZest
If you Google how to manipulate decimal values, or you type `rate = 1.23`,
you're almost always going to get floating point. `1.1 + 2.2 == 3.3` yields
false in most commonly used programming languages, with the notable exception
of SQL.

That's what it means that we've established floats as the standard. You have
to specifically choose _not_ to use floats, look up your language's entirely
non-standardized approach for representing exact decimals, and then maintain
constant vigilance against entirely intuitive things like decimal literals.

------
kazinator
> _Zero Divide By Zero:_

> _This question asks whether in floating point 0.0 /0.0 is a non-NaN value,
> which it is not. NaN generation is desirable here, since it will propagate
> to the output as a NaN and thus make the user suspicious._

Horrible nonsense. What is desirable is an exception so that it's loudly
confirmed to the user that something has screwed up. We don't want to lead the
user on with subtle hints and suspicions.

Even a dollar store calculator from 1985 gets this right by locking up with an
E on he display, requiring a clear.

~~~
jmalicki
This is configurable. Sometimes you want to see 99% of your results and know
those were weird ones, and you can deal later.

------
gcc_programmer
60% of the people who took the survey develop software to support their main
role. So, 60% of the people are not really software engineers but coders by
necessity: could this explain the results easily?

------
Upvoter33
60% of the time, developers understand floating point 100% of the time.

