This is not inherent to floating point; a floating point representation that was based on decimal digits instead of binary ones would not have this issue.
It's not too hard to understand limited precision and rounding; we deal with both all the time. The part that's hard to understand is that you have to perform rounding simply to put a number like "0.1" into a floating-point value.
Specifically, in 0.1 as an IEEE float is represented as:
val = 0x3dcccccd
sign = 0
significand = 4ccccd (1.6000000)
exponent = 0x7b (-4)
This is made even more confusing by the fact that rounding performed on output will make it appear that 0.1 has been exactly represented in many cases.
So take a data type that can't exactly represent some numbers that look very simple in decimal, but that appears to represent them exactly when you print them out, but then compares not equal when you try to actually compare them. No wonder people get confused!
If that fraction is representable as an exact "decimal" number in base b that means that p/q = m/b^n = m * b^(-n), where m and n are integers too. For example, 3/4 = 75/10^2 or 0.75 in decimal, 3/2^2 or 0.11 in binary.
That means p * b^n = m * q. We said p and q have no common factors, so all of q's factors go into b^n. That means that q divides b^n, or in other words:
p/q is representable as an exact "decimal" number in base b
if and only if q divides some power of b.
For example, 0.1 is 1/10. But there is no power of 2 which is divisible by 10, so 0.1 is not exactly representable in binary.
As another example, 1/3=0.33333.... because there is no power of 10 divisible by 3.
Or like integers in base pi! ;)
I'm working on a program right now to give a detailed dump of a double/float's value, to make this as clear as possible (and I'm learning a lot in the process).
EDIT: got an initial revision out there; check out: https://github.com/haberman/dumpfp
The idea is that algorithms should be designed with a minimum required precision, but no maximum, and should not fail if higher precision is used.
I think this rule serves very well.
"This design allows the CLI to choose a platform-specific high-performance representation for floating-point numbers until they are placed in storage locations. For example, it might be able to leave floating-point variables in hardware registers that provide more precision than a user has requested. "
I have a numerical code that I cross-compile with gcc -O0, gcc -O3, clang, icc, and MS Visual C, all on the same machine. Each version can give slightly different results. The fact that they don't give wildly different results is mainly down to the stability (in the ODE sense) of the problems run. Unstable problems would definitely give different results.
Also, it might have higher precision in most cases but just not in that case where your code needs it most.
Real-world examples are x86 compilers that use the 80-bit floating point registers for intermediates, but flush them to 64-bit results (sounds stupid, but it is not; flushing typically is done to the bytes that hold the result variable). There, the register allocation algorithm used by the compiler can be the determiner of which operation sequences get done in 80 and which in 64 bits.
gcc: http://gcc.gnu.org/wiki/FloatingPointMath (the 'Note on x86 and m68080 floating-point math' section), http://gcc.gnu.org/onlinedocs/gcc-3.4.4/gcc/Disappointments.... ("On 68000 and x86 systems [...] you can find that a floating point value which is not a NaN is not equal to itself")
Microsoft's compilers: http://msdn.microsoft.com/en-us/library/e7s85ffb.aspx
The title isn't accurate though. The article is mostly about using IEEE 754 to manage computational error, which is a pretty esoteric subject.
Most programmers spend precious little time using floating point and don't need to worry about computational error. Even among those who use it a great deal -- those doing numerical analysis -- computational error doesn't seem to be something they concern themselves with. My guess is that in most instances it's swamped by experimental error.
My thoughts on this article are:
1: It's certainly good that it's available online. It'd be better with a different title.
2: It's good that modern CPUs support IEEE 754 for the rare instance that people use it.
3: It's worth reading the first third or so of it, just to get an idea of what computational error is
4: The most important thing the "typical" programmer needs to know is that dollar values shouldn't be represented in floating point. Ideally they shouldn't be represented as integers either because the integer rounding truncates towards zero, which is not how an accountant would round.
You just shouldn't calculate with 1 cent accuracy if the rounding methodology is important to your business, but truth be told most businesses have a lot more to optimize than the rounding methodology used.
Perfectly true. I can see exceptions being managing other people's money--banks, accountants, and the IRS had damn well better get this right.
I think this is true of experienced programmers, but I see inexperienced programmers tripping over floating point misunderstandings quite often. In almost every case they shouldn't have been using floating point at all.
For example, if you're working with money, you don't need floating point.
0. Do not calculate monetary amounts in FP. Businesses don't react well to being told about acceptable imprecision.
Even if you think you have integers ... sometimes you don't.