
How many floating-point numbers are in the interval [0,1]? - mgdo
http://lemire.me/blog/2017/02/28/how-many-floating-point-numbers-are-in-the-interval-01/
======
mtklein
The correct answer is 1,065,353,216.

This is easy to work out yourself if you remember one basic, handy property of
floats: adjacent floats are adjacent in bit representation, except -0.0f and
0.0f.

For example, 0x00000000 is +0.0f. 0x00000001 is the smallest non-zero positive
float. 0x00000002 is the second smallest.

The only exception to this is -0.0f and +0.0f, which are 0x80000000 and
0x00000000. The rule works with denormal floats, normal floats, and even right
on the line between the two. If you want the next positive float, you always
just add 0x00000001.

Now, +1.0f happens to be 0x3f800000. Recall +0.0f is 0x00000000. The number of
values between the two is 0x3f800000 - 0x00000000 == 0x3f800000. Write that in
decimal and you get 1065353216.

~~~
vog
The correct answer is 1,065,353,217. You have an off-by-one error in the
following line of argument:

 _> The number of values between the two is 0x3f800000 - 0x00000000 ==
0x3f800000_

This should be corrected to:

 _> The number of values between the two is 0x3f800000 - 0x00000000 ==
0x3f800001_

In general, the number of numbers in the closed integral interval [a, b] is
not b-a, but 1+b-a. For example, the closed interval [0,10] contains 11
integers, not 10.

(The above assumes that the original question assumes the floating point
number -0 to be outside of [0,1]. Otherwise, you'll have to add 1 more.)

~~~
lightedman
I was under the impression (after several thousand attempts) that the correct
answer is infinity, limited only by your addressable memory and bitspace.

~~~
afandian
I think people are assuming IEEE 754 floating point (as per the article). Your
memory is finite, 32 bits.
[https://en.wikipedia.org/wiki/Floating_point](https://en.wikipedia.org/wiki/Floating_point)

------
jfaucett
Honestly, I think IEEE 754 floating point numbers are a pretty bad way of
dealing with real numbers and just cause tons of headaches that every math
library has to deal with. Even high level programmers aren't shielded from the
NaN nonsense. It would be great if we had an underlying implementation we
could ignore and that allowed us to think at a more mathematical level.

I actually found something recently on this topic while googling for any
alternatives, its called Unum and looks very interesting:
[https://en.wikipedia.org/wiki/Unum_(number_format)](https://en.wikipedia.org/wiki/Unum_\(number_format\))

Does anyone know more about this? Are there any chips out there that support
it?

~~~
nine_k
If NaN is nonsense, what would you have 1/0 or acos(2) to produce? (An
exception is the same NaN, only less convenientlying packaged.)

~~~
dlubarov
> An exception is the same NaN, only less convenientlying packaged.

I prefer an exception because I'd rather have my code fail fast, and
immediately point me close to the source of the bug, rather than letting a NaN
or Infinity propagate through my code base and cause some harder-to-debug
problems down the line.

That's the main thing I don't like about JavaScript. 1/0 is Infinity,
Math.acos(2) is NaN, Object().foo is undefined, etc. I find myself wasting a
lot of debugging time just figuring out where those (usually expected) values
came from originally.

~~~
dbcurtis
Exceptions are utterly painful for vectorized operations. Goodbye Tensorflow
if you say goodbye to NaNs

~~~
greglindahl
That's an excellent explanation for why scientific computing has been using
vectors for a long time and has traditionally faulted when hitting the first
NaN. Is there something about Tensorflow that makes propagating NaNs useful?
And if so, why are you generalizing from Tensorflow to all "vectorized
operations"?

~~~
rtpg
my guess is that this is a performance argument. If you don't have to check
the results, your circuits to do huge vector calculations can be simpler.

------
dekhn
He references one of my favorite functions, 'nextafter'.
[http://en.cppreference.com/w/cpp/numeric/math/nextafter](http://en.cppreference.com/w/cpp/numeric/math/nextafter)

I'm the kind of person who understands floats by looking at the underlying bit
patterns, and nextafter makes it easy to traverse the integers.

~~~
gajjanag
> He references one of my favorite functions, 'nextafter'.
> [http://en.cppreference.com/w/cpp/numeric/math/nextafter](http://en.cppreference.com/w/cpp/numeric/math/nextafter)

It is a cool function. I remember using it for some testing I did while
checking the accuracy of various implementations of basic math functions.

------
astrodust
> Of all the float-pointing point numbers your computer can represent, a
> quarter of them lie in [0,1].

Many people think floating point numbers are magically precise. They're not.
They're far more accurate at lower magnitudes and precision fades as you work
with larger values.

~~~
ryandrake
Correct me if I'm wrong, but my understanding is that a consequence of this is
that it's better to store, for example, angles as radians from -pi to pi
rather than as degrees between 0.0 and 360.0, in order to take advantage of
all that precision between 0 and 1.

~~~
jzwinck
I don't see why that would help. Of course there is more precision in terms of
decimal places around 1 than around 360. But if you use 360 each 1 represents
a much smaller piece of the circle.

To put it another way...it's about significant digits. A float has about 7 and
a double about 15. It does not matter whether you use 0 to 360 or 0 to 3.6
million...the number of digits that are meaningful remain the same.

~~~
mirimir
For binning data in spreadsheets for histograms etc, I typically use text() to
set significant digits, and then value(). It's also good sometimes to make
sure that 0 is really 0.

------
Certhas
Ummm... maybe I am mistaken, but as the fp numbers are not uniform in [0,1]
chances are I don't want to pick one at random. Rather I want to pick a fp
representation of a random real between 0 and 1. In that case the readers
suggested strategy seems fine.

Am I wrong?

~~~
petters
The author says in a comment

> If hitting exactly zero is much less probable than hitting exactly 0.5, then
> I would argue that you do not have a uniform distribution…

But due to the non-uniformity of floating point numbers, I think the author is
wrong here.

We can think of each floating point number as an interval. The probability
that it is picked should ideally be equal to the size of the interval.

~~~
The_suffocated
Yes. The effect of having a coarser partition around a mesh point is countered
by the effect of a higher frequency of hitting that partition. This actually
is the essence of importance sampling. For the purposes of numerical
integration, as long as the integrand is relatively smooth with respect to the
partition size, there shouldn't raise any serious problem. On the contrary, if
the integrand does vary wildly between two adjacent floating point numbers,
then the real issue here is that the precision (single, double, quadruple,
etc.) used is not fine enough, rather than the deviation from uniform
distribution.

------
ldarby
I had a similar question a while ago: how many decimal digits does it take to
represent the smallest possible 80 bit float. The answer is ~16000:

    
    
      #include <stdio.h>
    
      int main(int argc, char **argv)
      {
            unsigned long long int mant, exp ;
    
            mant = 0;
            exp = 1;
    
            typedef struct _bitfield80 {
                    unsigned int sign:1;
                    signed int exp:15;
                    unsigned long long int mant:64;
            } bitfield80;
    
            union {
                    long double a;
                    bitfield80 b;
            } union80;
    
            union80.b.sign = 0;
            union80.b.mant = mant;
            union80.b.exp = exp;
    
            printf ("%0.17000Lf\n", union80.a);
            return 0;
      }

~~~
jacobolus
Depends what you mean by “represent”.

The smallest positive “denormal” 80-bit float is (I think) 2^(2 − 2^(14)). I
just represented it using 5 decimal digits and two minus signs. :-)

~~~
yiyus
The Berry paradox takes this a bit further.

It may appear that "the least integer not nameable in fewer than nineteen
syllables" is 111777. But "the least integer not nameable in fewer than
nineteen syllables" is itself a name consisting of only eighteen syllables,
and therefore the least integer not nameable in fewer than nineteen syllables
can be named using only eighteen syllables (!).

There are alternative expressions, for example wikipedia mentions "the
smallest positive integer not definable in fewer than twelve words" and "the
smallest positive integer not definable in under sixty letters".

------
dnautics
for many applications, consider randomizing in [1,2) and then subtracting one.
Although many of the fp values in [0,1) will be inaccessible, you are
guaranteed uniformity, and also the lattice that you're drawing from will be
fixed.

~~~
SamReidHughes
How do randomize in [1,2)? You might as well just generate numbers from [0,
2^53) or [0, 2^24) and divide.

If you didn't do that on the way to generating in [1,2), you're going to have
fine-grained non-uniformity anyway.

~~~
Dylan16807
>How do randomize in [1,2)?

0x3f800000 + 23 random bits

~~~
SamReidHughes
Well. That'll do it. Probably no slower than the integer to float conversion
too. It gets you multiples of 2^-52 instead of 2^-53 though.

~~~
strainer
I see your talking 64 bit floats there which is my habit too. Having only 24
bit mantissa float32s seem insufficient for producing practically uniform
variates - the missing 2^24th can be spotted averaging just a billion or so of
them. I dont know if it has been improved recently but last year Chrome's
Math.random was only putting 32 bits into its float64 - the missing four-
billionth can be suggested from the average of a few hundred billion variates.

~~~
SamReidHughes
Yeah, I really think a better way is to imagine generating a uniform real and
rounding -- generating [0,1) instead of [0,1] is just sick. If you do rand() *
k + n you can get [n, n+k] anyway, so it's not a good primitive to get a half-
open interval with. You can generate an integer n in [1, 2^25], then take (n
>> 1) * (1 / (float)(1 << 24)) to get a pretty good [0,1].

~~~
mark-r
For a 64-bit floating point number, the difference between [0,1) and [0,1] is
for all practical purposes nonexistent. The chances of getting 1.0 exactly are
so low that you can't build a test to know if it's excluded or not.

For a 32-bit floating point number it's more of a problem, but your
application would have to be extremely demanding to detect the difference.

~~~
strainer
A rough calculation, a float64 rng which might reasonably supply billions of
rnds an hour for simulations, would have a fair risk of hitting '1.0' every
million hours of use, which means its important for production quality code to
not run that risk.

A prng called Alea works with floating point arithmetic to generate rnds
securely confined to [0,1] using a 'subtract with carry' scheme. It involves
this kind of arrangement:

    
    
      state=state*something+carry
      carry=floor(state)
      return state-carry //this is 0<1
    

Its very similar to linear congruential generator where top bits are masked
away, in this case they are 'floored' off and fed back into the state. Alea
seems to generate high quality rands and works extremely quickly in javascript
engines by avoiding integer math.

My rnd utility module uses a similar algorithm. There may be a note of caution
whether floating point math is standardised enough to use for a seedable prng,
but so far my machines and phone have agreed in tests.

[http://strainer.github.io/Fdrandom.js/](http://strainer.github.io/Fdrandom.js/)

~~~
mark-r
My argument was for the other direction, using [0,1) when you actually want
[0,1]. I agree that if 1.0 would cause a problem in your code you should use a
rng that avoids it. All of the floating point random number generators I'm
familiar with deliver [0,1).

~~~
strainer
Interesting, I assumed java and javascripts behaviour were common. I dont
recall seeing this notation before either [0,1) maybe i will now %)

~~~
mark-r
I usually use C++ or Python, so I had to look up the others.

From Java: "Returns a double value with a positive sign, greater than or equal
to 0.0 and less than 1.0."

From Javascript: "Return a random number between 0 (inclusive) and 1
(exclusive)"

So yes, all 4 are consistent in _not_ including 1.0 in the range of random
real numbers. I'd call that common.

For more on the notation see
[https://en.wikipedia.org/wiki/Interval_(mathematics)#Includi...](https://en.wikipedia.org/wiki/Interval_\(mathematics\)#Including_or_excluding_endpoints)

~~~
strainer
Apologies I wasn't following properly, and thanks for the interval math
pointer.

------
yason
I don't like the divisions, though, and all the analysis that comes with it.

A way that is intuitively more robust would be to generate 23 or 52 random
bits from an integer RNG for the mantissa (given you trust you integer RNG is
unbiased). For example, take a random uint64_t, clear the sign bit, rewrite
the exponent with a predefined value, and let the mantissa be untouched. That
will give you a random floating point number in the range between 0.5-1.0,
1.0-2.0, or 2.0-4.0 etc, depending on the exponent you choose.

You can trust that the value is entirely random to the precision offered by
the mantissa's size. Then you can control the accumulation of error by what
you choose to do with the random number.

You still can't get to numbers that aren't representable by floating point, of
course, but everything will start from the most random and unbiased value you
can think of. You might have to introduce some error by subtracting the lower
bound and multiplying appropriately to get to your desired range. Or, if you
have the luxury to choose your desired range appropriately, you might be able
to use the random value directly.

~~~
jxy
Hitting each representable floating point number with an equal probability
does not generate an approximately uniform distribution of [0,1). For a
sufficiently small positive number, eps, you have to have equal probabilities
of numbers falling in [0,eps) and [0.5,0.5+eps) for a uniform distribution,
and using random bits would fail this requirement.

------
murkle
Article says 1,056,964,609 / 1,056,964,610

Quick Java program says 1,065,353,216 / 1,065,353,217

Which one is right? Or are Java floats just "different"?

    
    
      static {
        float a = 0;
        long count = 0;
        while (a < 1) {
          count++;
          a = Math.nextAfter(a, 2);
        }
        System.out.println("count = " + count);
      }

~~~
drv
nextAfter is probably also including the denormals (an additional 2^23 values
near 0).

~~~
mturmon
Yep:

    
    
      1,056,964,609 - 1,065,353,216 = 8388607 = 2^23 - 1
    

because the denormal that is all-zero is already present.

------
js8
Slightly OT but regarding floats - I was wondering if we could get by just
with an exponent and ditch the mantissa. Each number would be represented as
an integer exponent of 2^-n for n corresponding to the precision.

------
aethertron
The more I read about the gnarly details of floating point specs and
implementations, the more they look like a messy hack. I think it'd be nice to
just use rational numbers in my programs.

------
santaclaus
What about nan? The test '!(x < 0 || x > 1)' will say yes, while 'x >= 0 && x
<= 1' will say no.

Or -0 for that matter.

~~~
amelius
Common sense approach: like the name says, NaN is not a number. The question
was about floating point numbers, not floating point values. So I suppose we
shouldn't count NaN. Further, -0 (an arbitrarily small number less than 0)
doesn't logically fall in the range [0, 1], so it is excluded.

~~~
mark-r
There's no need to exclude -0.0, since mathematically and by all tests
available to the floating point unit it's equal to +0.0.

------
riprowan
I can't believe I'm the only person to ask "what bit depth?"

There is a pretty big assumption baked into the title.

Edit: hi, I see that I made a mistake, can someone correct my
misunderstanding?

~~~
pharrington
IEEE 754 floats are 32-bit. Doubles are 64-bit.

------
kumartanmay
What is the practical application of this?

~~~
paulddraper
Currently working on monetization strategy.

Doing A/B tests with ads.

Looking for first round investors.

~~~
ejanus
): I will buy your IPO.

------
pmoriarty
Asking how many "are" in the interval implies that these numbers exist in some
sense in that interval.

A clearer way to phrase the question might be: "How many floating-point
numbers between 0 and 1 can be generated without duplicates?"

------
stcredzero
(I invoke my pedant-pass.) As formulated in the title, "How many floating-
point numbers are in the interval [0,1]?" you could argue that this is the
cardinality of the Real Numbers. What the article says it is really talking
about are single-precision IEEE 754 floating-point numbers. However, I could
define any number of my own floating-point representations at various sizes.
The cardinality of all possible floating point representations would be the
same as that of the real numbers. This is true even for irrational numbers, as
one could formulate floating point representations that specially represent
those numbers.

(Also, this is arguably an exception to Betteridge's Law, and arguably not an
exception to Betteridge's Law.)

~~~
jacobolus
What makes you think any arbitrary real number qualifies as a “floating point
number”?

I have never seen the term used in anything like that context.

To me, the term “floating point number” is about number representation, not
number identity. It is an inherently finite quantity in every instance I’ve
ever seen.

Examples of floating point numbers: The sexagesimal cuneiform 42 25 35 written
on the Babylonian tablet YBC 7289; 3.1415 × 10⁰; 1.010101010101₂ >> 2.

Not examples of floating point numbers (in my opinion): 1/√2; _π_ , 1/3.

~~~
theoh
The GP is claiming that all possible floating point systems, taken together,
would contain the same number of values as there are points on the real line.

Consider a floating point system based on
[https://en.wikipedia.org/wiki/Golden_ratio_base](https://en.wikipedia.org/wiki/Golden_ratio_base)

~~~
stcredzero
Thanks! To be just a bit more precise, I am claiming that all possible
floating point systems, taken together, would contain the same number of
values in [0,1] as there are points on the real line.

~~~
Dylan16807
Nobody uses "floating point number" to mean "a number that could theoretically
exist in a floating point system I will invent for you after you tell me the
number". That's not being pedantic, that's insisting on a wrong definition.

~~~
stcredzero
_Nobody uses "floating point number" to mean "a number that could
theoretically exist in a floating point system I will invent_

Put a full stop there -- Oh yes, people do mean that! Most of the time, people
are referring to a bit representation in a specific system like IEEE's.

 _a floating point system I will invent for you after you tell me the number
"_

The above part of the sentence isn't a definition of "floating point number."
It's how I use the common notion of "floating point number" in my argument.
Your objection only looks like it works because you conflate the two concepts.
If you don't conflate the two concepts, then you are arguing that most low
level programming that deals with 32 bit floats doesn't actually deal with
"floating point numbers." That is a reasonable assertion for a reasonable set
of definitions. However, it's not the one I'm using, which also fits the
reality of the mental models most people are actually using.

In terms of program correctness, you can find many examples where using an
abstract concept of "floating point" instead of the concrete representation
will produce errors. So then why is it "the correct one" as you say?

~~~
Dylan16807
I can't parse what you're saying. What two concepts are you accusing me of
conflating?

There are many different floating point systems, but they all share certain
attributes. None of them can exactly represent sqrt([insert large prime]). If
you want to invent a bunch of such systems post-facto, you are abusing the
term. The union of all systems that can reasonably be called "floating point"
is not the set of real numbers. It's countable, as a very loose upper bound.

