
High Expectations for Low Precision at Cern - rbanffy
https://www.nextplatform.com/2017/07/12/high-expectations-low-precision-cern/
======
foota
This article is a non sequitur. Spends the first half taking about the rising
usage of low precision work in ml and then talks about how CERN is still using
high precision.

Seems like the missing bit is talking about how CERN could benefit from low
precision? But it's certainly not in the article.

~~~
cryptonector
It mentions GPUs. GPUs generally only offer low-precision floating point
arithmetic. GPUs can do many more 32-bit floating point operations per-second
than CPUs.

There must be a huge demand for running numerical workloads on faster kit
(GPUs), but the attendant loss in precision can make it very difficult to get
good results in high-energy particle physics. The idea here might be to
develop numerical methods that make up for low-precision arithmetic yet
preserve the speed advantage of using GPUs. If so, TFA is very poor indeed.

~~~
skummetmaelk
Low precision ML is less than 8 bits. 32-bit floating point is the highest
precision used.

~~~
trsohmers
The Titan and Tesla NVIDIA cards are still built with 64 bit/double precision
in mind with "only" a 50% performance reduction compared to FP32 in most of
those devices. It is the GeForce gaming cards and the ones being targeted now
for ML that are gimping FP64, where it is 1/4th or even 1/16th the speed of
FP32.

The vast majority of "real" HPC still requires FP64 (only large industry
categorized under HPC that I know is perfectly fine with FP32 is geospatial
oil & gas stuff)... a valid LINPACK has a hard FP64 requirement.

------
supahfly_remix
This article confuses me. Machine learning needs less precision (for some
reason), but the workloads that CERN performs are scientific computing, not
ML. Or, are they doing ML and so need less precision?

~~~
redcalx
> for some reason

When training a model you're typically following a gradient which may be very
small so you want high precision. One a model is trained and you're just using
it you can generally get by with very low precision, ultimately we're usually
talking about placing some input vector in one or more binary categories and
the affect of small signal variations higher up in the model generally don't
amount to much w.r.t the final categorisations.

~~~
pps43
The folk theorem of statistical computing [1] says that when you have
computational problems, often there’s a problem with your model. So chances
are that if your gradient vanishes with 32 bit floats, switching to 64 bit
floats won't help much.

[1]
[http://andrewgelman.com/2008/05/13/the_folk_theore/](http://andrewgelman.com/2008/05/13/the_folk_theore/)

------
_FKS_
Most likely related:

John L. Gustafson presenting "Beyond Floating Point: Next Generation Computer
Arithmetic"
[https://www.youtube.com/watch?v=aP0Y1uAA-2Y](https://www.youtube.com/watch?v=aP0Y1uAA-2Y)

With papers here: \- UNUMS:
[http://ubiquity.acm.org/article.cfm?id=3001758](http://ubiquity.acm.org/article.cfm?id=3001758)
\- Posits:
[http://johngustafson.net/pdfs/BeatingFloatingPoint.pdf](http://johngustafson.net/pdfs/BeatingFloatingPoint.pdf)

~~~
trsohmers
There has been a lot of talk about lower precision and approximate computing
over the past couple of years, which I hope will be recognized as a fad with
John's (and to a lesser extent mine/REX Computing's) work... Why sacrifice
precision/accuracy if you don't have to? Posit's are a great solution for this
for a vast number of problems, and the Stanford lecture is great.

The thing about Posits in particular is, depending on the environment (e.g.
es=3 for 32 bit Posits as referenced in the Beating Floating Point paper), you
can actually have greater accuracy/precision and dynamic range than 64 bit
IEEE floats while using fewer bits. The OP article here about CERN is doing
something significantly worse, which is just using lower precision and taking
the accuracy loss. While this may be acceptable in some applications, I would
not want to losing any accuracy in the original data from expensive scientific
experiments and using potentially inaccurate results for future calculations
and assumptions.

------
tehsauce
"last month after running experiments in which particles within the LHC
detectors collide about 1 billion times per second, with each collision
generating a petabyte of data."

1 billion petabytes per second haha

------
tempay
I feel like this article is confusing numerical precision with "precision
measurements". The latter typically means that the focus has moved from
discovery of a given process, to measuring its properties with high precision,
where high precision at LHC is often an uncertainty below 1%.

------
kwrobel
Also in NLP domain a precision reduction may be applied:
[https://arxiv.org/abs/1706.06363](https://arxiv.org/abs/1706.06363)

