
High Performance Computing: Are We Just Getting Wrong Answers Faster? (1998) [pdf] - collapse
https://www3.nd.edu/~markst/cast-award-speech.pdf
======
GlenTheMachine
I think it's worse than this, at least in some fields. In my field, aerospace
engineering, the undergraduate curriculum typically has a numerical methods
course. But the course is usually focused on basic algorithms — Newton's
method, basic numerical integrqtion schemes, etc. — and does not go deeply
enough to allow one to derive or understand the numerical methods used in the
design of actual aircraft or spacecraft (FEM, solutions to the Navier Stokes
equations, solutions to the PDEs that describe thermal transfer, etc.) so the
software that performs these calculations is often treated by engineers as a
black box. Admittedly, in a well run project there is usually hardware testing
to back up the analysis, but such testing cannot realistically reproduce the
conditions a spacecraft will encounter on orbit, or that an aircraft will
encounter in near-crash situations.

I once sat on the external departmental review board for my alma mater, along
with Mike Griffin (former administrator of NASA). We were interviewing recent
graduates, and I brought this point up and got the interviewee to admit that
he did not understand and could not double check the results of his FEM
analysis tool. I was trying to get the board to rcommend beefing up the
numerical methods portion of the curricula. Dr. Griffin, on hearing the
interviewee admit this, agreed to add the recommendation. But nothing changed
in the curriculum.

~~~
noobermin
A lot of code is treated like a black box. For example, can you describe me
how ssh works other than schematically? Do you know the details of the
cryptographic bits? What you asked for in terms of the fundamental numerical
algorithms like FEM is somewhat close to this. They hopefully (especially if
they are trying to get a graduate degree) should be able to schematically
describe it, but in no way should every scientist have to reinvent FEM or
finite difference methods again and again just like people shouldn't implement
their own crypto: it can be rather hairy and making a mistake could be
disastrous. Writing complicated algorithms should be left to experts who can
be trusted.

~~~
a1369209993
> Writing complicated algorithms should be left to experts who can be trusted.

How exactly do you propose we _get_ those experts, if colleges aren't capable
of training them?

~~~
wenc
> How exactly do you propose we get those experts, if colleges aren't capable
> of training them?

In the majority of cases, colleges (undergrad) aren't in fact capable of
training them. The material is too specialized for undergrad. However colleges
can prepare those who are talented and interested in the subject matter for
further studies.

Many experts in numerical analysis start specializing in graduate school, and
then acquire their expertise through postdocs, academia and industry
experience.

------
nestorD
Note that there are at least three ways to get the wrong answer with the
correct code :

\- Modeling and discretization errors (usually unavoidable but well
understood).

\- Uncertainties from the inputs (we have tools to get a feeling for their
impact on the result).

\- Floating point arithmetic (scientist tend to be unaware of its impact
and/or consider that it is negligible next to the other sources of error)

My PhD covers methods to quantify the impact of floating-point arithmetic and
suffice to say that you probably grossly overestimate the number of
significant digits of your simulation. On the subject I recommend "What Every
Computer Scientist Should Know About Floating-Point Arithmetic" :
[https://www.itu.dk/~sestoft/bachelor/IEEE754_article.pdf](https://www.itu.dk/~sestoft/bachelor/IEEE754_article.pdf)

~~~
btrettel
"Model inadequacy", i.e., the model being wrong, is also a major source of
error.

In my training, it was basically assumed that floating point errors are
negligible these days, so I'll check out your recommendation. If you've
published any of your own more recent work, I'd be happy to see that as well.

~~~
mehrdadn
> In my training, it was basically assumed that floating point errors were
> negligible these days

Yeah, that's what most of us think initially. The problem is that the effects
of error can compound in many ways -- nonlinearities, feedback, etc. can all
wreck your accuracy enough to make the results meaningless.

A really instructive counterexample for me was solving quadratics. Easy,
right? Just try writing a solver for ax^2 + bx + c = 0. Now try breaking it.
If all you did was use the quadratic formula, you should find that it's quite
easy to break it, and surprisingly difficult to write a robust solver. (If
you're completely stuck, consider the most obvious edge case: what if a ≈ 0?
Any other scenarios you can think of?)

------
btrettel
Interesting paper. I work in fluid dynamics and tend to agree that
computational power often is wasted.

To elaborate on a point not discussed in the paper: Many methods used in
practical engineering won't necessarily converge to the empirically correct
answer as the computational cost increases, say, by decreasing the time step,
increasing the number of computational particles, and/or refining a grid. This
is true even if the uncertainty is quantified. Without this guarantee,
additional computational expenditure could simply be wasted. The problem isn't
the lack of computing power, it's the model.

One reason is that "reliable" physics like, for example, the Navier-Stokes
equations for fluid flows, are not being used as the model because the
computational cost for solving those equations directly is far too high.
Instead lower cost models are being used which may be loosely based on the
reliable model, but ultimately is not the same thing. A numerical
implementation of the lower cost model might converge to something (which is
not necessarily right) as the computational cost increases, but converging to
the right answer is less likely than the reliable model. It depends on the
reliability and generalizability of the lower cost model and how well it has
been tuned.

(To those familiar with computational fluid dynamics, convergence is one
reason to prefer LES to RANS, though it's worth pointing out that not all LES
methods are convergent.)

A hypothesis I have is that computing resources have caused many scientists
and engineers' theoretical abilities to atrophy. Perhaps they were never that
good on average to begin with, but in my academic field I work in theory
partly because I've found so much low hanging fruit. I don't see much of a
point in making expensive computational models if applying basic regression
techniques with a simple theory-based model is similarly or even more
accurate. These simple models are much more efficient.

Also: I wasn't aware of attempts to bound the results. I attended an
uncertainty quantification seminar before and asked one of the lecturers about
this more generally and they seemed skeptical of its utility.

~~~
snaky
As Vladimir Arnold, who used to know a thing or two about fluid dynamics in
particular, once said

> At this point a special technique has been developed in mathematics. This
> technique, when applied to the real world, is sometimes useful, but can
> sometimes also lead to self-deception. This technique is called modelling.
> When constructing a model, the following idealisation is made: certain facts
> which are only known with a certain degree of probability or with a certain
> degree of accuracy, are considered to be "absolutely" correct and are
> accepted as "axioms". The sense of this "absoluteness" lies precisely in the
> fact that we allow ourselves to use these "facts" according to the rules of
> formal logic, in the process declaring as "theorems" all that we can derive
> from them.

> It is obvious that in any real-life activity it is impossible to wholly rely
> on such deductions. The reason is at least that the parameters of the
> studied phenomena are never known absolutely exactly and a small change in
> parameters (for example, the initial conditions of a process) can totally
> change the result. Say, for this reason a reliable long-term weather
> forecast is impossible and will remain impossible, no matter how much we
> develop computers and devices which record initial conditions.

[https://www.uni-muenster.de/Physik.TP/~munsteg/arnold.html](https://www.uni-
muenster.de/Physik.TP/~munsteg/arnold.html)

~~~
narimiran
>> _a small change in parameters (for example, the initial conditions of a
process) __can__ totally change the result. Say, for this reason a reliable
long-term weather forecast is impossible and will remain impossible_

(emphasis mine)

A small change in initial conditions _can_ totally change the result in
chaotic systems such as weather forecast, or a much simpler example — double
pendulum.

There are systems where a small change changes the results only slightly. E.g.
a single pendulum.

------
triska
In this context, I also highly recommend John Gustafson's book, _The End of
Error_. It is full of great examples, and at one point contrasts advances in
numerical computing with improvements in other areas:

 _" In 1970, a printer might produce something that looks like this, and take
about 30 seconds to do so: [picture] Over forty years later, a laser printer
still might take 30 seconds to put out a single page, but technology has
improved to allow full-color, high-resolution output."_

The book makes a compelling argument that working with floating point numbers
is similar to printing thousands of low-quality pages per second, instead of
getting a single high-quality page in 30 seconds. A few paragraphs later, the
book states:

 _" Eventually it becomes so obvious that change is necessary that a wrenching
shift finally takes place. Such a shift is overdue for numerical computing,
where we are stuck with antediluvian tools. Once we make the shift, it will
become possible to solve problems that have baffled us for decades. The
evidence supporting that claim is here in this book, which contains quite a
few results that appear to be new."_

For more information, see:

[https://www.crcpress.com/The-End-of-Error-Unum-
Computing/Gus...](https://www.crcpress.com/The-End-of-Error-Unum-
Computing/Gustafson/p/book/9781482239867)

and John Gustafson's home page:

[http://johngustafson.net/](http://johngustafson.net/)

~~~
jcranmer
unums are at best very controversial. In actuality, I don't think I've heard a
numerical analyst (save for Gustafson) come out in favor of them.

The original proposal that Gustafson had involved a variable-width format,
which is going to absolutely kill performance in practice (particularly on the
GPU-based supercomputers that are all the rage nowadays). The posit format was
essentially a recognition that variable-width doesn't work.

The more serious error is that, well, interval arithmetic turns out not to be
useful in practice. Essentially, the potential accumulation of error is
exponential in the number of steps in the worst case, so interval arithmetic
in practice too often gives you the answer "the result is 1, but it could have
been anywhere between -inf and +inf", which is completely useless. He may
complain that he doesn't want to use a numerical analyst to get work done, but
the rebuttal is that numerical analysis is still necessary anyways, so
promising that one isn't necessary is a false promise.

~~~
zlynx
At least with (-inf, +inf) you know that the answer is nonsense and that you
need to go back to the algorithm to find a better order of operations.

With ordinary floating point, all too often people don't realize their results
are nonsense. Adding measurement and computation error bounds to their graphs
is illuminating.

~~~
triska
This is indeed a key point in John Gustafson's arguments: With floats, even
though there are specialized flags and values that let you detect certain
problematic situations, these flags are often not readily accessible in
programming languages, and hence these issues tend to go unnoticed.

~~~
jcranmer
> these flags are often not readily accessible in programming languages

Unless you use C, C++, or Fortran. Which accounts for most numerical
algorithms code. Yes, most people aren't aware of them, but not being aware of
all the functions in the standard library isn't the same as them not being
present there.

