
How Did Software Get So Reliable Without Proof? (1996) [pdf] - gwern
http://www.gwern.net/docs/1996-hoare.pdf
======
gefh
It turns out software was only reliable in the dimensions we we carefully
looking at. I'm not sure 'proof' would have helped any - we might well have
proved that, say, Flash could successfully display any valid animation you
could throw at it without even thinking about whether it was also pwning your
machine.

~~~
lmm
Most of the security vulnerabilities involve C undefined behaviour. Any
proven-correct program necessarily won't invoke undefined behaviour.

(We could also have used memory-safe languages)

~~~
gefh
Many, but I'm not sure most. However they nearly all involve someone (be it
the language designer, compiler writer or application writer) not thinking
about all the malicious possibilities of the code in question. Because that's
not the problem they're trying to solve right now, and they're only human. And
I only use security vulnerabilities as an example - the next thing we're all
not thinking about might be power consumption, or creating AI paradoxes, or
whatever. But we're definitely not thinking about it, and no-ones going to be
proving it.

~~~
lmm
> And I only use security vulnerabilities as an example - the next thing we're
> all not thinking about might be power consumption, or creating AI paradoxes,
> or whatever. But we're definitely not thinking about it, and no-ones going
> to be proving it.

I don't think you have to be thinking about it. Every time some new class of
vulnerabilities has arisen, e.g. SQL injection, it just hasn't affected well-
formed programs written with good tools, and certainly not verified programs.
Correct programs just don't have these problems, as a side effect of being
correct.

------
DawkinsGawd
I haven't gotten a chance to read the entire paper however the reference to
the Therac-25 in the beginning was slightly tilted. Yes - it was a
programmatic error that dispersed a deadly amount of radiation. But, if I
remember correctly, the Therac-25 is often used in Software/Computer
Engineering ethics classes as an instance in which a hardware safety mechanism
would be appropriate. So while the article is right that a software
malfunction did disperse a deadly amount of radiation, the fix for this isn't
necessarily a software solution

------
intrasight
How did living things get so reliable without proof?

~~~
pmiller2
Living things aren't very reliable. They don't last very long, in general, and
their replication mechanism is highly error-prone.

~~~
humbledrone
That's an extremely narrow view of life. A broader view might be that life on
Earth began a few billion years ago, and has managed to hang on ever since. Do
you realize that you are, most likely, the direct descendant of one of those
early lifeforms? Every single one of your ancestors, for billions of years,
has managed to produce viable offspring in an unimaginably long chain of
living organisms. How can you call that unreliable? Sure, bits of your DNA are
being corrupted every second. Sure, tens of billions of your human cells die
and are cast away each day. And, sure, each agglomeration of cells we call a
human only lasts a few tens of years. But life has enough redundancies,
contingency plans, and diversity that it will in all likelihood keep going for
a long, long time.

Having a single instance of a particular organism last for a long time just
isn't what life is about. If you look at the full context of what life is
doing, it is way goddamn more reliable than anything humans have designed,
proofs or not.

~~~
pmiller2
I agree with your argument as far as it goes, but if you're responding to me,
you're making a straw man argument. I was saying that _living things_ ( _i.e._
individual organisms) aren't very reliable; you're saying that _life_ is very
resilient. That's a true statement, but utterly non-responsive to the point I
was making.

~~~
taneq
Individual organisms are incredibly reliable when you consider the range of
functions they perform. Compare, say, a quadcopter drone (or any other small
air vehicle) with a parrot.

Lifespan without service or repair? Drone, weeks. Parrot, decades.

Autonomy before recharging/refueling? Drone, 20 minutes. Parrot, hours to
days.

Cognitive faculties? Drone, slim to none. Parrot, long term autonomous in
complex environments.

Reproduction? We can't build things that do that _at all_ yet, let alone pack
one into a 1kg autonomous aircraft.

Yes, we can build simple machines to replicate one function of a living
creature, and those machines will be more reliable over short durations, but
only because they are orders of magnitude simpler. You might as well call an
LCD screen "unreliable" because it won't outlast a photograph.

------
taneq
"Beware of bugs in the above code; I have only proved it correct, not tried
it." \- Donald Knuth

~~~
seanwilson
Surely code you haven't proved correct is going to have way more bugs in it
than code you tried to prove is correct? Maybe I'm misunderstanding but this
quote is always posted as if it's some kind of rebuttal against formal
methods.

~~~
DawkinsGawd
I always thought that correctness (in computer science) only asserted that an
algorithm acted as specified. For example, say the specification was: The
input accepts an element in the set of real numbers, and the corresponding
output is that number added to the next element in the set of natural numbers.

If this algorithm is used to dispense some type of medication, and I enter 3.1
and it murders you with 10 grams of alprazolam... well.... the algorithm is
still correct.

~~~
seanwilson
> I always thought that correctness (in computer science) only asserted that
> an algorithm acted as specified. For example, say the specification was: The
> input accepts an element in the set of real numbers, and the corresponding
> output is that number added to the next element in the set of natural
> numbers.

Nobody is saying the formal proof approach will find all bugs, just that it
helps find more bugs compared to not using it. You could make the same
argument with acceptance testing: unless you specifically tried 3.1 as the
input during testing, it might not give you the output you expect. Formal
proof can be complimentary to traditional automated tests and QA.

------
plugnburn
Systems for processing highly classified governmental/military confidential
information often are required (depending on the country laws) to provide
formal proof of their entire work. Other than that (and because it is too
expensive to run such an analysis) people just need to look at the code to
check whether it's doing what it's supposed to do.

------
amelius
> How Did Software Get So Reliable

Lol!

------
pron
> _Specialisation involves a deep commitment to a narrow selection of
> presentation, reasoning methods, paradigm, language and application area, or
> even a particular application. The whole point of the specialisation in
> formal methods is to restrict the notational framework as far as necessary
> to achieve some formal goal, but nevertheless to show that the restrictions
> do not prevent successful application to a surprisingly wide range of
> problems. This is the reason why specialist research into formal methods can
> run the risk of being very divisive. An individual researcher, or even a
> whole community of researchers, becomes wholly committed to a particular
> selection of specialisations along each of the axes: say an operational or
> an algebraic presentation of semantics, bisimulation or term rewriting as a
> proof method, … The attraction of such a choice can be well illustrated in
> certain applications, such as the analysis of the alternating bit protocol
> or the definition of the stack as an abstract data type. The perfectly
> proper challenge of the research is to push outwards as far as possible the
> frontiers of the convenient application of the particular chosen formalism.
> But that is also the danger: the rush to colonise as much of the available
> territory can lead to imperialist claims that deny to other specialisms
> their right to existence. Any suggestion of variation of standard dogma is
> treated as akin to treason. This tendency can be reinforced by the short-
> sightedness of funding agencies which encourage exaggerated claims to the
> universal superiority of a single notation and technique._

> _The consequences of the fragmentation of research into rival schools is
> inevitable: the theorists become more and more isolated, both from each
> other and from the world of practice, where one thing is absolutely certain:
> that there is no single cure for all diseases. There is no single theory for
> all stages of the development of the software, or for all components even of
> a single application program. Ideas, concepts, methods, and calculations
> will have to be drawn from a wide range of theories, and they are going to
> have to work together consistently, with no risk of misunderstanding,
> inconsistency or error creeping in at the interfaces. One effective way to
> break formal barriers is for the best theorists to migrate regularly between
> the research schools, in the hope that results obtained in one research
> specialisation can be made useful in a manner acceptable by the other. The
> interworking of theories and paradigms can also be explored from the
> practical end by means of the case study, chosen as a simplified version of
> some typical application. In my view, a case study that constructs a link
> between two or more theories, used for different purposes at different
> levels of abstraction, will be more valuable than one which merely presents
> a single formalisation, in the hope that its merits, compared with rival
> formalisations, will be obvious. They usually are, but unfortunately only to
> the author._

~~~
lmm
> There is no single theory for all stages of the development of the software,
> or for all components even of a single application program. Ideas, concepts,
> methods, and calculations will have to be drawn from a wide range of
> theories, and they are going to have to work together consistently, with no
> risk of misunderstanding, inconsistency or error creeping in at the
> interfaces.

I'm not convinced. I remain hopeful that one or other technique will show
itself adequate for complete coverage of real-world programs. Compare
programming languages - naïvely one would think different languages have
different specializations and it's important for programmers to be fluent in a
wide variety of them and good at interfacing between different languages. But
I'm increasingly coming to believe that good general-purpose languages are
possible and that better systems are built in a single language, by
programmers specialized to that language.

~~~
pron
Ah, but there's a difference between formal verification and programming
languages. The problem of general formal verification (known in general as the
model-checking problem[1], regardless of the approach used which could be
model-theoretic or proof-theoretic) is literally the hardest problem in all of
computer science. Any computational problem can be reduced in polynomial time
to program verification (i.e. model checking). In fact, all NP-complete and
PSPACE-complete problems are trivially reduced. Finding a general tractable
solution (even if it includes human intelligence) is therefore a mathematical
impossibility. The best we hope for is a solution that is applicable to
programs people write. The question is then whether there is one well defined
"class" of programs humans write to which an efficient algorithm (again --
even one involving humans in the loop) applies. So far results from decades of
verification research show that this is not the case, and there are at least
several such classes, but maybe the classes aren't general enough. If humans
are part of the algorithm (e.g. interactive proofs, manual dependent types),
the class might be wide, but our experience has shown that verifying programs
in this way takes tremendous effort, even for small programs, and even when
the programs are kept especially simple for the sake of verification. Even if
there is a solution that is applicable to most (I'll settle for most)
programs, it is so far nowhere in sight. Most efforts are directed at finding
partial solutions for specific domains.

[1]: Called that because it is the problem of _checking_ whether a given
program is a satisfying theory -- i.e. a _model_ \-- for the given logical
property.

~~~
lmm
That it's a hard problem doesn't necessarily mean we're approaching it wrong.
Indeed the fact that it's mathematically proven to be hard offers some
confidence that we might not be.

Are there examples of cases where a system was very hard to verify with any
one general technique, but much more practical to verify by using several
domain-specific techniques and some kind of bridge between them (i.e. not
simply coming up with a general approach for which the specialized techniques
were, well, specializations - since as far as I can tell your quote is
advocating for something different)?

~~~
pron
Sure. In fact, I believe this is the norm rather than the exception. Software
systems often use model-checkers, abstract interpretation and manual deduction
to be verified. Each have their strengths and weaknesses. In fact, you can see
all three methods used in one of the common representations of logic (which
I've picked because I know you're a Scala guy): types. Simple types (like H-M
types) perform crude abstract interpretation, but can only prove properties of
finite state machines. Liquid types use model-checking algorithms to infer and
check some kinds of dependent types. These, however, still don't cover all
properties, so in practice, people need to resort to manually-inferred/checked
dependent types. Using those alone would be extremely cost prohibitive.

In the safety-critical software world, you see projects using model checkers
to verify individual modules and deductive proofs (say, in Isabelle) to verify
the interaction among modules or vice-versa. In the less-safety-critical
software world, you see people commonly use static analyzers (employing
abstract interpretation), even though those can only verify certain
properties, because both interactive deductive proof and automatic model
checking are too cost prohibitive.

But the problem in general is this: we are not at a point where any single
approach nor a combination of several lets us fully verify a real-world
program even of moderate side at any acceptable cost. I don't believe any
moderately-sized program has ever been fully verified using any technique.

~~~
lmm
> But the problem in general is this: we are not at a point where any single
> approach nor a combination of several lets us fully verify a real-world
> program even of moderate side at any acceptable cost. I don't believe any
> moderately-sized program has ever been fully verified using any technique.

Sure. Which means I think we simply don't know enough to say one way or the
other what kind of approach we will need for effective proofs. Maybe we will
need to combine multiple techniques and have bridges between them. But maybe
we really will be able to extend one particular technique to cover all the
important cases, or find a single general approach that recovers all the
useful techniques we've found so far as special cases.

~~~
pron
Right. But it is important to understand that this is the hardest problem in
computer science (human-level AI is much easier), and so general-enough
solutions must somehow be aligned with how humans tend to construct programs
(because a completely general solution is impossible, and solutions that are
not aligned with how people program can only hope to solve human-produced code
by accident). For example, we do know that we cannot create a general-purpose
language that will make verification easy for all programs written in it; that
is a mathematical impossibility. Whatever approach may work one day, it will
have to be based on empirical observations of programs people tend to write.

~~~
lmm
> For example, we do know that we cannot create a general-purpose language
> that will make verification easy for all programs written in it; that is a
> mathematical impossibility.

Not true. We know that we cannot create a Turing-complete language with that
property. We haven't proven that we'll never be able to create a general-
purpose non-Turing-complete language. I might argue that Idris is already an
example of such a thing.

~~~
pron
> We haven't proven that we'll never be able to create a general-purpose non-
> Turing-complete language.

Well, if you're referring to total languages, then they are Turing complete
for all intents and purposes.

Total languages are not immune to the halting problem (or, rather, to its
finite generalizations known as "bounded halting"). The classical formulation
of the halting problem simply tells you that verifying a program in general is
undecidable. But this is simply because a Turing machine is a finite structure
that can describe an infinite one. So in some way, the halting problem is
undecidable in the same way that finding the maximum or sorting an infinite
list doesn't terminate. Yet once the list is finite (or the Turing machine is
known to terminate), you then need to ask yourself how hard is it to find the
maximum element in a finite list, how hard is it to sort it, and how hard is
it to verify a program that is known to terminate? The answers to all these
questions are well known, and quite different from one another. Finding the
maximum takes Ω(n) (worst-case), sorting is Ω(n lg n), and verification is
Ω(|S|), namely at least linear in the number of states the program takes until
it terminates (this directly follows from a trivial generalization of the
halting problem called "bounded halting"). So sure, total programs are known
to terminate, and all we have left to do now to prove them correct is to
overcome a super-exponential lower complexity bound... In practice that is
just as undecidable as for programs that are not proven to terminate.

And, BTW, the lower complexity bounds for verification of finite state
machines written in expressive languages is already PSPACE-complete. There
cannot be a useful _general_ programming language with verification bounds
that are less than exponential (unless, that is, PSPACE=P); but that is a good
thing: verification of total-functional programs is super-exponential. So my
point is that even completely general verification of freakin' finite state
machines is out of our reach in large-enough programs. It's doubtful you can
find a useful computational model that's simpler.

Also, notice that any program could trivially be made total without changing
its semantics. Give me a program you want verified, I'll add a decreasing
counter starting with 10100 to all its loops/recursions, and now your program
is total and you'll never know the difference. Does that make it any easier to
verify?

The total-language people know this. They don't claim that this makes
practical program verification any easier. In fact, the motivation for
totality has nothing to do with program verification but with using C-H to
prove general mathematical theorems. Non-termination, as you know, is the same
as inconsistencies in those proofs (although not in proofs about programs;
non-termination introduces no inconsistency to those). However, some people
who are only familiar with the classical formulation of the halting theorem
have concluded that totality => no halting problem => verification is
possible! This is false, and was abundantly clear even in the 50s. Kurt Gödel,
in a letter to von Neumann in 1956, mentions in passing this finite (and
obvious) generalization of the halting theorem before raising (for the first
time!) the question of P vs. NP.

~~~
lmm
> unless, that is, PSPACE=P

Which of course remains an open question.

> even completely general verification of freakin' finite state machines is
> out of our reach in large-enough programs. It's doubtful you can find a
> useful computational model that's simpler.

Eh, maybe. I find it very hard to reason about finite state machines in
programs as a human, so it doesn't surprise me that proving correctness
properties about them would be hard. I'm not at all convinced that a general-
purpose programming language necessarily includes support for fully general
finite state machines.

> Also, notice that any program could trivially be made total without changing
> its semantics. Give me a program you want verified, I'll add a decreasing
> counter starting with 10100 to all its loops/recursions, and now your
> program is total and you'll never know the difference. Does that make it any
> easier to verify?

Yes - it eliminates the class of errors where one has "implemented" something
by an infinite loop. It helps for concrete programs in the same way that it
helps for theorems, no?

~~~
pron
> Which of course remains an open question.

Technically yes, but as Scott Aaronson says, a universe where P=NP would
probably look very different from ours (and PSPACE is harder than NP). In any
event, verifying effectively Turing-complete languages (even total ones) is
much harder than EXPTIME (and we know that EXPTIME > P).

> I find it very hard to reason about finite state machines in programs as a
> human

I didn't mean it like that (programming with explicit states). Just a model of
computation where the number of states is bounded. Any program with finite
memory is a finite state machine. Every language must support at least that.

> it eliminates the class of errors where one has "implemented" something by
> an infinite loop. It helps for concrete programs in the same way that it
> helps for theorems, no?

No, because when you prove a _program_ property A you know that you've really
proven "A or no-termination". So being able to prove any property is not an
inconsistency. This is not the case for mathematical theorems that don't have
running time, and where "no termination" is meaningless. If running time is 0,
then A just means A, and if A is a contradiction then you have a real problem.

Now, what about eliminating non-termination from programs? Well, you're not
really doing that either, because as far as we're concerned, a program that
takes 10^100 steps doesn't really terminate at all. One could claim that
people are more likely to accidentally write a function that loops forever
than one that loops 2^64 times. This may be true, but this again relies on
empirical facts rather than general mathematical properties.

In any case, what makes you think that languages like Idris may make
verification easy? The few cases of programs verified in this mostly-manual
way took _years_ to prove, even for very small and intentionally "dumbed down"
code.

~~~
lmm
> I didn't mean it like that (programming with explicit states). Just a model
> of computation where the number of states is bounded. Any program with
> finite memory is a finite state machine. Every language must support at
> least that.

Any program is (or can be modeled as) a finite state machine, sure. But useful
programs are a small subset of the space of possible finite state machines.
It's not beyond hope that we could find a subset that allowed us to write most
useful programs but was easier to reason about.

> One could claim that people are more likely to accidentally write a function
> that loops forever than one that loops 2^64 times. This may be true, but
> this again relies on empirical facts rather than general mathematical
> properties.

I am worried about not-terminating-in-reasonable-time (though much less
worried than I am about looping forever), but can't we just make that one of
the properties we keep track of and prove with the same tools?

> In any case, what makes you think that languages like Idris may make
> verification easy? The few cases of programs verified in this mostly-manual
> way took years to prove, even for very small and intentionally "dumbed down"
> code.

I guess my experience that keeping track of more properties in the type system
doesn't make coding any harder. Compare Rust - writing Rust programs
essentially means proving that things have valid lifecycles and are not used
outside of them. This is a complex property that mainstream programmers were
not previously proving at all - that many had despaired of ever being able to
prove. And yet in practice people who write Rust find that not only can they
do it, but the additional proof effort quickly becomes barely noticeable.
Likewise with Idris you're proving termination in finite (though possibly
unreasonable) time and once you get used to doing that it doesn't actually
seem noticeably harder than not doing so. So I'm kind of hopeful that we can
just keep improving the type systems used by mainstream programming languages
until writing a program naturally includes proving the important properties of
that program without even noticing.

~~~
pron
> It's not beyond hope that we could find a subset that allowed us to write
> most useful programs but was easier to reason about.

It is beyond hope that we find a subset that's _generally_ easier to reason
about. Verifying one line programs of this sort:

    
    
        if (a && ~b || c || ~d && e ...)
          crash()
    

is already NP-complete (SAT). Adding some loops that are of depth 2 makes it
PSPACE-complete (TQBF, aka QSAT). I can't think of a useful programming
language that doesn't have even small-constant-depth loops (which makes it
finite state) and simple boolean algebra. Software verification is about
verifying some subsets of programs that humans happen to write. It is
categorically impossible to write a general-purpose language where _all_
programs are easier to verify than PSPACE.

> but can't we just make that one of the properties we keep track of and prove
> with the same tools?

Sure, with super-exponential complexity :) But again, this is in the worst
case. We hope that humans _usually_ write programs that are simpler.

> I guess my experience that keeping track of more properties in the type
> system doesn't make coding any harder.

If you mean H-M types as in Haskell, then the properties those types enforce
are either trivial (i.e. true for all programs, e.g. function returns Int), or
a result of a finite state machine (that's the proving strength of H-M types).
You cannot prove even a single bit of a program's output that's a result of
non-constant-depth recursion (there's a special case for recursion which is of
depth that's a constant multiple of the input length, like `map`). If we take
Haskell, you can replace each of your functions with constant-depth recursion
(0 or 1 would suffice in virtually all circumstances) and the program will
still typecheck yet will be completely wrong.

As for Rust's linear types, that is precisely what I'm talking about. A
certain class of bugs (memory errors) that humans are known to make and that
verifying with abstract interpretation is easy. _That 's_ what software
verification aims to do. We cannot hope of proving general programs generally
correct, regardless of the language they're written in.

BTW, thinking of type systems muddles understanding in this case, because type
systems are just notation. The verification taking place in most sound type
systems is abstract interpretation. Yes, we can find more important properties
to verify with abstract interpretation, but it has been proven that some
properties will forever elude abstraction.

The way abstract interpretation works is that it translates the program into
an "abstract" program which is just a finite state machine, with the number of
states equal to the number of concrete types, and running an exhaustive model
checker on that; this means that the complexity of inferring this kind of
types is Ω(|S|), namely, linear in the number of concrete types. As abstract
types can multiply the number of concrete types, the inference can become
exponential in the worst-case, but it usually doesn't. Once you introduce
dependent types, the algorithm is still Ω(|S|) -- linear in the number of
state, i.e. types -- but now the number of types is unbounded, and so the
problem becomes undecidable.

> until writing a program naturally includes proving the important properties
> of that program without even noticing.

You're right, but this entirely depends on which properties are found to be
important (and has little to do with type systems specifically; again, this is
mostly notation[1]), and that, in turn, is rather empirical.

[1]: Not entirely, though, but mostly.

~~~
lmm
> I can't think of a useful programming language that doesn't have even small-
> constant-depth loops (which makes it finite state) and simple boolean
> algebra. Software verification is about verifying some subsets of programs
> that humans happen to write. It is categorically impossible to write a
> general-purpose language where all programs are easier to verify than
> PSPACE.

Eh. Again, I don't think we necessarily need arbitrary boolean algebra -
something that's hard enough to reason about as a human - and may be able to
capture the valuable use cases in a simpler model. Fundamentally if we have
any hope of doing verification at all, surely we can characterize the
constraints we need for verification and then push them back to the
programming language level.

> If we take Haskell, you can replace each of your functions with constant-
> depth recursion (0 or 1 would suffice in virtually all circumstances) and
> the program will still typecheck yet will be completely wrong.

Could you expand on this / give an example? I don't think I follow. And I take
it from what you're saying that this doesn't apply if we permit dependent
types?

> BTW, thinking of type systems muddles understanding in this case, because
> type systems are just notation. The verification taking place in most sound
> type systems is abstract interpretation. Yes, we can find more important
> properties to verify with abstract interpretation, but it has been proven
> that some properties will forever elude abstraction.

Properties that are important for real-world programs? Abstract interpretation
sounds very similar to how we reason about programs as humans. It just seems
like to the extent that we understand how to do something at all we should be
able to formalize that understanding.

If we're talking about using programs to address open questions - a best-
effort at the travelling salesman problem, say - then I can see how you might
end up with a program that had a pile of heuristics that seemed to work but
with no concrete understanding of how or if it worked - except that even then
you'd probably use a well-known, formally validated heuristic rather than
using the program as a medium for mathematical research. I just think
(empirically) it would be very rare as a business requirement to need to do
something completely unproven, and therefore it might be possible to have a
useful general-purpose programming language that didn't allow doing that.

~~~
pron
> Fundamentally if we have any hope of doing verification at all, surely we
> can characterize the constraints we need for verification and then push them
> back to the programming language level.

That is not what the verification community is trying to do at all, because
trying to find a language that is not even NP-complete to verify and is still
_generally_ is absolutely impossible (explanation later). What the
verification community is trying to do is to come up with approaches that make
it possible/easy to verify _some_ code, which is hopefully the majority of
code. As far as languages go, the hope is that a language can make some
verification of some code easier than in other languages. A general-purpose
language that is at least useful in some cases, and in which _all_ programs
are easily verifiable can only exist if P=NP. The reason is this: at best, you
can hope that the proof of correctness is checkable in polynomial time. This
puts it in NP. If _constructing_ a proof of correctness is possible in
polynomial time, so is coming up with the solution in the first place. Namely,
such a language wouldn't even need to be programmed. It will be a constraint
solver (that somehow is in P). SQL minus boolean expressions might be the best
you could do. Possibly useful, but certainly not general-purpose.

> Properties that are important for real-world programs?

Yes, and they are quite common in distributed algorithms.

> It just seems like to the extent that we understand how to do something at
> all we should be able to formalize that understanding.

Yet we can't because when we reason about something we assume a certain
regularity of the state-space, that is often wrong in practice. That link I
showed you where there are visualizations of state spaces, shows the one for
the CAN bus protocol. It is the most complicated example on that page, and it
is the very real protocol used by different components in your car that need
to communicate.

> I just think (empirically) it would be very rare as a business requirement
> to need to do something completely unproven

That depends a lot on what you do. If you design schedulers for OS kernels,
concurrent data structures, or distributed algorithms you see that these
difficult cases are the rule rather than the exception.

I work on concurrent and distributed algorithms, and I do formal verification
(TLA+ is my current tool of choice) most day every day. TLA+ offers both a
theorem prover (a la Isabelle/Coq/Agda) as well as a model checker. Manual
deductive proofs are virtually never worth the trouble (that would take me
years), and even with a model checker -- where you just press a button and if
there's a bug, any bug, you get a full trace of the execution that led to the
bug -- it's still a lot of work, because distributed algorithms are just
essentially complex.

------
seanwilson
The paper is talking about rigorous formal mathematical proofs of correctness
like in the sense of of the seL4 kernel
([https://sel4.systems/Info/FAQ/](https://sel4.systems/Info/FAQ/)) where they
have confirmed the kernel is completely free of bugs such as buffer overflows,
null pointer exceptions, use-after-free, etc.

~~~
ktRolster
Of course, there might be bugs in the specification (the proof the made was
that the kernel matches the specification)

~~~
seanwilson
> Of course, there might be bugs in the specification (the proof the made was
> that the kernel matches the specification)

Nothing is perfect, but this approach catches more bugs than any other
technique. Also, some specifications are simple to specify e.g. it's not hard
to specify that an array index must be within the bounds of the array and if
you did get this wrong you'd notice quickly as you'd be unable to prove
obvious facts are correct.

~~~
ktRolster
_but this approach catches more bugs than any other technique_

Have you seen the specification for sel4? It's fairly complex.

------
Retra
All programming is proof. You just don't often know what exactly it is you're
proving.

Anyway, the point is moot: software is not that reliable. It's good at the
common case, and is very brittle at the edge cases. That's my experience
anyway.

~~~
trentmb
I was a math major and while trying to get a job in programming I got asked
how I felt my skills in mathematics would transition to programming-

The language (spec) is my set of axioms, the standard library existing
theorems, and my program the proof.

It's kinda drab now that I think about it, but for a 'firing from the hip'
answer I still stand by it.

~~~
qohen
Not so drab -- that sort of thinking can lead to a pretty big deal, namely,
the Curry-Howard Isomorphism[0]. From the linked WP page:

 _In other words, the Curry–Howard correspondence is the observation that two
families of formalisms that had seemed unrelated—namely, the proof systems on
one hand, and the models of computation on the other—were, in the two examples
considered by Curry and Howard, in fact structurally the same kind of objects.

If one now abstracts on the peculiarities of this or that formalism, the
immediate generalization is the following claim:_ a proof is a program, the
formula it proves is a type for the program. _More informally, this can be
seen as an analogy that states that the return type of a function (i.e., the
type of values returned by a function) is analogous to a logical theorem ...
and that the program to compute that function is analogous to a proof of that
theorem. This sets a form of logic programming on a rigorous foundation:
proofs can be represented as programs, and especially as lambda terms, or
proofs can be run._

(Of course, in the context of those job interviews you mentioned, this sort of
thing would probably be overkill, unless the job involved functional-
programming...in which case, though, they might well not have asked you how
your "skills in mathematics would transition to programming" in the first
place :-) ).

[0]
[https://en.wikipedia.org/wiki/Curry–Howard_correspondence](https://en.wikipedia.org/wiki/Curry–Howard_correspondence)

