
Formal Verification Creates Hacker-Proof Code - azuajef
https://www.quantamagazine.org/20160920-formal-verification-creates-hacker-proof-code/
======
ahelwer
Worth noting formally verified programs are unhackable to the extent three
things hold true:

1) Your model of the world corresponds to the world in which the program runs
- for example, hardware is usually assumed to be bug-free.

2) The properties you've proven the program to hold are sufficient to stop the
hacker from accomplishing their goals.

3) No bugs exist in your theorem prover.

~~~
hinkley
> 3) No bugs exist in your theorem prover

Seems to me that at some point we want theorem provers to be self hosting, as
it were.

That of course won't fix problems with bugs in hardware, and hardware that
behaves like analog devices (rowhammer, cold boot, data recovery, stray gamma
rays...)

~~~
simondedalus
Gödel proved that we will never get such a prover:
[https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_...](https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_theorems)

Particularly, he proved that it is impossible for a formal system to prove its
own consistency.

~~~
informatimago
However, prover[p-1] can prove prover[p], modulo the bugs prover[p-1]. Then
prover[p-2] can prove prover[p-1], modulo the bugs in prover[p-2]. If you can
have prover[i-1] is simplier than prover[i], and prover[0] is so simple that
any 5-years old can prove it's correct, then you win.

~~~
randallsquared
Something seems wrong about this. Could you not describe the combination of
all provers as a formal system? Would this system then not run afoul of
Goedel?

~~~
crpatino
Theoretically, you are 100% correct.

In practice, it is still worthwhile to have N redundant independently
developed provers to prove each other, and to benchmark against each other.
This will help uncover subtle bugs that result from ordinary programming
mistakes.

This will, however, not protect against deeper issues that are common to all N
provers. e.g. Potential biases or mistaken assumptions in the formal
verification comunity.

At the end of the day, it is an engineering decision. How much of the
available resources are you willing to through at the practical problem of
making your implementation approach asymptotically to the theoretical limits.
The fact that those limits is indeed a peg or two below perfection is an
independent issue.

~~~
nickpsecurity
"In practice, it is still worthwhile to have N redundant independently
developed provers to prove each other, and to benchmark against each other.
This will help uncover subtle bugs that result from ordinary programming
mistakes."

I strongly advise that to deal with both mistakes and subversion. Mutually-
suspicious parties in various countries using different logics, compilers,
hardware, etc. with a common, executable spec that's basically state machines
or functions. That way they get same output from same input.

So, when I saw Milawa was verified first-order, I immediately looked for other
verifications of first-order provers (found 1). Then the idea would be diverse
implementations of executable parts of that stack with diverse reviews of
logic combinations they did. Exhaustive testing done by each party. Even do
implementations of key parts of TCB in other logics to check results. The
prover with its code is trustworthy when everyone got the same results on same
logics, code, shared tests, and individual tests. Plus have paper copies of
the resulting text files with hashes stashed away somewhere in case a nation-
state wants to throw money at countering diversity via hacking host systems.
;)

Then we rinse and repeat for more complex provers. The fun never stops in this
game.

------
contingencies
This comment on the page is insightful:

 _Anecdote – Erlang designer Armstrong does contract work for formal
verification with high-assurance applications. A widget-maker for a large
European car manufacturer wanted to ensure their widget was fully compliant
with all of the specifications as issued by the overseeing administrative
authority (ISO, ANSI, whatever). He and his team of engineers sat down and
took 1k + /\- .2k pages of written specifications, translated them into
invariant properties, and let 'er rip. Of the ~foo bugs they found, ~40% of
the bugs were internally inconsistent bugs _within the specification _. [An
average car of model year 2016 has somewhere between 500-800 MCUs within them.
Scary prospect…] So, lets say LDAP 's RFC has 3 'bugs' in them (a proposition:
p in section one, then a proposition: ~p in section 3). Microsoft wants to
implement Active Directory to have LDAP compliance to work with Kerberos,
Apple devices, etc etc. By definition, this implementation will be buggy.
These bugs are of a class which have no fix!_

... if you want to read some formal proofs from the software and mathematics
worlds, see [https://www.isa-afp.org/](https://www.isa-afp.org/)

~~~
iheartmemcache
Haha, that was my comment! Glad you liked it. Never thought it'd show up on
HN. I normally don't comment on Quanta, but there were so many half-informed
remarks propagating misinformation that I felt compelled to make not one, but
two, comments in that thread (and formal verification isn't even my domain of
expertise!)

------
CiPHPerCoder
Formal verification doesn't mean your design can't be flawed in other ways
that your theorem prover can't identify.

See also: Software AES with cache-timing information leaks.

~~~
lmkg
The practice of formal methods has progressed by acknowledging and accepting
those limitations. It now attempts to prove useful properties of your code,
rather than stamping it "correct" or "incorrect."

FTA:

“We’re not claiming we’re going to prove an entire system is correct, 100
percent reliable in every bit, down to the circuit level,” Wing said. “That’s
ridiculous to make those claims. We are much more clear about what we can and
cannot do.”

~~~
CiPHPerCoder
The headline is what I take issue with. Calling it hacker-proof just because
it's formally verified is wrong.

~~~
nickpsecurity
Yeah, they really need to knock that off. Formal verification creates hard-to-
hack code is still a decent headline. Or "Researchers using formal
verification stopped hackers in their tracks." That's entirely accurate plus
can be followed by specifics to understand what context it was true and what
contexts it might be true elsewhere.

Hacker-proof is just going to hurt branding of formal methods the second a
hacker takes out a high-assurance system. Whereas, the formal methods and
high-assurance literature written by anyone with sense acknowledges plenty of
assumptions that might be targeted plus just human error. Even the people they
quote in the article don't say it's hacker-proof!

------
jaibot
"Beware of bugs in the above code; I have only proved it correct, not tried
it." \- Donald Knuth

~~~
JadeNB
> "Beware of bugs in the above code; I have only proved it correct, not tried
> it." \- Donald Knuth

"Beware of people slinging around the famous Knuth quote as if it invalidated
progress in program proving." \- Me

(Not a reference to you personally, but I think that this quote is too often
used to disparage all work on program provers, rather than as just—as I take
it to be—pointing out that, no matter how good your prover is, you should
still check your code.)

~~~
nickpsecurity
That and the Godel one above. For first point, they could link to a
presentation on high-assurance systems showing one needs a combination of
specs, proofs, testing, and human review for max assurance. For the second,
the alternative is showing how the prover problem is greatly reduced by using
simple, proof checkers and/or logics in the process that are easy to verify by
hand.

------
sklivvz1971
Side-channel attacks, attacks that depend on physical characteristics of the
system, are a thing [0] and formal correctness doesn't really prevent them.

Also there's crazy stuff like Row Hammer [1] which depends on escaping the
abstraction that memory bytes are independent entities.

While these are maybe crazy or extreme attacks, they are still practical in
some cases. Any abstraction can eventually be hacked simply because an
abstraction is a simplification of a more complex system which can be
manipulated.

Reminds me of the joke of the junior functional programmer convinced that
their code is side effects free and who then proceeds to crash the garbage
collector because memory allocation is not considered a side effect...

[0]: [https://en.wikipedia.org/wiki/Side-
channel_attack](https://en.wikipedia.org/wiki/Side-channel_attack) [1]:
[https://en.wikipedia.org/wiki/Row_hammer](https://en.wikipedia.org/wiki/Row_hammer)

~~~
nickpsecurity
I addressed this here:

[https://news.ycombinator.com/item?id=12762323](https://news.ycombinator.com/item?id=12762323)

The military, EE's, and even RAM suppliers knew about those issues. There are
products in high-security (esp Defense) that try to counter as many as they
can afford to. The problem was the knowledge was ignored or intentionally not
applied. The first just keeps happening in general in so-called INFOSEC. The
second is a result of market incentives where the suppliers' goal is to hit
specific performance, energy, area, cost, etc targets to maximize market share
and profit. On cost, they also want to minimize that in general for the
executives and shareholders to pocket more income. This really affects RAM
vendors as people involved in QA have said on various forums that they're
basically just pushed to get shit out the door as quickly as possible since
competition is intense. That they do this with ever-increasing complexity in
designs and physics in shrinking nodes means problems can only go up.

So, it's not a new discovery in terms of hardware correctness or even an
accident. These problems are intentionally there to increase profits. Compare
that to the design of things like 1802, NonStop, AAMP7G, rad-hard ASIC's...
all that trade off marketable criteria at higher prices to get higher
reliability. They exist but almost nobody buys them. That's why such practices
aren't and won't be status quo.

EDIT to add NonStop link to show what earlier systems accomplished with
careful hardware and software design. Similar, even more clever, techniques
were used inside ASIC's as well.

[http://www.hpl.hp.com/techreports/tandem/TR-86.2.pdf](http://www.hpl.hp.com/techreports/tandem/TR-86.2.pdf)

~~~
sklivvz1971
The point that I was trying to make is that they have proven correctness
within a specific abstraction, but the article incorrectly say the code is
"hacker-proof". Hacking in most cases happens because we can break through
abstractions.

~~~
nickpsecurity
"The point that I was trying to make is that they have proven correctness
within a specific abstraction, but the article incorrectly say the code is
"hacker-proof"."

The title is terrible as several of us argued here. The article itself at
least describes the process accurately with many quotes talking about how
tricky it is or that you should test it anyway. Just a terrible title...

------
no_protocol
[Audio]

Edit: Link was originally a podcast that auto-played. Has since been updated
to a written content link.

------
nabla9
No new breakthroughs in formal proofs or deriving programs, just incremental
progress. The cost of formal methods stays high. The high cost means that it
does not work well with iterative programming, or changes in specs.

So far only some military and safety critical applications and parts of
Intel's silicon have been worth of the cost.

I took my first course in formal methods using the book Programming: The
Derivation of Algorithms by A. Kaldewaij. It would be perfectly reasonable to
implement something similar to Kaldewaij's guarded programming language today
over some other programming language and use it to derive formally proven
programs if someone really values the correctness. It would be also cool and
fun but not for everyone.

~~~
philipkglass
I hope that one of these days (years?) we might start seeing formally verified
versions of code for dealing with standards that don't change rapidly and that
have proved to be security problems in the past. I'm thinking of things like
font rasterizers, TIFF readers, full PDF readers, and JavaScript engines
intended for browsers.

~~~
nickpsecurity
The problem with many of these formats is they're Turing-Complete languages
with input driving what they do. The input is also malicious. So, the solution
so far is to isolate and mediate such things with verification going into the
mechanisms for that.

An example of one for exactly your types of programs is Microsoft's Xax:

[https://www.microsoft.com/en-us/research/wp-
content/uploads/...](https://www.microsoft.com/en-us/research/wp-
content/uploads/2016/02/xax-osdi08.pdf)

My preferred approach is the old-school method of highly-assured microkernels
that just separate everything else from one another. Here's a browser OS based
on that principle:

[https://www.usenix.org/legacy/event/osdi10/tech/full_papers/...](https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Tang.pdf)

~~~
sliverstorm
Aren't microprocessors Turing-Complete machines that undergo formal
verification?

~~~
nickpsecurity
The kind of verification they undergo varies considerably. The amount of
errata in modern CPU's shows both they're not fully verified for correctness
and that the verification they get works wonders given number of components &
interactions. Even if fully verified, the verification would be that the
Turing-Complete hardware properly implemented a model of the Turing-Complete
hardware.

That kind of buys us nothing in terms of whether we'll know if programs are
malicious or contain them, eh? And look at the giant, losing game that
followed between the mice and the well-fed cats. It's why safety-critical
systems often used deterministic, state-machines running on simple MCU's or
PLC's. :)

------
crypto5
Curious if there is an opposite process: having some program, ask some prolog-
like system if given program has any vulnerabilities, and how to exploit them?

~~~
nickpsecurity
You're basically describing static analysis but with Prolog and a weaponize
option. I haven't looked up bug-to-attack research in ages but last I did it
was generating exploits from patches. That one was pretty evil if one is aware
of why delaying patches often makes sense for mission-critical apps. They need
testing as they can do damage themselves. So, instantly turning even a chunk
of patches into exploits can give an attacker an advantage in that windows
between release and application.

Example:

[http://bitblaze.cs.berkeley.edu/papers/apeg.pdf](http://bitblaze.cs.berkeley.edu/papers/apeg.pdf)

------
andybak
Mods - please label this audio (and auto-playing)

------
shpat
Worth noting that we've open sourced the toolset used to generate code for the
Little Bird, available at [http://ivorylang.org](http://ivorylang.org)

------
rhinoceraptor
Code can still be exploited through vulnerable hardware, though.

------
45h34jh53k4j
pls warn podcast in title, i dislike podcasts

------
nixpulvis
I wouldn't call anything unhackable.

~~~
macawfish
Sometimes the best hack is to leave a unhackable program alone. And by
unhackable I mean proprietary.

------
platz
how would they anticipate timing attacks with this method?

~~~
nickpsecurity
Lots of work in that area. Here's an example:

[http://www.cse.psu.edu/~dbz5017/pub/pltiming-
pldi12.pdf](http://www.cse.psu.edu/~dbz5017/pub/pltiming-pldi12.pdf)

~~~
platz
Impressive!

------
StyloBill
Here is the readable version.

[https://www.quantamagazine.org/20160920-formal-
verification-...](https://www.quantamagazine.org/20160920-formal-verification-
creates-hacker-proof-code/)

~~~
sctb
Thank you, we've updated the link from
[https://www.quantamagazine.org/?powerpress_pinw=29937-podcas...](https://www.quantamagazine.org/?powerpress_pinw=29937-podcast&utm_source=Quanta+Magazine&utm_campaign=2f0887e749-Quanta_Newsletter_Feb_27_2015&utm_medium=email&utm_term=0_f0cb61321c-2f0887e749-389523389)

