
Formal Verification Creates Hacker-Proof Code (2016) - Alekhine
https://www.quantamagazine.org/formal-verification-creates-hacker-proof-code-20160920/
======
tptacek
Formal methods are a powerful security tool because they produce useful code
so slowly that they necessarily minimize attack surface.

~~~
sterlind
I hear this claim a lot but I question it's truth. I'm faster with formal
tools than without.

For instance, this week I needed to build a distributed locking mechanism so a
collection of asynchronous actors could only access resources in one region at
a time. I described my two requirements in TLA+: only one zone at a time, and
everyone who wants a zone eventually gets it (with the assumption nobody goes
down forever - weak fairness.) I had several bugs which it caught, until I
finally arrived at a protocol that was _proven correct_ (at least for a
handful of workers on a handful of zones.) It took around a day from start to
finish, and now I know my design is literally flawless.

Sure, I didn't prove it in the general case, nor did I prove that my code
implements the spec, but this was still extremely useful! Having the machine
check your designs frees you from worrying if you've caught all the races and
corner cases.

~~~
drdrey
TLA+ doesn't prove anything to be "correct" or "flawless", I find it extremely
worrisome that you claim as much. It failed to exhibit traces that violated
your invariants, which says something completely different.

~~~
p0llard
I've never actually used TLA+, but my understanding is that it does
_exhaustive_ model checking (presumably using some suitable simulation
relation), so in this case failure to find a violating trace indicates that no
such trace exists, and the invariants always hold.

Correctness might well be defined solely by a set of invariants, so I don't
quite understand your point here?

~~~
drdrey
Imagine the following scenario, you have encountered an actual bug in
concurrent code, and you think it would be great to work on a model and
invariants that lets TLA+ find the bug (as an exercise).

So you start writing a model, and a set of invariants, you run the tool and...
it finds nothing. Does that mean you just proved the bug does not exist? No,
you _know_ the bug exists, what it means is that you didn't write a correct
model on your first try. So you iterate on it, add a few things you missed,
put some extra labels, read the docs a bit more, maybe refine your invariants,
make them stronger, work through a couple traces that don't actually make
sense in practice (e.g. actors just never taking any action whatsoever).
During all this time, you either found nothing or things that were not
interesting. At no time have you ever proved anything correct, you just
iterated on your model until you managed to capture a real life scenario.

It takes some convincing to make the jump from "this model does not violate my
invariants" to "I think this model and the invariants are close enough to what
I care about that I think it's good enough"... I don't know how to go from
there to things like "proven correct" and "flawless".

~~~
sterlind
I mean, specific to my scenario, my invariant was "only one zone at a time"
and the liveness property was "everyone who wants a zone eventually gets one",
so respectively:

1\. \A z \in Zones: Active(z) => ~\E o \in Zones: z # o /\ Active(o) 2\. \A w
\in Workers: \A z \in Zones: WorkerWantsZone(w, z) ~> WorkerHasZone(w, z)

Unlike your scenario, I am checking the spec here. I know that code that
properly implements the spec satisfies these two properties, therefore I've
proven the spec correct. You can disagree on the importance of those two
properties but I've mechanically checked that those properties are always
preserved in my finite models, so I know the spec is correct, at least for the
model size.

You know, it's really not too hard to get started with this stuff. You can
always nit-pick but why not try it out and see if it's useful for you?

~~~
drdrey
I am using it, I just don't use it to make claims of correctness.

------
est31
Note that existence of a formal proof for security is no silver bullet. A real
world example of this is the WPA2 key exchange's formal proofs not catching
the KRACK vulnerability.

[https://galois.com/blog/2017/10/formal-methods-krack-
vulnera...](https://galois.com/blog/2017/10/formal-methods-krack-
vulnerability/)

That being said, it's a very valuable tool and should be applied more, not
less.

~~~
rrmm
Yeah the subtitle can be, "Formal Verification highlights the difficulty of
specifications". Which wasn't a secret in the first place.

But yeah, fewer bugs is better than more.

------
MaxBarraclough
> One of those verified building blocks comes with a proof guaranteeing that
> someone with access inside one partition won’t be able to escalate their
> privileges and get inside other partitions.

The article is presumably referring to the seL4 formally verified operating
system. There's not a single mention of it by name, but it lines up, as
they're discussing the unmanned Little Bird, which is one of seL4's success
stories.

[https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5597724/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5597724/)

[https://www.newscientist.com/article/mg22730392-600-unhackab...](https://www.newscientist.com/article/mg22730392-600-unhackable-
kernel-could-keep-all-computers-safe-from-cyberattack-2/)

[https://en.wikipedia.org/wiki/L4_microkernel_family#High_ass...](https://en.wikipedia.org/wiki/L4_microkernel_family#High_assurance:_seL4)

------
Animats
I'd like to see a few "hard nuts" created, formally verified secure all the
way down to the instruction level on a somewhat simple CPU (i.e. not x86). A
BGP server and a DNS server would be a good start. Those have a specific task
and a well defined external interface that changes little.

~~~
docandrew
Here's a formally-verified DNS Server:
[https://ironsides.martincarlisle.com/](https://ironsides.martincarlisle.com/)

Not all the way down to the instruction level, but it's a start.

------
empath75
Even 100% formally verified code could be vulnerable to side channel and other
hardware attacks. There could also be bugs in the verification system, the
compiler, the spec that they’re verifying, nothing is hacker-proof.

~~~
not2b
Yes, a formal proof shows that a system does not have a specified class of
bugs, subject to specific assumptions. This is very valuable, but it is better
to think of it as another (very powerful) bug finding technology than a
complete assurance that nothing can go wrong.

------
zemnmez
formal verification continues to come up, but it has the issue that you can
only formally verify problem domains you already expect -- and the advanced
threat actors for which formal verification is necessary have the toolkit to
go off-script to find vulnerabilities.

I think the hype around formal verification also tends to misdirect away from
the verification that strong type systems already do if used correctly, though
vitally not at the protocol level.

~~~
Klasiaster
Type systems can encode invariants even on the protocol level. For example you
can hand-write or generate (multi party) session types to avoid deadlocks etc.

For Rust there is [https://crates.io/crates/session-
types](https://crates.io/crates/session-types) from
[http://munksgaard.me/papers/laumann-munksgaard-
larsen.pdf](http://munksgaard.me/papers/laumann-munksgaard-larsen.pdf)

~~~
zemnmez
this is an interesting approach that I hope we see more of! but i specifically
meant to refer to security components of a protocol: e.g. that whether a party
can impersonate another with access to some information that is public or can
be disclosed via some attack.

~~~
Jhsto
Are you aware of Pi-calculus and ProVerif? ProVerif sounds to address at least
some of your requirements. Links: \-
[https://en.wikipedia.org/wiki/%CE%A0-calculus](https://en.wikipedia.org/wiki/%CE%A0-calculus)
\-
[https://prosecco.gforge.inria.fr/personal/bblanche/proverif/](https://prosecco.gforge.inria.fr/personal/bblanche/proverif/)

------
cryptica
The idea of formal verification of code is kafkaesque. Its popularity is
proportional to the inefficiency and irrationality of the economic system
within which it exists.

Code is an expression of pure logic. So if there is any incorrect assumption
or flaw in that logic, it can be verified by simply reading the code.

Given that premise, the usefulness of formal verification can be disproved in
a single English sentence:

Formal verification does not remove the element of human error from a system;
it removes it from the code and adds it to the system of proofs which verify
that code... Proofs can be written with incorrect assumptions (human error) in
the same way that code can be written with incorrect assumptions. As one of
the people in the article acknowledges, writing good proofs is challenging;
"challenging" inescapably means "prone to human error".

Formal verification proponents seem to have conveniently forgotten to apply
formal verification to prove the effectiveness of formal verification.

The great irony of this article is that it attempts to rely on exhaustive
proof (which is the basis of unit testing) to prove the effectiveness of
formal proof - I.e. we tested formal verification with that helicopter that
one time and the hackers couldn't hack it for 6 months, therefore formal proof
is effective - That's a lousy unit test case and it doesn't prove formal
verification.

~~~
docandrew
The difference is that code with bugs can still compile, but proofs that are
unsound won't check. Now granted, you can still prove the wrong thing or _not_
prove the thing that is supposed to make your system robust is taking place,
so it's still very much a human endeavor with all that entails.

I think a dedicated red team going after a system for 6 months and not finding
anything is a pretty good "unit test", personally. It's _evidence_ of
effectiveness, not _proof_.

------
sesutton
Doesn't the KRACK vulnerability undermine the idea that formally verified
equals perfectly secure?

~~~
jhanschoo
You're partly right, formal methods don't catch problems that you assume does
not happen in your assumptions (herein, the "operational semantics").

If this plan to formally verify internet algorithms included the Wi-Fi
standard and implementations, the KRACK vulnerability wouldn't be an issue
because it wouldn't verify (with respect to certain specifications, if they
realized that those specifications were important to begin with).

But in the first place, most operational semantics on this level still stay at
layers of abstraction high enough that they don't guard against Spectre and
Meltdown.

Nevertheless, note that formal verification is also widely used at the
microchip level.

------
ncmncm
Title is verifiably false.

First, formal verification doesn't create anything. It only certifies code
created by other means.

Second, formal verification only certifies that the software implements the
machine-expressible details of the spec. It doesn't certify anything expressed
only in English.

Finally, it doesn't and can't ever certify that the spec itself is correct or
complete. Most bugs and security holes result from incorrect, incomplete, or
inconsistent specs. (Any program satisfies an inconsistent spec, but
inconsistent specs can at least be identified by formal means.) Very few specs
can be formally expressed completely.

So, formal verification is useful, but neither creates hacker-proof code, nor
ensures that code is hacker-proof.

~~~
kyboren
That's not entirely true. Synthesis is a popular topic within the "formal
verification" umbrella.

I agree that proving the software satisfies some safety properties and is
always consistent with some specification doesn't mean that a system is truly
"hacker proof".

However:

1\. It's a step in the right direction, and something like a necessary
condition but not a sufficient condition.

2\. It's an attractive, succinct way to describe the value of formal methods
and their potential (as the cited DeepSpec project intends to realize) to
indeed make a truly hacker-proof system.

~~~
ncmncm
It's an attractive, succinct way to _mislead_ people about the value of formal
methods and their potential.

Synthesis from specifications completely fails to address omissions and errors
in the specifications, except insofar as they get translated to something that
is more evidently wrong, or are revealed to be inconsistent.

------
Exuma
Can someone explain how this works from a 7/10 technical perspective? I hate
reading articles that talk to you like youre five when trying to explain
advanced concepts. I just want to know how it works in 1 paragraph and thats
it.

~~~
dfee
I understand your frustration with Quanta on this, and agree completely. I
often times make it through about 2/3 of an article and just give up - just
get to the point (and preferably open with that point)!

Here was a lengthier diatribe of mine on a former Quanta article:
[https://news.ycombinator.com/item?id=22549988](https://news.ycombinator.com/item?id=22549988)

~~~
twic
Preach! It's a little surprising to me how often Quanta articles hit the front
page here. They are always about a very exciting subject, but they very rarely
tell you anything much about that subject.

------
cryptica
>> You’re writing down a mathematical formula that describes the program’s
behavior and using some sort of proof checker that’s going to check the
correctness of that statement

The obvious flaw in that approach is that clearly the 'proof checker' can
itself be vulnerable to human error. Any complex program is going to have many
'proof checkers' which depend on each other to verify different parts of the
system; if any assumption about what constitutes a valid proof for any segment
of the logic is wrong, then that would introduce a vulnerability in the same
way as a regular programming bug would.

So then the real question about the effectiveness of formal verification would
be; given the same amount of development time, are incorrect assumptions
introduced when writing the 'proof checkers' less likely than bugs introduced
when writing regular programming code?

Aren't programming bugs always caused by incorrect assumptions? What is it
about formal verification which frees a complex system from the evils of
incorrect assumptions?

Incorrect assumptions about the proof or incorrect assumptions about the
logic, what is the real practical difference?

------
namelosw
From my shallow understanding, formal verification usually proves the given
program always satisfies some properties/specifications. It's pretty good when
you expect the software to fully behave in the way you want it to be.

However, in terms of security, usually the loopholes come from those parts or
in those ways people didn't see them coming - so how do people write
specifications about what they don't know?

------
rurban
Nobody mentioned Ada/Spark yet? This is doing exactly that, but in a real
programming language, not with a seperate proof system. You only write code
once.

Use widely in the industry. With such a system you can also generate test
cases automatically to verify the assertions. Such as with cbmc for the C
language if one cares, and the manager doesn't trust the proof system.

------
microtherion
"Be careful about using the following code — I've only proven that it works, I
haven't tested it." — Donald Knuth

------
juskrey
Formal verification only verifies assumptions made by verifier about the
system.

------
jzer0cool
> "just not the sorted list we were probably hoping for ... > "For every item
> j in a list, ensure that the element j ≤ j+1 ... Yet the list [1, 2] also
> satisfies the definition since it is a sorted list, just not the sorted list
> we were probably hoping for.

If the input satisfies and happens to be the output in this case, why is
article saying this to be a bug? Why "just not the sorted list we were
probably hoping for"?

~~~
computerphage
> That is, given the list [7, 3, 5], she expects that the program will return
> [3, 5, 7] and satisfy the definition. Yet the list [1, 2] also satisfies the
> definition since “it is a sorted list, just not the sorted list we were
> probably hoping for,” Parno said.

Because the human meant "sort _this_ list", not just "return _a_ sorted list".
They put in [7, 3, 5] and got back [1, 2]. Then they realized their
specification was incomplete.

~~~
jzer0cool
> For every item j in a list, ensure that the element j ≤ j+1

What would be the model example for the correct formal definition? When I hear
"sort", there are implied constraints/parameters so [1,2] would not have even
been a possible solution.

~~~
computerphage
To find the missing constraints, you can try asking what's wrong with the list
[1, 2].

You might imagine trying to have an argument with a person trying to sell you
a list sorter. The salesman is just going on and on about how "but look, each
element of the list _I_ gave you is in the right order! So you should buy my
very nice list sorter!". I would want to yell "but you didn't include any of
the numbers from the input list! You just forgot about 7, 3, and 5."

So, another formal constraint might be something like this:

    
    
        for each element, i, in the input list, the element i must be in the output list.
    

Then, you might again try putting in [7,3,5] and get back a list like this:
[1,2,3,4,5,6,7]. Again, imagine arguing with the pushy salesman. The salesman
says "Look, you wanted a list with the numbers 7, 3, and 5, and you wanted a
list in order, this is both of those things, so you have to pay me for my fine
list-sorter now!" And again, I would want to object and say something like
"but wait a second, I want ONLY the numbers that I gave you in the input list
in the output list!". So, you end up with something like:

    
    
        for each element, j, of the output list, the element j must be in the input list.
    

Now, you try putting in [7,3,5] again. What's the worst thing that might come
out the other end that's still following all of these 3 rules? Can you spot
the next problem?

------
saithound
Needs (2016) in the title.

------
dk8996
I think this similar to what ADA is doing in crypto space.

~~~
exdsq
I don’t understand what “hacker-proof code means”, but yes IOHK (who develop
Cardano) do use formal methods such as Isabelle and Coq alongside some process
specification languages.

~~~
MaxBarraclough
> I don’t understand what “hacker-proof code means”

What's unclear about it?

~~~
exdsq
You can't verify the OS, you can't verify the network connection between your
apps, you can't account for poor business requirements that result in formally
verified hackable code, etc...

~~~
MaxBarraclough
> You can't verify the OS

You can. Look at seL4. This is an area of ongoing research.

> you can't verify the network connection between your apps

In what sense? You can verify the crypto, and the networking stack.

> you can't account for poor business requirements that result in formally
> verified hackable code

Not sure what you're saying here. Are you referring to the possibility that
the formal specification might not be adequate to guarantee security, even
discounting side-channel attacks?

------
m3kw9
And since that time what have they secured?

