
OpenSSL's squaring bug, and opportunistic formal verification - jsnell
http://kryptoslogic.blogspot.com/2015/01/openssls-squaring-bug-and-opportunistic.html
======
nullc
The proof appears to be incorrect.

Consider the case where c2,c1,c0 are all UINT_MAX.

In OpenSSL that can't be the case because the outside environment constrains
the accumulator; but I see nothing on that page showing such a constraint
there.

I'm not familiar with the z3 python interface, clearly something interesting
implicit must be going on there. (As it does appear to be expressing that the
values are finitely ranged, and doesn't appear to be expressing a constraint
for ref)

In general I've had bad luck getting results out of SMT solvers on finite
ranged problems-- often they just get stuck. If you can ensure there is no
overflow then its easier, but then again it's also easer to reason about them
absent the mechanical prover in that case too.

The other major hurdles is extracting the code in a way that can be usefully
presented to solvers without a lot of potential for human error or extreme
effort (e.g. OpenSSL is 400k lines of code, good luck with making progress if
you're hand rewriting things for the prover). Frama-C
([http://frama-c.com/](http://frama-c.com/)) can help with this and I've used
it to good effect on very small pieces of code.

Hopefully in the future we'll see more programming languages with better
accordances for using formal methods-- ATS is pretty interesting in that
respect--, and also bridging the usability gap enough to see them widely used
in industry.

~~~
klogic
The proof is implicitly modulo 2^96, which I believed to be the intention of
the original code, but it does miss a possible overflow when c is too large,
which shouldn't happen in its intended use: c2 is already meant to be the
overflow limb. I've added an addendum to the post describing, and fixing,
this.

~~~
nullc
Thanks!

One of the things I find useful for avoiding overconfidence in proof results,
beyond making sure to test the proof on multiple broken versions, is to spell
out many additional implications of the function that I and/or the caller
might expect of it and ask for proofs on them too, even if they seem somewhat
redundant.

For example, cout = 2 _a_ b + cin; I would also prove that for all a,b,cin
that cout >= cin; for a or b ==0 that cout == cin, etc.

------
nanolith
Beautiful. Formal verification does take more time over empirical testing
methods such as unit testing, but when it comes to mission critical components
such as cryptography or system code, this is time well spent.

~~~
wsxcde
edit: just to be clear, I'm responding to this statement:

> _Formal verification does take more time over empirical testing methods_

I don't even know if that is true, at least, I'm unconvinced about it being
true in hardware verification. The amount of manpower and effort companies
like Intel and AMD put into simulation-based verification is insane and a lot
of that could be replaced (and has in some case been replaced) by a well-
designed formal assertion suite.

The real bottleneck is people. It's a lot easier to find competent engineers
who can hack around with tests and simulation suites than it is to find people
who can hack around with abstractions for scalability in a model checker.

~~~
nanolith
There is a pragmatic perspective and a theoretic perspective. Pragmatically
speaking, empirical methods can get one pretty far. It's better to have full
coverage of empirical tests across an entire system than limited coverage of
formal tests, for instance. So, on that account, you may be correct from a
pragmatic perspective.

From an epistemological perspective, a test suite to confirm a property using
chosen examples is not nearly as good as a formal proof that guarantees a
property over a given domain. For simple cases, such as branch coverage, the
test suite may be good enough. As complexity increases, or when testing things
like modular arithmetic over an elliptic curve, unit testing leads to false
confidence.

But, just like in empirical testing, it's possible to build bad proofs. This
is where the specialization comes in. There may be plenty of competent
engineers who can hack at a test suite, but their results will not be nearly
as comprehensive as someone trained in formal verification.

It all comes down to epistemology. What do we know, and how do we know it?
What possible defects exist in our system, and how bad can they be? Can we
ship yet?

~~~
nullc
Any kind of testing or review can lead to false confidence. This is why its
important to use multiple approaches in the hope that their errors are not
completely correlated.

~~~
nanolith
I agree.

