
Verdi – Formally Verifying Distributed Systems (2016) - mindcrime
http://verdi.uwplse.org/
======
im_down_w_otp
Currently stuck between a rock and a hard place with things like this.

Many of the other formal verification tools make it very easy to have your
implementation drift or be entirely unmoored from your specification, but they
let you keep working at a level of abstraction from your problem that's still
very familiar. Though things like SMACK and DiVinE are helping to decrease the
gap between spec and code.

Using things like Coq + program extraction brings the overlap between spec and
implementation into much, much closer alignment, but brings with it additional
problems. Writing a complex program in a very abstract language further away
from an engineer's typical problem domain, being fairly limited in types of
languages supported for extraction, and/or still having to have an awful lot
of faith in the extractor (which itself is unverified as near as I can tell)
are all things that are currently keeping me out of Coq for immediate use
cases.

The good news though is that there's a lot of fairly high-profile work being
done (like Verdi) to increasingly bring formal methods to increasingly complex
software problems in ways that make using formal methods more approachable and
usable, and that's truly wonderful.

~~~
nickpsecurity
Try TLA+ (learntla.com) for concurrency, Alloy Analyzer for structure, and
SPARK or Frama-C for code-level stuff. See if you can feed TLA+ or Alloy
properties into stuff like SPARK. E-SPARK did that with Event-B. These are the
low-hanging fruit for verification vs full build out in Coq or Isabelle. Then,
you write checkers for it to apply testing like KLEE, property-based, or fuzz
testing.

These will get you close to full verification with less work. Each of the
three languages has been used in a lots of CompSci and industrial work. Note
that SPIN is similar to TLA+ and you can Google its industrial use for ideas
of TLA+ power in concurrency analysis.

~~~
pron
Java also has code-level specification language called JML, with a rich set of
verification tools[1] (in fact, Frama-C's specification language, ACSL, was
heavily inspired by JML).

Also, if you know TLA+, there's little reason to learn Alloy as well, as TLA+
is a superset of its features (as well as of Event-B, and you can compile TLA+
to Event-B to use its tools[2])

SPIN is not so similar to TLA+, as its language, Promella, is relatively low-
level, while TLA+ allows an arbitrary level of detail, and so is well suited
to specify very large, very complex applications, rather than small, well-
contained ones. Also, if you absolutely must write proofs and have the
considerable resources, I believe that Promella requires proving in Isabelle
(which takes a _long_ time to learn), while TLA+ has its own user-friendly
proof language that hides Isabelle under the covers.

[1]: E.g. [http://www.openjml.org/](http://www.openjml.org/) and there are
others.

[2]:
[https://www3.hhu.de/stups/prob/index.php/The_ProB_Animator_a...](https://www3.hhu.de/stups/prob/index.php/The_ProB_Animator_and_Model_Checker)

~~~
nickpsecurity
Re Alloy

Most people that use these tools told me TLA+ and Alloy do different things
with their automated tooling. They usually say TLA+ looks at ordering of
things with Alloy looking at structuring. I only found a few people comparing
them directly, though. Do you have a resource showing TLA+ does same thing as
Alloy or as easily?

Re SPIN

Im saying they were used similarly. You wrote in one artocle like set theory,
model-checking for temporal errors, and so on are new. That was the default in
high-assurance decades ago using things like Z specs with SPIN for
concurrency. Then, they tried to go for verifying high-level code, then rough
code, code then assembly, and so on with refinement proofs. However, it
_started_ with algorithmic verification with specs in set theory, first-order
logic, and ASM's. Writeups like yours tell me Lamport's concept was to
similarly use those easier formalisms leaving off anything but the algorithm.
Then add more automation to get more bang for buck for more developers. A good
idea since the subset of that which Z and SPIN offered had already prevented
serious, sometimes deep, errors in prior systems.

Now, far as SPIN, it was used to chech the properties of concurrency models,
hardware specs, caching, protocols, standards, and distributed systems.
Industry found bugs in all these things using it with relatively little
training vs other methods that required rare specialists. Sound familiar? TLA+
now shows up as tool of choice for quite a few of those areas with SPIN still
used a lot due to maturity & being well-known. That's why I said they're
similar: people are specifying and model checking similar things with them
similarly with much less effort than provers.

The easier proving on TLA+ and high-level nature are real benefits ovee SPIN.
It's why I recommend TLA+. ;)

~~~
pron
> Do you have a resource showing TLA+ does same thing as Alloy or as easily?

See, _Why Amazon Chose TLA+_ [1]. They found Alloy pleasant enough but a bit
weirder than TLA+, and TLA+ is richer in terms of expressivity and features. I
learned Alloy and played with it after knowing TLA+ well, and while I think it
is an _extremely_ nice and useful tool (although its syntax is indeed weird),
it is absolutely trivial to express anything you can in Alloy in TLA+, and
that's without even TLA (i.e., just the +). Their tooling is a bit different,
so I guess there may be cases where Alloy would be more convenient even if you
already know TLA+.

> Im saying they were used similarly. ...

Yeah. Deep formal methods are, in general, much more similar to one another
than they are different.

[1]:
[https://link.springer.com/chapter/10.1007/978-3-662-43652-3_...](https://link.springer.com/chapter/10.1007/978-3-662-43652-3_3)

------
relyio
Remember, two years ago James Wilcox and Doug Woos were formally proving
Raft's linearizable semantics using Verdi:
[https://news.ycombinator.com/item?id=10017549](https://news.ycombinator.com/item?id=10017549)

~~~
appleflaxen
Yes; because it's mentioned in the article:

> Both the Verdi framework and our verified Raft implementation are open-
> source. Please feel free to submit comments, issues, and pull requests on
> either repository.

[1] [https://github.com/uwplse/verdi-raft](https://github.com/uwplse/verdi-
raft)

------
nickpsecurity
You might find this interesting:

[https://blog.acolyer.org/2017/05/29/an-empirical-study-on-
th...](https://blog.acolyer.org/2017/05/29/an-empirical-study-on-the-
correctness-of-formally-verified-distributed-systems/)

------
palmskog
Some new developments of the framework in the last two years:

\- packaging via OPAM to separate framework code and system code

\- support for specifying and proving liveness properties

\- support for proving Abadi/Lamport style refinement mappings between systems

The following workshop paper gives an overview of current and upcoming work:
[http://conf.researchr.org/event/CoqPL-2017/main-
verification...](http://conf.researchr.org/event/CoqPL-2017/main-verification-
of-implementations-of-distributed-systems-under-churn)

------
awinter-py
It's awesome that these tools exist, but the verification codebase for Raft is
10s of thousands of lines of specification & proof files. Just installing
verdi on linux nearly bested me (some interaction btwn ocaml package
management, coq, and native code). I tried running the verification suite and
gave up after 10 or 20 minutes of full CPU.

TLA has warts too -- the only way I could interact with it is via a custom
java IDE.

It would be great to see lighter-weight, more accessible ways to embed proofs
in real-world programs and run them in test suites.

~~~
naasking
> It would be great to see lighter-weight, more accessible ways to embed
> proofs in real-world programs and run them in test suites.

Not sure what this means exactly. Do you mean extracting the part of the proof
that can't be verified statically, for whatever reason, as some kind of test
suite?

~~~
awinter-py
I think I'm asking for something in between a mathematically verifiable proof
and fuzzing. A lightweight way to translate runnable code into a symbolic
execution environment that can be exercised (not necessarily verified)
exhaustively or stochastically.

There are relatively few functions that I want to do this for, and I'm okay
with doing some manual work to define the possible inputs and how the
functions talk to each other (i.e. mock out network / filesystem / database).

------
sriram_malhar
I liked the layered approach in the IronFleet paper. IronFleet: Proving
Practical Distributed Systems Correct

[https://www.microsoft.com/en-
us/research/publication/ironfle...](https://www.microsoft.com/en-
us/research/publication/ironfleet-proving-practical-distributed-systems-
correct/)

It uses TLA for the distributed protocol part, and Hoare style verification
for sequential code, which has excellent toolchain support.

