
Software Foundations - rfreytag
https://softwarefoundations.cis.upenn.edu/
======
nickpsecurity
I'd recommend people interested in learning formal methods _do not_ start with
things such as Coq or Isabelle/HOL. Most people find the process of learning
or using those so difficult that they quit with a potentially-permanent
aversion to the term formal verification. In engineering, it's often advised
to use best tool for the job esp with highest ROI. So, if you're interested in
formal methods, I strongly encourage you to start with so-called, lightweight
methods that give a lot of return for little investment. They teach you how
abstract specifications with tool-assisted reasoning can quickly catch deep
errors tests often won't.

Two of those with lots of benefit in realistic projects are Alloy Analyzer
(esp for structure) and TLA+/PlusCal (esp for order):

[http://alloy.mit.edu/](http://alloy.mit.edu/)

[https://learntla.com/introduction/](https://learntla.com/introduction/)

These are immediately useful. Especially in distributed systems for TLA+.
Then, you get a book such as Chlipala's Certified Programming with Dependent
Types or Pierce et al's Software Foundations. Chlipala is also doing a new one
called "Formal Reasoning About Programs." You can try to use those to get some
elementary stuff done with the heavyweight methods. It will probably take you
1-2 years to get proficient enough for real stuff. Two outcomes are common:

1\. You don't make it. Don't get mad: most don't even when they're smart. Good
news is you enjoyed lighter stuff that you'll probably continue to apply and
expand on for rest of your career. If other users are lucky, you might have
run into their limitations enough to start improving the tooling,
documentation, or something else. Those will get better since you learned them
but avoided heavyweight stuff.

2\. You made it! Now, best result for us IT folks is you apply your skills to
real-world software formally-verifying useful algorithms, protocols,
compilers, OS's, and so on. For inspiration, be sure to look at CRYPTOL,
miTLS, CompCert, CakeML, RockSalt, seL4 or VAMOS, Verdi, verified garbage
collectors, foundational backing of static analyzers like Verasco, tools with
automation such as SPARK Ada, ACL2 for hardware, and maybe rewriting-based
stuff like C semantics in K Framework.

If you can understand heavyweight verification, then apply your knowledge to
more stuff like that since you'll (a) possibly be having real-world impact and
(b) be doing work challenging enough that you can be very proud of your
successes.

~~~
dbcurtis
Inspired by your post, I spent a few minutes reading the Allow web site. I'm
wondering exactly how to benefit from Alloy? I have a problem that involves
multiple interacting state machines communicating over protocols that I don't
necessarily control, over unreliable network connections. Lots of
retry/recovery arcs.

So assuming I could specify this system with Alloy, what would I learn? I'm
guessing I would at least detect gaps in the protocols and missing error
recovery arcs in state machines. But that doesn't yield a correct
implementation... it might help me fill in gaps in the validation test
coverage space. Is this what I should expect?

~~~
tel
Most practically: your personal knowledge of the system will grow tremendously
through the process of trying to formalize it and debug the errors you
encounter. It's likely that basic assumptions you have about the protocol are
false and that real bugs arise due to these subtle imperfections.

Some formal system technologies let you "extract" algorithms into other
languages. Model checkers like Alloy don't. What you take away is what you
learn and what you can use of that learning to convince others.

A good concrete takeaway would be information you could use to design a test
suite that breaks the system even where others believe it to be solid.

~~~
dbcurtis
My backround in hardware and software validation is long and deep, so I
clearly see the benefit of getting information about the test space that needs
to be covered. Still, it seems like it ought to be possible to do some kind of
automated test stimulus generation using what Alloy already does. I am
imagining a machine-readable report in some format that is easy to feed into a
problem-specific test generator script.

In a past life, I worked on automatic test pattern generation for sequential
logic circuits. Which, if you want to tilt at windmills for little $return
(1), is as good a pursuit as any. But it strikes me that this is a very
similar problem. Given a description of a state space, create stimulus that
covers it, sensitizing all the logic paths in a way that produces an
observable result that differs from expectations.

(1) The central lesson of trying to sell test: Electronic Design Automation
dollars are like western US water law, the first to divert wins. In the case
of EDA, the up-front design tools get all the dollars, and there is hardly a
trickle of money left by the time it reaches test phase.

~~~
nickpsecurity
Hmm. I don't use Alloy nor do I know hardware yet. I'll still try to dig up
something. (comes back) It looks like a lot of CompSci folks built test
generation in Alloy for languages such as Java. Might have not released any
general tools for Alloy, though. Good news is they describe the benefits and
some specifics where hardware people might get ideas for homebrew tools.

[https://repositorio-
aberto.up.pt/bitstream/10216/59949/1/000...](https://repositorio-
aberto.up.pt/bitstream/10216/59949/1/000145607.pdf)

[http://www.dcc.fc.up.pt/~rvr/resources/MAP-i/2014s7c-mapi.pd...](http://www.dcc.fc.up.pt/~rvr/resources/MAP-i/2014s7c-mapi.pdf)

One application to hardware with Alloy:

[http://www.samkirk.com/atpg-in-
alloy4/ATPG_in_Alloy.pdf](http://www.samkirk.com/atpg-in-
alloy4/ATPG_in_Alloy.pdf)

Honestly, I'd have thought TLA+ would be better of the two for hardware given
it deals with concurrent things. Also, it can do stuff similar to SPIN model-
checker that was used extensively for hardware verification in the past. For
hardware, I'd have experimented with SPIN for lightweight method with ACL2 for
heavyweight esp thanks to Jared Davis at Centaur contributing ACL2 components
he used.

[http://spinroot.com/spin/whatispin.html](http://spinroot.com/spin/whatispin.html)

[https://www.cs.utexas.edu/~jared/publications/2011-memocode-...](https://www.cs.utexas.edu/~jared/publications/2011-memocode-
centaur.pdf)

Just for kicks, I did some quick Googling on test generation for sequential
circuits to see if I could find anything interesting for you. Me, too, as I
enjoy seeing what people do in your field. Most were paywalled but this one
wasn't:

[http://www.smc2017.org/SMC2017_Papers/media/files/0080.pdf](http://www.smc2017.org/SMC2017_Papers/media/files/0080.pdf)

~~~
dbcurtis
Thanks for the links. My current application is not test generation for
hardware, though. I'm working with autonomous robots that need to interact
with real world obstacles that can't be easily predicted (laundary carts,
golden retrievers, bride's maids....), need to navigate spaces where WiFi
comes and goes, and need to interact with other automated systems running
their own state machine that I don't control. Something that could identify
"Recovery scenario needed here" has good value.

~~~
nickpsecurity
Apparently, I totally misread what you needed. Sorry. Your domain is very
interesting. I briefly studied it a long time ago. They were using fusion of
data coming from different components with quite non-deterministic algorithms
such as fuzzy logic or neural networks. There wasn't any way to verify them at
the time. I've seen a few attempts at such things but would need to know what
kind of algorithm you use for sense, learn, and act (or whatever they call
it). No guarantees here.

------
lou1306
This is really neat, but in my opinion conflating theory _and_ actual
verification through Coq might be a bit too much for a newcomer.

For those who are looking for a softer introduction to the field, I highly
recommend _" Reactive Systems: Modelling, Specification and Verification"_ by
Aceto, Ingólfsdóttir, Larsen, and Srba (full disclosure, I am a student at
Aceto's current affiliation).

------
ahelwer
These are the perfect books to work through on a Sunday afternoon. Did the
first part of Logical Foundations three years ago, and still look back on that
day with fondness. Nothing has scratched that itch for perfect machine-
enforced certainty in quite the same way since then.

------
auggierose
These are very pretty books, and at the same time very ugly. Thank god I
already know the material in these books, I wouldn't want to learn it this
way.

------
rambojazz
Do these books come with a free license?

~~~
dtornabene
click through, they're in html

~~~
dqpb
Ironically, they are rather hard to click through. You have to return to the
toc to get to each next chapter.

