
Lessons Learned Developing Software for Space Vehicles - someperson
http://lwn.net/Articles/540368/
======
rurounijones
I would love to see a more in depth article about what kind of programming you
need to do for a system controlling a rocket.

I remember reading that they cannot use exceptions and have to handle every
possible scenario. Part of me wonders how much of the source code is just
error handling.

~~~
w3pm
No dynamic memory allocation, no recursion, no longjmp, etc. Program execution
must be entirely predictable, every time.

SpaceX's programming rules are probably very similar to those used at NASA
JPL: <http://lars-lab.jpl.nasa.gov/JPL_Coding_Standard_C.pdf>

~~~
paulrademacher
That's just the tiniest tip of the iceberg. Doesn't get into formal specs,
verification, validation, etc.

This article is often referenced for NASA's software practices, although it's
now almost 20 years old: <http://www.fastcompany.com/28121/they-write-right-
stuff>

Is there anything more comprehensive and current than that?

------
tgflynn
_They do not have hard realtime requirements_

If spacecraft avionics doesn't have hard realtime requirements what does ?

~~~
biotech
Parts of the spacecraft will have hard realtime requirements, but those parts
can be modular and controlled by soft-realtime systems

~~~
tgflynn
Wouldn't those modules be considered part of avionics though ?

I got the impression the article was talking about the totality of avionics
software run on the spacecraft, perhaps that impression is inaccurate.

------
speeder
Very interesting to see how game development affects space research...

Granted, this might explain Richard Garriot fiddling with space companies, and
John Carmack manufacturing space engines!

If I was rich I would do space research too!

Also, I must be a software engineer, since I don't like Justin Bieber either.

~~~
spc476
Well, Richard Garriot's father was an astronaut, so that may have had
something to do with his decision to fiddle with space companies.

------
cju
Here are some papers about software/µP in space for those who want more
examples/details:

* Space Software Validation using Abstract Interpretation (2009)

<http://www.di.ens.fr/~rival/dasia09.pdf>

From the conclusion: _This study has shown that embedded space software are
difficult to analyze due to non linearity (mainly in quaternion computations)
and complex control command algorithms involved (e.g., Kalman filter)._

 _It should be noticed that the software architecture which suits static
analysis by abstract interpretat ion best is also the more readable one and
maintainable one. This technique can thus be a metrics of good architectures.
In spite of these difficulties, abstract interpretation techniques can greatly
improve the quality of space embedded software._

* Qualification of the Atomated Transfer Vehicule (ATV) Flight Control

ftp://ftp.elet.polimi.it/outgoing/Marco.Lovera/ESAGNC08/S01/01_Clerc.pdf

A lot of information about the test process, the usage of Hadware-In-the-Loop
and Monte Carlo testing.

* Requirements review, Needs for launchers, space vehicles & orbital infrastructures (2006)

[http://microelectronics.esa.int/conferences/ngmp2006/D2-1200...](http://microelectronics.esa.int/conferences/ngmp2006/D2-1200-ESA_RT.pdf)

High-level requirements for On-Board Computer (OBC).

------
cpeterso
I'm curious for more details about the software stack used on their rockets.
If they are using gcc and gdb, it sounds like everything is C. Which is a
scary thought. :\

~~~
wyager
I would be scared if they _weren't_ using C (or a derivative like C++).

C is a highly deterministic language. It has no "features" per se. If you
point to a line of C code, I can tell you with very high certainty what
assembly opcode(s) it will compile to. The only thing C does that might be
considered a "feature" is memory abstraction, which is kind of important for
any large programming project.

Now, if you're trying to advocate using Java or Python or something, you've
obviously never worked on embedded software. Not only does any very-high-level
language introduce much higher resource requirements, but they are also much
less deterministic (which is BAD for spacecraft).

Proper C code is not less stable than any very-high-level language. In fact,
there is a lot less that could go wrong.

~~~
pjmlp
Actually I am really scared that they use C or C++ instead of something sane
like Ada.

Unless their static analysis tools just make C or C++ look like Ada.

~~~
testbro
I wouldn't be supplied if they used MISRA [1]. It can be enforced just using
static analysis too, and IIRC it's good enough for avionics used elsewhere.

[1] : <http://en.wikipedia.org/wiki/MISRA_C>

------
avar

        > Linux runs everywhere at SpaceX,
        > he said, on everything from desktops
        > to spacecraft.
    

Yet every time I've seen them advertise for sysadmin sysadmin or programmer
positions the indications are that at least all their internal network and
website run heavily on Microsoft software, but their actual spacecraft use
Linux/embedded.

It would be interesting to get an actual overview of what they use Linux for,
and what they don't.

~~~
devbug
They use Linux for embedded systems, i.e., the Dragon and Falcon; and Windows
on the ground.

[1]
[http://www.reddit.com/r/IAmA/comments/1853ap/we_are_spacex_s...](http://www.reddit.com/r/IAmA/comments/1853ap/we_are_spacex_software_engineers_we_launch/c8bodrj)

------
jacques_chester
> _They have tried various techniques for estimating time requirements, from
> wideband delphi to evidence-based scheduling and found that no technique by
> itself works well for the group. Since they are software engineers, "we
> wrote our own tool", he said with a chuckle, that is a hybrid of several
> different techniques. There is "no silver bullet" for scheduling, and it is
> "unlikely you could pick up our method and apply it" to your domain._

It so happens that I'm working on a tool that amongst other things would allow
combined PERT 3-point estimates and Delphi Wideband estimates.

However the most important lesson in estimating is: do not rely on one
estimation method. Ever. Use as many as you can. For high stakes estimates,
try to have at least one of each of the 4 main classes of estimate: judgement,
analogical, parametric and decomposition/recomposition. My tool fits in the
4th category, which is IMO currently underserved.

Those of you in the machine learning world will recognise the vibe of ensemble
learning.

------
contingencies
Triply-redundant hardware must be a power consumption headache in such
environments. Anyone know what kind of power source these things use?

~~~
w3pm
The clockspeeds on the processors they're using are probably relatively slow.
Plus they don't have to worry about retina displays or WiFi or other major
power-consumers in today's devices... I bet the overall consumption isn't that
bad.

~~~
avmich
You bet. The problem of getting to orbit was solved "somehow" in 1950's, with
emphasis on reliability, then terminal control optimized the result some
decades later, when the hardware became available... But the basic task is
still way below capabilities of even modest modern computers. Space X may
afford to have quite a bit of redundancy.

------
kabdib
"Detecting the use of Emacs..."

Is that for real, or tongue-in-cheek?

------
martinced
_"The Byzantine generals' algorithm is used to handle situations where the
computers do not agree. That situation could come about because of a radiation
event changing memory or register values, for example."_

This always got me wondering: what happens if the algorithm used for handling
that is itself affected by a radiation event? Is that just too unlikely
because such few code is executed? Or what if the computer in charge of
verifying that the other computers do come up with a correct answer is itself
dying, isn't it a SPOF?

(curious mind wants to know)

~~~
wyager
Generally speaking, BGA solutions to fault-tolerance work by distributing
everything across multiple computers. So every computer compares results with
every other computer, and if a minority of computers find themselves in
disagreement with a majority, the computers in the minority adopt the stance
of the majority.

~~~
SeanDav
I don't believe this addresses the point the OP was trying to make. There is
an algorithm or just a simple piece of code which does the distributing, what
safeguards this algorithm?

Another way of putting it is "who watches the watchers?".

~~~
wyager
That's the whole point. There is no central authority; all computers are on
the same level. Each computer does the calculations independently, and then
they compare results. The computers in the majority either get the minority
computers to accept the results or they somehow override the minority
computers. I'm not sure how they negotiate, for example, low-level hardware
access, but that's just an implementation issue.

Take a look at the Bitcoin protocol. It's a very impressive example of many
different computers agreeing upon something precisely, despite the fact that
many parties have a vested interest in disrupting the Bitcoin network.

