
Software Engineering Within SpaceX - theanirudh
https://yasoob.me/posts/software_engineering_within_spacex_launch/
======
aphextron
>SpaceX also made use of Chromium and JavaScript for Dragon 2 flight
interface. I am not sure how that passed the certification. I assume it was
allowed because for every mission-critical input on the display, there was a
physical button underneath the display as well

I think that's all the validation we need for HTML/CSS/JS as the best tool for
UI development nowadays. I wonder if there was actual shared code from the
Dragon UI used in their online docking simulator. How neat.

~~~
cryptonector
The consensus regarding automobiles and touch interfaces is starting to form
that they are just a bad idea.

Physical switches, knobs, toggles, buttons -- these things can be activated
using one's hands _without_ needing to coordinate with sight, meaning, our
eyes can stay on the road.

There is no road to keep your eyes on in space though, so needing to
coordinate hands and eyes is clearly not that big a problem for the Dragon,
and might even be better than lots of physical inputs: you can cram more
virtual inputs into the same area by using menus and what not, and that might
make it easier to navigate one's way around them. Then again, complex menus
might make things worse in an emergency. There's not that much for the
astronauts to do in Dragon though, so it's probably all OK.

~~~
goshx
Do you have a source about the said "consensus"? It sounds more like
resistance to change than an actual rationale.

You have always had the need to look at the buttons, like reaching out to the
radio.

From my experience, the majority of people who drive a car with touch
interfaces don't want to look back to old fashioned buttons.

~~~
mlyle
> From my experience, the majority of people who drive a car with touch
> interfaces don't want to look back to old fashioned buttons.

Many automakers that had touch-heavy interfaces are moving back towards
physical controls, both because of market demand and evolving industry safety
considerations.

e.g. [https://www.motorauthority.com/news/1121372_why-mazda-is-
pur...](https://www.motorauthority.com/news/1121372_why-mazda-is-purging-
touchscreens-from-its-vehicles) [https://jalopnik.com/honda-follows-mazda-by-
ditching-some-to...](https://jalopnik.com/honda-follows-mazda-by-ditching-
some-touchscreen-contro-1842564375)
[https://www.usatoday.com/story/money/cars/2013/06/17/ford-
my...](https://www.usatoday.com/story/money/cars/2013/06/17/ford-myford-touch-
infotainment-sync/2432489/)

I have no problem with touch controls in general, but replacing a volume knob
I can find blindly with a relatively small pair of "Vol+" and "Vol-" touch
targets is mildly infuriating. It's OK as the driver because there is an
actual tactile control on the steering wheel, but downright unpleasant as a
passenger.

~~~
goshx
One of the issues with those automakers is that their interfaces are in
general unresponsive and slow. The one in my BMW sucked. The Jaguar I-PACE is
very slow as well.

Tesla got this right with both the touch screen and the physical buttons on
the steering wheel for things like volume control.

~~~
BeetleB
> One of the issues with those automakers is that their interfaces are in
> general unresponsive and slow. The one in my BMW sucked. The Jaguar I-PACE
> is very slow as well.

> Tesla got this right with both the touch screen and the physical buttons on
> the steering wheel for things like volume control.

Perhaps there's an auto manufacturer or two that got it right. However, I've
never driven a car with physical buttons that got it wrong. When I used to
shop for cars in those days, I _never_ had to consider if those buttons were
compatible with me. Now when I buy for a newer car, it's an added headache to
consider, and one that I've not seen add any real value. Going on a test drive
really will not tell me enough about whether the interface is good. And worst
of all, _whether it is or isn 't affects my safety_.

~~~
filoleg
>Perhaps there's an auto manufacturer or two that got it right. However, I've
never driven a car with physical buttons that got it wrong.

You gotta give it some time, because modern physical control interfaces in
cars had over half a decade to evolve to reach this point. With regard to
touchscreen interfaces in cars, it simply feels like those are still stuck
somewhere in the pre-iPhone era of touchscreen interfaces for phones in terms
of usability compared to physical controls.

------
danans
Years ago, astronaut Chris Hadfield told an audience of software engineers
(including me) that the moment the space shuttle was in stable orbit, the crew
would pull out laptops and set up an ethernet network for all the scientific
work of their expedition, as the space shuttle's own computers, though limited
in raw computing power, ran software that was so thoroughly tested that there
was every reason not to "upgrade" them in any way to support the scientific
work.

~~~
thechao
As a child, a friend's mother was a programmer for the Shuttle (mid-80s to
early 90s). Her job sounded _awful_ : a 'science person' (I was a child,
remember!) designed a piece of math; an engineer (software?) would take that
piece of math and reduce it to an algorithm; my friend's mother would program
that algorithm in machine code, in parallel to an assembly listing. Each
instruction (and all the data) would be reviewed line-by-line. She had to
provide reasoning (roughly a proof) that each instruction correctly
implemented the algorithm. My guess is the algorithm went through the same
equivalence checking to the 'math'. I don't know how much code she wrote, but
it was only a few programs (functions) _per year_.

That was my view of 'programming', and I wasn't disabused of that until very
late 90s.

~~~
rustybolt
Sounds lovely. Now I write code with vague requirements that interfaces with
ill-specified functions, on tight deadlines. It surprises me that it doesn't
contain a lot more bugs. Maybe we just haven't found them yet.

~~~
ngcc_hk
And whatever you wrote And right to the spec， user said it is not what they
want. And demo why it is not working or useful ... Of course that is nothing
to do with them, that is not what they said or your it guys analysis wrongly.
As spec goes ... do not understand it.

To be fair it takes a lot of interaction (and understanding ) to fix it. And
you wonder how these operate here.

------
tectonic
Good writeup! In general, the direction in modern aerospace is to use COTS
(commercial off-the-shelf) parts with redundancy and failback for radiation
hardening.

If you’re into this sort of thing, I co-write a weekly newsletter about the
space industry and often talk about software.
[https://orbitalindex.com](https://orbitalindex.com)

~~~
sq_
Thank you so much for the Orbital Index! I've been subscribed for a while now,
and I look forward to it every week. Great content!

------
extrapickles
When you think about it from a first principals perspective, having multiple
touchscreens is better than only having physical switches. When a switch is
damaged/fails, you are out of luck. When a touchscreen is damaged/fails, you
use the one next to it. On a rocket you do not have the mass or room to have
more than 1 of all but the most critical of switches.

There have been quite a few missions that nearly caused death or mission
failure directly due to a switch getting broken (Apollo 11, lander return
engine-arm switch) or going faulty (Apollo 14 abort switch).

What really matters is that they have no single point of failure (touch
screens can do everything switches can, an individual touch screen is not
important, and switches can cover abort/return scenarios to protect the crew).
For the software, it only matters that its been fully tested, including random
bit flips and hardware failure.

From a cost savings perspective, its vastly cheaper to verify that 3
touchscreens are working correctly than the 600 switches they replace.

~~~
colllectorof
_> When a switch is damaged/fails, you are out of luck. When a touchscreen is
damaged/fails, you use the one next to it._

This is a trivial problem to solve on a physical interface. One solution could
be what is commonly used on hardware synthesizers. A shift button or switch.
You engage it and all controls begin to perform their secondary functions. You
get redundancy for the price of one extra control and a secondary set of
labels in a different color.

Also, use of displays to virtually label buttonss is common. In such case you
can reassign a control if one fails.

In any case Dragon capsule had physical buttons for important functions as a
backup.

~~~
extrapickles
The touchscreen frees you from the complexity that comes with giving switches
alternative modes, and gives you the mass to have multiple copies of critical
switches. Also multimode switches greatly increase the complexity and failure
modes, so they need to be done so that if a switch is triggered in the wrong
mode its recoverable (eg: the switch for aux radio power isn’t also the undock
switch).

When you get to the point of having displays for the switches why not go full
touchscreen and eliminate all of that cost and complexity of a bunch of tiny
displays?

------
randtrain34
Note that the data within the post is from an AMA ___7 years ago_ __

~~~
Fiahil
Tech moved a bit in that timeframe!

> We leverage C#/MVC4/EF/SQL; Javascript/Knockout/Handlebars/LESS/etc and a
> super sexy REST API.

Nowadays it would be ".Net Core/SQL; Typescript/React/GraphQL"

~~~
pjettter
Webassembly

------
wlesieutre
Interesting read. I've wondered about their use of big touchscreen interfaces
having heard a friend's experience with the similar setup in a Model 3.

On multiple occasions they've had to pull off the highway to turn their car
off and on again to get the screen working. Not really an option on your way
to space.

~~~
rootusrootus
> On multiple occasions they've had to pull off the highway to turn their car
> off and on again to get the screen working.

Surely not? The touchscreen is run by the media computer which does not
control the car. You can reboot it with the 'two finger salute' while you are
driving down the highway. Some things will be unavailable (you cannot, for
example, engage autopilot while it is rebooting), but the car still runs &
drives.

I hope they just miscommunicated the situation to you, otherwise they are
really working too hard just to fix a touchscreen. Turning the entire car off
is kind of a pain in the ass. I've never done it. And I have only rebooted the
touchscreen a couple times ever. Your friend may want to schedule a service
appointment if they really do have to power cycle the whole car, because that
is super abnormal.

~~~
wlesieutre
I'm not suggesting it had any function on that automotive functions, but I was
also not aware of the media center reboot with steering wheel buttons. From
what he described to me I don't think he was either, I'd have to check and
will pass that along if not!

I'm a little surprised that the media computer doesn't have a built-in
heartbeat check and know to reboot itself if it stops responding. I've heard
of other cars and embedded systems doing that.

EDIT - asked him about reboot via steerling wheel:

 _> It’s not great because you lose lots of feedback. No speedometer, no sound
from turn signals, etc. But it does work._

~~~
mav3rick
It's called a watchdog. And it's 99% running Android and should have that.

~~~
comboy
Last time I've heard they were using Ruby on Rails..
[https://news.ycombinator.com/item?id=17835760&p=2](https://news.ycombinator.com/item?id=17835760&p=2)

------
yasoob
Hi guys! I am the author of this article. Excited to see it on the first page
:)

------
naringas
I would have expected them to use formal verification tools in the vein of
TLA+ and such... or maybe use ADA for mission critical systems?

But they only mention Astree[1] which seems to be a propietary analyzer for C
code

[1]
[https://en.wikipedia.org/wiki/Astr%C3%A9e_(static_analysis)](https://en.wikipedia.org/wiki/Astr%C3%A9e_\(static_analysis\))

~~~
dahfizz
TLA+ is basically an academic tool. It allows you to verify your
specification, but it is useless to detect bugs in your implementation. In my
experience, the process of writing your specification in TLA is a buggy enough
process to make it all a waste of time.

~~~
superqd
No, not just for academia. Apparently, Amazon uses it to check their designs
for correctness.

[https://lamport.azurewebsites.net/tla/formal-methods-
amazon....](https://lamport.azurewebsites.net/tla/formal-methods-amazon.pdf)

------
stevofolife
Taken from the article: "We leverage C#/MVC4/EF/SQL;
Javascript/Knockout/Handlebars/LESS/etc and a super sexy REST API."

Knockout.js, good times.

~~~
hu3
I believe Trello uses Knockout.js

And at least before Atlassian bought it (I haven't used since then), it was a
responsive and plasant piece of software.

------
chasd00
at the bottom of the article they mention model rockets and the three levels
of certification. Each level grants you access to more powerful motors and
therefore higher or larger flights. The hobby is self-goverend by NAR and
Tripoli who manage level certification.

It's a fun hobby, although large motors get pricey. The largest can be 4-5
figures per launch. However, you can get very advanced and do things you
wouldn't typically expect in a hobby.

Here's a two stage ( 4" diameter booster, 3" diameter sustainer ) reaching
over 200k feet in altitude. The Karman Line is about 330k feet.

[https://www.youtube.com/watch?v=g0imcpdLdB8](https://www.youtube.com/watch?v=g0imcpdLdB8)

------
mips_avatar
I'd love to work with physical software (software that interacts with the real
world through sensors and actuators), as a C developer, how should I move into
this space? Every time I try intro to ARM kits I feel like I'm in over my
head.

~~~
Rebelgecko
Start out with an Arduino, or ESP32/ESP8266. The ESP boards are probably the
best bang for your buck out there if you're just playing around— you can start
out with the Arduino environment (C++-ish), or use something like PlatformIO
to interface a bit more directly with the hardware. It's not as low level as
an ARM board where you have to worry a bunch about setting up clock
multipliers and all that jazz, but the power to do so is there when you're
ready for it.

The best way to learn is to solve a problem that motivates you. Maybe you want
a phone notification when your laundry is done, or when a room in your house
exceeds a certain temperature. Work on some little projects like that

If you want to get even lower level, then try out the ARM boards again.
Someone else mentioned STM32 boards, which are great in many ways but IMO not
very user friendly (Their STMCube software actually makes me mad). You can
also try your hand at FPGA development. Verilog and VHDL are both popular
languages, and preferences between the two tend to depend on domain. There's
compilers that let you program FPGAs in C, but shying away from that.

If you want to look for work, probably the big industries to check out are:

Consumer embedded hardware: companies that make printers, cell phone radios,
etc

Medical devices

Aerospace/defense

------
dang
A thread from a few days ago:
[https://news.ycombinator.com/item?id=23368109](https://news.ycombinator.com/item?id=23368109).
The current article quotes from it, in fact.

This article looks like a fine overview but when it comes to follow-up posts,
the test is: does the new submission contain enough SNI (significant new
information) to support a substantially different discussion? In this case it
looks like not, but I can't really tell.

[https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...](https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=by%3Adang%20%22significant%20new%20information%22&sort=byDate&type=comment)

[https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...](https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20follow-
up&sort=byDate&type=comment)

------
0xDEEPFAC
No mention of Ada or their methods of writing the important software. I wonder
what they use.

"Avionics Test team

...The main objective is to write very comprehensive and robust software to be
able to automate finding issues with the hardware at high volume...."

~~~
lavezza
The "important" software is the Flight Software that runs on Falcon and
Dragon. That is written in C++.

------
theanirudh
The astronauts show parts of the touchscreen and physical controls here:
[https://youtu.be/llbIzbOStt4?t=150](https://youtu.be/llbIzbOStt4?t=150)

------
drummer
It would be awesome if some SpaceX engineers would give a few presentations at
events like CppCon and talk about their software development process including
some code examples and demos.

------
chrisfinazzo
Hearing about the Flight Software and Avionics teams reminds me of this,
although they don't seem to be on that level quite yet.

[https://www.fastcompany.com/28121/they-write-right-
stuff](https://www.fastcompany.com/28121/they-write-right-stuff)

------
theanirudh
I wonder how they manage not to have accidental taps on the touch screen
during liftoff and or re-entry. As I understand there are a lot of G's and
violent vibrations and I would assume it's hard to keep a steady hand?

(Atleast this is my understanding from watching Apollo documentaries/movies
etc.)

~~~
NikolaeVarius
The screen compensates, and they train with vibrations in mind.

~~~
theanirudh
Impressive, especially considering some of the touch targets are pretty small.

~~~
NikolaeVarius
They probably aren't bothering with using non critical buttons on a launch
that is automated anyway

------
oxguy3
I'm so relieved to hear all the redundancy and testing in place. I had heard
that the touchscreens were built in Chromium/JS and was rather alarmed. Don't
get me wrong – I do a lot of web stuff and I love that environment, but I've
never seen a web app I would trust two human lives to. This, however, sounds
like they really thought it through and made it safe.

~~~
paxys
A contrary opinion - a tool/framework used and tested by billions of people
every day is a lot less prone to crashing when used for its intended purpose
than something custom-built. There are tons of developers out there building
complex apps by following beginner JavaScript tutorials, but SpaceX is
obviously going to enforce better standards. HTML/CSS/JavaScript/V8 are all
extremely solid technologies that have stood the test of time, and there is
nothing better to build a user interface with today.

------
ChrisMarshallNY
This was cool!

Thanks for sharing that with us.

I'd be interested in finding out how they iterate. I'm absolutely positive
that they do.

~~~
thoughtpalette
Off topic but we share the same name. Except I'd be ChrisMarshallCHI.

~~~
ChrisMarshallNY
I met another Chris Marshall that worked for Kodak, back in the 1990s. Have
you ever worked for Kodak?

~~~
thoughtpalette
I have not. Though, I have been eyeballing chrismarshall.com for years,
setting calendar reminders on expiration's and such.

Have you been doing the same? Was curious if all the CM's are competing for
that domain lol

~~~
ChrisMarshallNY
I've had cmarshall.[com|org|net] for a long time. I recently also fetched
chrismarshallny.[com|org|net|dev], along with a bunch of various social media
handles (see my HN ID). Right now, they point to my LLC site, but I'll
probably set up a personal site; sooner or later. I did notice that a squatter
had grabbed the Pinterest name. I don't care.

I've decided that ChrisMarshallNY is my "Google Me" name. It's working fairly
well.

~~~
thoughtpalette
Ahhh nice. Fair enough. I settled on thoughtpalette on everything but I wish
whomever owned our namesake domain would do something with it \\_O.o_/

------
fallingmeat
Does this article imply that RTCA/DO-178B is used as a means of demonstrating
compliance in some way, or otherwise is used to define lifecycle processes for
their development/verification/systems teams? Anyone know where this was
mentioned by SpaceX?

------
scep12
> The Flight Software team is about 35 people.

I'm shocked the discussion is about UI tech and not that there was only 35
people on the team that built the software to land Falcon 9.

Surely it's changed in the last 7 years. Anyone know the size now?

------
MrSaints
They will be doing an AMA on Reddit again soon according to
[https://youtu.be/y4xBFHjkUvw?t=674](https://youtu.be/y4xBFHjkUvw?t=674)

------
jonpurdy
> The secondary ports go into the primary ports, which are heavy-duty
> actuators that connect to what’s called a “summing bar,” which is no more
> than a massive steel rod.

"In Rod We Trust"

------
sammycdubs
This seems like a remarkably small team for the scale of what they're
building! For some reason I imagined they'd have legions of engineers.

~~~
bdcravens
I think we’ve been conditioned to believe that any application of medium
complexity requires hundreds or thousands of developers. Snarkiness aside,
they have a very concrete set of functional constraints and total control over
the hardware, network, and software environment. Issues like resource
management, dependency conflicts, security, scale, etc are taken out of the
equation, things that are often the biggest time and resource sinks.

~~~
junon
IME, more bodies usually always equates to more bugs and slower release
cycles.

------
Animats
It's interesting that the mission control console systems are written in
LabView.

~~~
themeiguoren
I can see how it’d be a decent tool for plotting up a bunch of raw telemetry
streams, but as someone who had to write a moderately complicate program in
LabView once, I’m astonished they’ve scaled it up that far and very glad it’s
not me who had to do it. It’s a real PITA to work with.

------
b20000
doesn't sound like a great idea to involve 10 different technology stacks

------
f00_
touch screens are a bad choice to me

I want the buttons and knobs.

Love the old soviet control rooms posted awhile ago:
[https://designyoutrust.com/2018/01/vintage-beauty-soviet-
con...](https://designyoutrust.com/2018/01/vintage-beauty-soviet-control-
rooms/)

Need John Carmack's opinion of SpaceX

~~~
josefresco
Oh so beautiful: [https://main-designyoutrust.netdna-ssl.com/wp-
content/upload...](https://main-designyoutrust.netdna-ssl.com/wp-
content/uploads/2018/01/18-12.jpg?iv=146)

~~~
f00_
i love the windows xp screen saver ):<

