
Apollo 11 Had a Hidden Hero: Software - psim1
https://www.wsj.com/articles/apollo-11-had-a-hidden-hero-software-11563153001?mod=rsswn
======
a-wu
I recently started listening to a fantastic podcast called 13 Minutes To The
Moon[0] produced by the BBC. It’s a retelling of the last 13 minutes until the
Apollo 11 lander touched down. Episode 2 talked about the software and the
1202 program code. They interviewed the Guidance controller who was only 26 at
the time and talked about how the average age of the flight controllers was
27. I thought that was incredible, you don’t see that much responsibility
placed in such junior people anymore.

[0]
[https://www.bbc.co.uk/programmes/w13xttx2](https://www.bbc.co.uk/programmes/w13xttx2)

~~~
Angostura
It's a really good podcast, but I found it a bit bloated in places, I could
have done without the Hans Zimmer theme, of which they are so inordinately
proud.

~~~
a-wu
That’s funny because I actually love the Hans Zimmer theme.

~~~
dereknewsom
I really loved the Hans Zimmer music too. I got pumped up every time I heard
it. I felt some bloat at times when they really dug into that 1202 error. It
just kept coming up again and again.

~~~
philjohn
Yes - it's bloated for a reason ... all of the things they've spent long
episodes on are so you know what's going on when they do the realtime
recordings of cap com.

------
spamizbad
CuriousMarc's Apollo Guidance Computer (AGC) restoration is a must-watch:
[https://www.youtube.com/watch?v=2KSahAoOLdU&list=PL-_93BVApb...](https://www.youtube.com/watch?v=2KSahAoOLdU&list=PL-_93BVApb59FWrLZfdlisi_x7-Ut_-w7)

~~~
acomjean
I’ll second that. I particularly liked the part on home the memory worked

~~~
SpikeDad
I'm amazed at the technical skill and knowledge he and his team has.
Fabricating simulations, replacement parts, unpotting modules and repairing
them, etc, etc, etc.

I thought the Xerox Alto restoration was amazing - this beats it handily.

------
playing_colours
There is a good book on the computer systems of Apollo project: Digital Apollo
[https://mitpress.mit.edu/books/digital-
apollo](https://mitpress.mit.edu/books/digital-apollo)

It describes interesting questions that were faced on the project: say, how
much control over a lunar module could be given to a computer system, and how
much they should rely on human engagement.

~~~
bumby
It's interesting to me that 50+ years later this debate on system control is
still very much relevant to aerospace today, like the recent 737-MAX issue.

~~~
bronson
There's not really any debate. The MAX ignored very basic controls lessons
that were learned and institutionalized, as you say, over 50 years ago.

~~~
bumby
I agree that the 737-MAX wasn't a good example.

My limited understanding in a broader sense (as opposed to the specific
737-MAX case) is there is still considerable debate on who should have
ultimate control (human or software) of the aircraft when making critical
decisions. I remember reading an article a while back that made it seem like
Boeing and Airbus were on different sides of this spectrum in basic design
philosophy, but I don't know how true that is.

~~~
bronson
It's like any good technical religious debate: both sides have good points and
the ideal that they're both working toward is somewhere in the middle.

Airbus has backed off trying automate everything (starting with AF 296 in
1988) and, as this MCAS debacle has demonstrated, Boeing is happy to add
serious automation. The days of the "pilot's plane" are long over. Now it's
ever thinnner shades of gray.

------
roboman
Best quote in the excellent article:

“You can’t get a de­gree in how to fly to the moon,” says Dana Dens­more, who
joined the lab in 1965 and be­came a con­trol su­per­vi-sor for the lu­nar-
lan­der soft­ware. “You had to get peo­ple who know how to think, who are
cre­ative and alert. It was all in-vented on the spot.”

~~~
dTal
She certainly sounds interesting.

[https://en.wikipedia.org/wiki/Cell_16](https://en.wikipedia.org/wiki/Cell_16)

------
_eric
Kind of off topic, but for those interested, this reminded me of the public
repo on github which has the original guidance computer source code for the
command and lunar modules[0].

[0]
[https://github.com/chrislgarry/Apollo-11](https://github.com/chrislgarry/Apollo-11)

~~~
retSava
Went looking for some smart-ass pull request (tongue in cheek) and was gladly
surprised there were some! Alas, just proof reading changes and such.

------
mopsi
This is the best description of AGC I have seen so far, from hardware design
to mission software, layer by layer: "The Ultimate Apollo Guidance Computer
Talk"
[https://www.youtube.com/watch?v=xx7Lfh5SKUQ](https://www.youtube.com/watch?v=xx7Lfh5SKUQ)

------
Zanneth
If anyone wants to learn more about the Apollo Guidance Computer, this talk[1]
from 34C3 will teach you everything you wanted to know and more.

The AGC had some really fascinating features for the time, including a
cooperative multitasking system, interrupt driven IO, and even a virtual
machine for doing linear algebra!

[1][https://youtu.be/xx7Lfh5SKUQ](https://youtu.be/xx7Lfh5SKUQ)

------
johncalvinyoung
I knew a retired couple near where I lived growing up who, I found out later,
were software programmers on the Apollo Program. Quite possibly worked in that
lab. The husband passed away soon afterwards, and I never got around to asking
the wife about her experiences (they both worked for the program!). But it was
really cool reading about how the core software was written.

------
jonshariat
I've been listening to "13 minutes to the Moon" from the BBC recently.
[https://www.bbc.com/news/science-
environment-48232627](https://www.bbc.com/news/science-environment-48232627)

One part jumped out at me as brilliant especially for the time: when the
computer was overloaded with tasks during the decent, it switched off all non-
essential extras like screen displays in order to free up processing power.
After that it shed even more to focus on only plotting the landing!

~~~
zeristor
A few weeks back they said it was due to the delta h discrepancy, and Buzz did
nothing wrong. This week they were saying that it was down to the docking
rendezvous system being turned on, which had not been done before.

I don't know if they came up with new information and forget to redo that
episode.

The delta-h discrepancy was a mismatch between the actual height above the
moon, and what the navigation system expected, I believe Mission Control took
over that.

I feel like the programme murkied the waters.

------
mmoez
Fastcompany published a similar article about the 4th crew member of Apollo 11
a few weeks ago. I shared it on HN but it did not get any traction.

[https://news.ycombinator.com/item?id=20173672](https://news.ycombinator.com/item?id=20173672)

------
roboman
Do yourself a favor: head to a newsstand or Starbucks and buy a copy of
today’s WSJ. This special insert on the Apollo mission is well worth it.

------
fortran77
It's nice to hear the Don Eyles story; I haven't heard it before, having only
seen stories about Margaret Hamilton and her role.

------
ozmaverick72
The article mentions the overload on the computer was caused by "a mismatched
power supply to a radar" triggering too many calculations. Any idea how or why
the power supply was mismatched ? Luckily software saves the day but what was
the root cause of the hardware fault ? Was that a case of a mismanaged change
in configuration ?

~~~
detaro
If I remember correctly, two AC power sources were required to be in phase for
measurements of the antenna direction to work correctly, but weren't actually
specced as such, and thus not built that way, and randomly ended up so out of
sync that it caused problems on that flight.

EDIT: this article has more detail
[https://arstechnica.com/science/2019/07/no-a-checklist-
error...](https://arstechnica.com/science/2019/07/no-a-checklist-error-did-
not-almost-derail-the-first-moon-landing/2/)

------
EvanAnderson
"The Charming Genius of the Apollo Guidance Computer" is a wonderful talk:
[https://www.youtube.com/watch?v=xY45YE7ggng](https://www.youtube.com/watch?v=xY45YE7ggng)

The AGC implemented a virtual machine to run higher-level computational
routines. The whole architecture is really interesting.

~~~
rst
Eyles has a memoir out: info available here. (The title is "Sunburst and
Luminary", which were the names for LM software builds for earth-orbit testing
and lunar landings, respectively.)

[http://www.sunburstandluminary.com/SLhome.html](http://www.sunburstandluminary.com/SLhome.html)

------
f_allwein
> When it came to hiring women for engineering or management, NASA “had a few
> women, and they kept them hidden,” says Ms. Densmore. “At the lab it was
> very different,” and there were opportunities for women.

Quite amazing what an important role women like Margaret Hamilton played in
the Apollo project, given how hard it is nowadays to get more women into
engineering positions. You may remember the famous picture of Hamilton next to
the stack of code she and her team wrote for Apollo: see
[https://en.m.wikipedia.org/wiki/Margaret_Hamilton_(software_...](https://en.m.wikipedia.org/wiki/Margaret_Hamilton_\(software_engineer\))

------
thomasjudge
Love the code comments referenced in the article

------
roboman
“You can’t get a de­gree in how to fly to the moon,” says Dana Dens­more, who
joined the lab in 1965 and be­came a con­trol su­per­vi-sor for the lu­nar-
lan­der soft­ware. “You had to get peo­ple who know how to think, who are
cre­ative and alert. It was all in-vented on the spot.”

------
elteto
This contains more technical information behind the 1202 alarms [0]. It is an
awesome read.

[0]
[https://www.hq.nasa.gov/alsj/a11/a11.1201-pa.html](https://www.hq.nasa.gov/alsj/a11/a11.1201-pa.html)

~~~
sytelus
Relevant text:

 _So what was happening during Apollo 11, as I recall, was that repeated jobs
to process rendezvous radar data (that of course were not really there) were
scheduled because a misconfiguration of the radar switches. Thus, the core
sets got filled up and a 1202 alarm was generated. The 1201 that came later in
the landing was because the scheduling request that caused the actual overflow
was one that had requested a VAC area._

Quite fascinating that these alarms automatically rebooted computer and picked
up essential jobs exactly where they had left off.

------
criddell
> MIT received the first major Apollo contract, the only one awarded to a
> university, and the only one given without competitive bidding.

Do these kinds of mission-critical components generally go to the lowest
bidder?

~~~
ghaff
Competitive bid doesn't have to mean lowest bidder. I've put tons of (non-
government) projects out for bid because you're usually expected to do so to
keep everyone relatively honest. I imagine the MIT Instrumentation Lab were
the only people with the inertial guidance expertise at that time.

~~~
benj111
"Competitive bid doesn't have to mean lowest bidder"

No you have to wine and dine the right people, and line a job up for them when
they retire.

~~~
Jtsummers
Which gets someone thrown in jail or a lot of fines these days. Not that it
doesn't happen, but it's a lot harder to hide now than 20+ years ago.

~~~
benj111
I was perhaps being slightly misanthropic. But the game keepers always seem to
turn poacher. High ranking military bods end up with defence contractors.
Financial regulators end up with working at banks (and visa versa), and
politicians seem to end up getting various well paid consulting jobs. I
suppose you could argue there are good reasons for it, I cant believe on some
level these public servants aren't thinking of their career after retirement.

------
mulmen
Neat story but is the software really hidden? I feel like I have heard this
story many times. It was one of the many feats of the Apollo program, I don't
think I would consider it hidden.

~~~
psim1
As a casual space program buff, I thought the same. But I think most people do
not realize how sophisticated this early computer and software were, and how
important to the mission. The article emphasizes the well-calculated trust
that the engineers and astronauts put into this first computer-controlled
spaceflight operation.

------
grandridge
Yeah, racist software

