
The Loyal Engineers Steering NASA’s Voyager Probes Across the Universe - gaius
https://mobile.nytimes.com/2017/08/03/magazine/the-loyal-engineers-steering-nasas-voyager-probes-across-the-universe.html
======
laydn
You know, getting these electronics systems ready, sending out to deep space
and communicating with these systems is an incredible feat.

But, for me, another amazing thing about these long term projects is the sheer
amount of knowledge that needs to be maintained and passed on to new people,
over the course of several decades. To pull it off, this need of knowledge
management and transfer must be so deeply engraved in the culture of the
organization. How do they do it in this day and age of "FAQs", "Forums" and
"Helpdesks"? :)

In all seriousness, what do you do to maintain "knowledge" in your
organization?

~~~
bitexploder
This is a good insight. Having read about NASA's culture some from books like
"An Astronaut's Guide to Life on Earth", they tend to be very dedicated to
documenting everything. Every last little detail goes into handbooks or
manuals about how they fixed problems or things that could have gotten someone
killed, adapting procedures to take into account even the smallest of
practical details. These engineers come from a different era. You can see all
the printed binders on their desks. I think in many ways the process of
documentation for space missions is more expensive, pain staking, and solid
than modern practices. I think also they do have the advantage of all working
in the same place. You just go to Tom, the human search engine/librarian, and
ask him what he knows about thing X. (The article mentioned Tom was the self
styled "librarian" of the project). Most organizations need a "Tom" of their
own and are poorer for not having one.

Our company is building a remote team, and similar to Gitlab, we are
discovering that staying in sync is not just a good value but a way of life.
Trello, wiki, and code help us repeat our assessment work and ensure everyone
is on the same page.

~~~
marsRoverDev
To give some kind of context on more "modern" methods, we use stupendous
numbers of PDFs, DOORS for requirements traceability and tons of code
commenting. At last count our project had 0.73 commenting lines per line of
code (and we use Allman style bracketing).

We are also putting non mission-critical information in a Wiki; basically as a
tl;dr of the more dense documentation.

~~~
NikolaeVarius
Oh man, I hated DOORS so much. POS software. I dont doubt its useful, but
using it was the bane of my existance

~~~
InternetOfStuff
Heh, DOORS is pretty crappy. But once you get used to its flavour of
crappiness, you can get shit done.

I think it's one of those tools which demand a lot of expertise in setting
them up, or you will paint yourself in a corner, with explosive paint.

------
bitexploder
There is so much cool stuff in this article. The engineers stuck in the 70s
with modern tools to aid them, but that only gets them so far. The love and
dedication to this spacecraft. These engineers believe in the mission of space
so deeply they married themselves to Voyager. A marriage more substantial than
many real marriages. Just thinking about this dedication and how space
exploration is, in my opinion, one of the few ways we can truly advance as a
species beyond our ruts here on Earth gave me chills. There is something,
existential, poetic, and sad about the lonely work these engineers have done
for so long.

If you didn't read this article the portraits of the engineers and snapshots
of the lives is moving.

~~~
rootbear
I love the Voyagers and I'm very happy that there are people dedicated to
keeping us in contact with them as long as they last. These Engineers are
splendid examples of taking the long view, something we could use more of
these days.

~~~
wjnc
For me the article has a somewhat sad component (next to technological
awesomeness):

"In retirement, Zottarelli told me, he would like to see Florida again. He
wonders how it has changed. In his garage is a 1954 Swallow Doretti, a fixer-
upper. ‘‘It probably needs new brakes,’’ he said. I asked him if there was
anywhere he liked to drive for fun. ‘‘No,’’ he replied. ‘‘Not anymore.’’"

Imagine that's your grandfather. With his expertise, shouldn't he be able to
'see Florida twice a year'? I would have wished for a way to have both this
awesome technological specialization over long stretches of time, but also
given the people on the program some more time to go their own ways. And then
the toll of being in the same hierarchy for 30+ years. Ack. But perhaps these
people self-selected for stoicism, who am I to judge ;)

~~~
secstate
That's a really well made point. And for what it's worth, stoicism doesn't ask
you to never travel or have fun :)

The most depressing line for me was him effectively waiting (albeit in a
joking manner) for his second stroke. No one should be worked to the point of
fatalism.

On the one hand I love the team's dedication. On the other hand, I don't think
it's just competitive pay that keeps young engineers out of that office. I
imagine that the energy and excitement in the early 80s does not really exist
anymore, and it's not fair to dump on young engineers for wanting that energy
where they work.

~~~
mturmon
The younger engineers ar JPL might rather work on Mars 2020:
[https://www.jpl.nasa.gov/missions/mars-2020/](https://www.jpl.nasa.gov/missions/mars-2020/)

One of the proposed instruments is a 1kg helicopter.

 __*

They have tried to build a pipeline with the Mars exploration missions, in
which senior people can move from mission to mission, and younger people can
train up on older, mature missions before they move to newer ones.

This pipeline did not really exist with the Voyagers.

------
sehugg
_... its oscillator, which allows it to accept a wide range of frequencies,
had quit, essentially shrinking the target for transmissions from Earth.
Assuming a much narrower bandwidth, and manually subtracting the Doppler
effect, they recalibrated their signal. It worked — but to this day, the same
calculation must precede every command._

IIRC from an old SciAm article, the temperature of the spacecraft's
electronics is also taken into account when doing this calculation.

~~~
pmcjones
Yes, the article later says "(On Voyager 2, because of the broken oscillator,
any change in temperature also tweaks the receiver frequency.)"

------
rootbear
A lot of what has been said here about software documentation and process is
true of NASA flight projects, especially manned. Outside of flight, not so
much. In some cases, it's a bit more like grad-school-project-that-escaped. An
exception would be the modeling code used for things like weather simulation
on supercomputers. The scientists need to know that their code actually
matches the model they're testing and that things like numerical precision and
error bounds are being handled properly. I'll add that the above is anecdotal,
based on my experiences at NASA Goddard.

~~~
mynameisnobody
Yup.

I worked in a satellite data office of NASA and there was no formal
documentation process. Much of the code running in the facility had been
written by a single developer over 20 years, and all of the knowledge left
with him. Masterful Perl can be frustrating stuff without documentation or
comments.

------
rodgerd
I recommend seeing The Farthest
([https://www.youtube.com/watch?v=znTdk_de_K8](https://www.youtube.com/watch?v=znTdk_de_K8))
if you have any interest at all in this.

~~~
pat_space
Wow, I got goosebumps from the trailer. Thanks so much for the recommendation.

~~~
mitchdoogle
Looks like it will air on PBS on August 23. [http://www.pbs.org/the-
farthest/home/](http://www.pbs.org/the-farthest/home/)

------
dingo_bat
How do you retain such talented people for such a long time? These people
could get jobs in the most cutting-edge company in their respective fields.
Still they choose to stay at the same place for decades.

~~~
Merad
It's a combination of things. Many (most?) of the people want to be there and
are passionate about what they're doing. Financially it's still a great job by
normal standards and once you have tenure (I think they may call it something
else) it's an extremely secure job.

I have a friend who works there - brilliant guy who went to school for physics
but is also very talented at low level embedded systems type development. When
he graduated with a physics masters he turned down a six figure offer to start
at NASA at around $60k. From what he's told me though, he's essentially
guaranteed to hit the six figure mark within 6-8 years. Last time we spoke he
had just started his PhD (paid by NASA) and was working on the Mars 2020
rover.

~~~
user5994461
It's also a matter of location.

6 figures in SF/NYC can be less than 60k somewhere else.

------
rbanffy
In the end, the engineer's prayer is probably a lot like the astronaut's.

"Please, dear God, don't let me fuck up."

------
yourapostasy
> ...[the heliosphere] blocks 75 percent of cosmic radiation...

It sounds like a solar system-scale Van Allen belt [1] when described like
this. So we have at least three layers of cosmic radiation protection
identified (heliosphere, outer VAB, inner VAB).

If the heliosphere has implications for high-precision, high-accuracy
independent-of-Earth interplanetary spaceflight similar to gravimetric and
magnetic readings on Earth do for submarines today, then mapping the
heliosphere would be a future asset.

[1]
[https://en.wikipedia.org/wiki/Van_Allen_radiation_belt](https://en.wikipedia.org/wiki/Van_Allen_radiation_belt)

------
danesparza
This article is pretty dang cool. Looking over the pictures of the NASA
engineers I couldn't help but think "Damn, that engineer looks TIRED". Did
anybody else think the same thing?

------
nlh
One thing I've always been amazed at when reading about spacecraft is the
incredibly detailed level of control the scientists have over the craft.

It's something I'm not used to here on earth -- when something breaks, I've
learned you either crack it open and replace the part or buy a new device :)

Can someone talk (or provide a link) about how this sort of system design
works?

------
mrunkel
I read far too far into the article before I realized it was the mobile site.
:)

In the mid 1980's I went on a tour of JPL in Pasadena and actually saw the
computers that were (at the time) in charge of recording and storing the
telemetry data. I'm vague on the exact details, but apparently the computers
were donated from the US Army and were field models (early "portable"
computers) and they operated on 48V DC power. So not only were the computers
themselves large fridge sized units, they had near matching transformers that
were fairly unreliable.

I recall reading that in the mid 90s NASA replaced all the mission control
systems for the space shuttle with a single Sun Workstation, so I assume that
JPL also at some point replaced the downlink computers for V'ger.

My hat is off to these many fine people who stuck it out with jobs that were
probably long periods of drudgery interspersed with moments of sheer terror.

------
CocoaGeek
Love the Sun workstation on Enrique Medina's desk 8)

~~~
FireBeyond
At the tender age of 39, I vividly remember my first IT job, for a bus company
of all things, where at age 20 I had a Sparc 5 on my desk. I still love those
Sun keyboards.

(We used it to solve for the most efficient driver/ bus scheduling).

------
odammit
I didn't know Sammy Hagar was moon-lighting at NASA. Very impressive.

