

Parts installed “upside down” caused Russian rocket to explode last week - moubarak
http://arstechnica.com/science/2013/07/parts-installed-upside-down-caused-last-weeks-russian-rocket-to-explode/

======
smoyer
"young technician"

I sure hope he doesn't end up being both a scapegoat and deemed a criminal.
Problems like this are _exactly_ why you let younger technicians be mentored
by older (and ostensibly wiser) ones. It's also why you have visual
inspections done by a different party.

~~~
bjz_
It's also why you design the parts so that they are _impossible_ to install
upside down. No matter how clearly things are labelled, even the most
experienced people will make mistakes.

~~~
crististm
The shuttle had a similar problem with a large piece of round metal that had
more than one way to install of which only one was right. The mechanics knew
this and asked for some markers on the piece to ease the process. The markers
were never added not because of the cost of four paint points but because of
the documentation that had to be updated.

So there is your problem - bureaucracy. I'm sure that an engineer knew about
the sensor mounting problem but nobody cared enough to make the change because
of the paperwork.

------
onan_barbarian
This is an illustration of Murphy's Law, almost perfectly:

"If there's more than one way to do a job, and one of those ways will result
in disaster, then he will do it that way."

Defensive design would be to make sure, as others have state on this thread,
that it's impossible to make a mistake (e.g. an asymmetrical mount for the
part).

~~~
mikeash
And Murphy coined his Law after being screwed by a part that was installed
upside down!

------
compumike
[http://www.youtube.com/watch?v=ipNjYMJo79o#t=77s](http://www.youtube.com/watch?v=ipNjYMJo79o#t=77s)
\-- ignition begins at about 1:20

~~~
andrewflnr
And now we know why they insist on such large perimeters. Imagine being on the
side it was tipping toward.

~~~
InclinedPlane
You mean like these folks? [http://www.youtube.com/watch?v=rfuXUr-
_Rns](http://www.youtube.com/watch?v=rfuXUr-_Rns)

------
SEJeff
This is why (in C code) I prefer what some people call "yoda conditions"[1].
It prevents problems where you accidentally assign when you mean to check
equality. Like mentioned above, you should engineer everything, software and
hardware, defensively. If the design simply forbade installing a part upside
down, it would make up for the pathetic QA/QE which caused this issue in the
first place.

[1]
[http://en.m.wikipedia.org/wiki/Yoda_Conditions](http://en.m.wikipedia.org/wiki/Yoda_Conditions)

~~~
krasin
Clang (and GCC 4.7, I believe) would issue a warning for that. Instead of Yoda
Conditions, it's recommended to use -Werror.

~~~
jussij
Microsoft C/C++ compilers also pick this up as a warning if the warnings
settings are set to max.

------
natch
Original source article:
[http://www.russianspaceweb.com/proton_glonass49.html](http://www.russianspaceweb.com/proton_glonass49.html)

------
iano
A similar issue was responsible for one of the V22 crashes ([1], page 4). In
that case the gyro wires were hooked up backwards, instead of the unit
installed backward. Both cases are good examples of small design details that
make or break a project.

[1] -
[http://www.fas.org/man/dod-101/sys/ac/v22-report.pdf](http://www.fas.org/man/dod-101/sys/ac/v22-report.pdf)

------
waster
This reminds me of a comment under the Asiana Flight 214 discussion at
[https://news.ycombinator.com/item?id=6012214](https://news.ycombinator.com/item?id=6012214),
that outsider views can help. In this case, as in the case of the Mars Climate
Orbiter
([http://en.wikipedia.org/wiki/Mars_Climate_Orbiter](http://en.wikipedia.org/wiki/Mars_Climate_Orbiter)),
perhaps review of the work by someone who hasn't been scrutinizing it all day
would have caught the errors? You stare at the same thing all day, and it's
easy to get lost/not see your own mistakes. This can apply to supervisors, who
might be inclined not to review their supervisees' work carefully because they
_always do such good work_ that it really doesn't occur to them that something
this big could go wrong. And at the same time, they may well have been staring
at it too long themselves.

And as an aside, isn't the Russian space program always "beleaguered"?

~~~
VladRussian2
>And as an aside, isn't the Russian space program always "beleaguered"?

a decade old story about one of the unsuccessful launches goes this way - "the
salary wasn't paid for several month, so not much work was done, once the
launch date came within a month the government finally paid the salary and the
satellite was built really fast, though some testing and some other things
didn't really make it ..."

------
InclinedPlane
Lest you think this sort of thing is something that could only happen to the
Ruskies, consider that the same sort of mistake has happened on NASA space
probes, twice.

The Galileo Jupiter atmospheric probe deployed its parachutes late because the
accelerometer designed to trigger the pyro charges to release the parachutes
was wired backwards. There were two different "g-switches" which provided
acceleration data at different scales. When the probe entered Jupiter's
atmosphere the software control system saw that the acceleration data it was
getting was all wrong, it marked one of the channels as failed and attempted
to use just the other channel, though the scale was all wrong so the data was
bogus (leading the software to think that the probe was in a different stage
of its entry than it actually was). The result was that the main parachute was
deployed very late and the only reason the parachute successfully deployed at
all at the speed and atmospheric density it was released into is by more or
less sheer luck.

The same mis-wiring happened on the Genesis sample return probe, which caused
it to collide with the ground rather than be captured in mid-air as per the
plan.

The funny part is that the accelerometers on both probes were tested in a
centrifuge, but the wiring harnesses for the tests were also wired backwards
as well.

~~~
grinich
The lack of integrated testing was by far the most impressive thing about the
Curiosity Rover/MSL landing to me. Most people who followed the news had no
idea that the skycrane deployment setup was never actually tested in full.
They validated components, like the hydrazine engines, radar, and bridle
release mechanism, but not the combination of all.

Software and circuits scale through abstraction and the static discipline. But
combining atoms just gets harder and harder.

(This sort of thing also happens when constructing buildings or bridges, but
there's usually not a million different points for potential catastrophic
failure.)

------
robomartin
"each of those sensors had an arrow that was suppose to point towards the top
of the vehicle"

It's a little unbelievable to me that aerospace designers today would design a
part who's only means of ensuring proper installation and alignment is a
sticker with an arrow.

If it was truly the case the error was one of engineering and not the fault of
the technician who installed the parts. Engineers working on such projects
should not make these kinds of mistakes.

As an EE I had to learn mechanical design on my own. I quickly learned that it
is paramount to have DFM (design for manufacturing) become a part of your
design DNA. You need to think DFM for every little part you design. You need
to think assembly for every little part you are designing. You need to ensure
that parts or assemblies can be put together almost by a blind person and do
so correctly.

The tools available are simple: Alignment pins, unequally spaced holes,
alignment tabs or machined features, unidirectional mating or clamping
methods, non-rotatable insertion of sliding elements (d-pin into d-hole), etc.

The same is true of EE design. Don't make a bunch of connectors exactly the
same unless swapping the cables or daughterboards that mate with them is OK.
One common technique is to use different pin count connectors. If a connector
is reversible (such as a pin header used for, say, an RC servo) you need to
make sure that reversing whatever mates to it will not cause damage. A more
sophisticated approach, when it can be justified, is to make smart connectors
that can deal with reversals by actively flipping signals around as required
--common on ethernet switches and routers that can deal with straight and
flipped signal pair cables.

Again, I find it hard to believe that aerospace engineers would make such a
basic error. You never know.

~~~
ajross
> _Engineers working on such projects should not make these kinds of
> mistakes._

Sounds like a plan. In fact, it's inspired me in my own professional life.
Henceforth, I shall never write another software bug again.

The problem with the idea of this kind of thought is that it's turtles all the
way down. You have an assembly mistake that could have been prevented by
engineering. But of course now it's an engineering mistake. And you can't
simply legislate the absence of mistakes. You can try to treat that with
"engineering process", I guess, which mandates the use of funny gadgets or
asymmetry or automated software checkers or whatever. And that probably helps.
But then you can apply that process mistakenly...

At some point you have to cut bait and ship (or launch, rather).

~~~
ghshephard
The idea of DFM is that any reasonably skilled engineer creates a design such
that any reasonably skilled manufacturing/assembly shop can put it together.

Without following the practices of DFM, it is more likely than not that some
part of the assembly will be done incorrectly, _even if_ the assembly team was
highly skilled and executing at a high level of competence.

With DFM, a moderately skilled EE, and a moderately skilled assembly team,
will correctly manufacture the designated system.

~~~
ajross
True, but missing the point. DFM can be misapplied, as it was here. So now you
need to propose a process (design review, I guess) that prevents DFM from
being misapplied. And that process can break down...

The fact that you can look at the wreckage or a rocket and "see" what the
problem was in hindsight is not proof that you know how to prevent any such
accident. I really with more design people understood that.

(It doesn't mean that whatever process you favor is a bad idea either, but
that it has limits. Getting to my original point: you can't fix bugs by
executive fiat.)

------
erenemre
Reminds me of
[https://kerbalspaceprogram.com/](https://kerbalspaceprogram.com/)

------
danso
As I've gotten more experienced, I've found it hard to work with people who
don't optimize for safety and efficency...not because I'm a worrywart, but
because experience had shown that fucking up (royally) is much less likely
caused by lack of skill, industriousness, or moral character, than by
unforeseen weaknesses, stress, and cascading errors. Virtually every ER
surgeon has spent more energy and time into their work than I have in mine,
and every day they deal with life and death, and yet failing to wash their
hands is not a rare occurrence...why should I think I'm any more foolproof?

