
SpaceX loses its third Starship prototype during a cryogenic test - rbanffy
https://arstechnica.com/science/2020/04/spacex-loses-its-third-starship-prototype-during-a-cryogenic-test/
======
dwaltrip
It's funny how extreme the reactions are here. It's either the end of the
world or exactly as planned. Both are nonsense.

To me, this looks like it was definitely a noteworthy fuckup, but in the big
picture it doesn't seem to be a major setback -- nothing like the recent
Boeing issues. That is, as long as a pattern of such events doesn't develop.
SpaceX continues to push the industry forward in interesting and exciting
ways.

To continue the arm-chair quarterbacking... I feel like a moderate decrease in
process aversion would be a good response to this incident. Reduce the chance
of similar future fuckups without slowing down too much.

~~~
gpm
From a process design/management view I'm not sure you're right. In fact I
think you're probably wrong, but there is definitely room for disagreement.

To oversimplify things massively, "how careful do we want to be to avoid
testing failures" is a parameter that SpaceX management gets to control. At
the extreme careful end everything costs billions of dollars because you're
paralyzed analyzing things and double/triple/quadruple/quintuple checking
things. At the not at all careful end of things you keep building "complete
rockets", pointing them at mars without any testing, and blowing them up.
Obviously neither is rational.

They've settled on some parameters that are basically "don't worry that much
about test failures, but keep the failures really cheap". They seem to be
doing pretty well by doing that. They're experimenting with manufacturing
techniques, hiring and training a workforce, building facilities. The
prototypes that they are building keep failing, but that looks to be a
relatively minor cost considering that the current primary goal is (according
to them) to build out the manufacturing processes and make the design easy to
manufacture.

Being slightly more careful would undoubtedly reduce the numbers of prototype
failures, but would it actually be worth the cost of slowing other things
down? Remember that the main cost of this program to SpaceX is engineering
salaries, the faster it goes the cheaper it is.

So of course this test didn't go as planned, of course it would be better if
it had worked, but would it have really been worth it to management to reduce
the probability of this test failing? Like I said, maybe they aren't being
careful enough, but I don't think we have any real evidence for that right now
and I personally doubt it.

~~~
bumby
I think this is where NASA went wrong in many respects. They got so risk
adverse, they built a bureaucracy of checks and oversight that moves slow.
That’s one of thew reasons the industry needs fresh blood like SpaceX

~~~
mcbutterbunz
Two reasons and probably more:

1\. The risk is different when there are human lives on the line.

2\. NASA is publicly funded and if they want to keep receiving funding, they
can’t be blowing up the public’s money.

~~~
ClumsyPilot
I feel NASA's approach is right for manned spaceflight, but is not sensible
for robotic missions - metal is cheap, design time is expensive. I feel the
public appearance is not a hard - nosed calculation, but Is instead a pr
requirement

~~~
sbuttgereit
I actually disagree with you. For one, consider that Starship is ultimately
intended for manned spaceflight... yet SpaceX is still trying things and
blowing stuff up (or imploding or...)

From my armchair perspective I think the risk adversity criticism is
justified. NASA should take much more risk even in human spaceflight... in the
early stages of platform development. First see if you can build the damn
thing and build it repeatedly. Later, after that period, reduce risk and prove
out and refine things to the human spaceflight standard. If you jump
immediately to, "my God people are suppose to fly on that thing!" well, you're
probably going to pay the Russians a lot of money for a fairly long time.

I do agree with you that the political management becomes a real issue.
Politicians and bureaucrats are naturally disposed to a certain cravenness
about such issues once they get to significant positions of power (senior
execs of many companies do, too). But I think a charismatic leader that's out
in front of the messaging could pull it off.

~~~
ClumsyPilot
Well, ironically and counter to your point, Russians had the same basic design
now for nearly 60 years!

------
mabbo
What I love about SpaceX is that they truly take on the start-up approach of
fast iteration, in complete opposite to most of their competitors.

This is a silly loss, as it sounds like it was a test procedure error and not
a valuable experimental failure. But even then, how much is lost? The rocket
was built on-site in a tent, welding some cheap steel together. They've got
more test articles being built. The engine is probably salvageable. This is a
month or two set-back at worst.

And just like when our software fails in a silly way, I'll bet you there will
be new automatic tests, procedures, etc, so that this kind of error can't
happen again. They'll learn, iterate, and continue to move fast (and break
things).

With nearly all of their competitors, any kind of failure like this would be
millions of dollars lost, months or years of setback. This fundamental
difference is why SpaceX is slowly coming to dominate the industry.

~~~
RobLach
Every bit of negative Elon Musk oriented news brings out an apologetic post
excusing failure and painting it positively, with a pile of democratic
support.

This is a failure, and failure is perfectly ok working on something like this,
but what drives people to volunteer their time to spin such things as
positively as possible or even signal boost ? I can’t believe it’s purely
investor driven. It’s almost as if people are finding personal purpose in
sacrificing their time to this cause.

~~~
biomcgary
Why do people cheer for sports teams? Many nerds avoid team sports, but we
share some of the same psychology as the more typical.

Many people like to dream big, but most of the time life is mundane. SpaceX's
efforts are the embodiment of sci-fi fantasy with a chance of becoming real.
Why not cheer?

~~~
scrumbledober
I have for years thought of SpaceX as my sports team to cheer for. I get just
as excited watching their launch livestreams as many get watching football
games.

------
bumby
“this may have been a test configuration mistake”

The article makes it seem like this is a good thing, I.e., there’s nothing
inherently wrong with the design.

If true, it means it may be a procedural escape which can be every bit as
dangerous. Some of the fallout from the CST-100 Boeing Starliner issues seem
to point to process escapes as well. Makes me wonder if in the rush to get
“boots on the moon” by 2024 we may be letting schedule get the best of us...

~~~
gpm
This is an early manufacturing path-finding test article and test, not a test
article and test designed to validate that the system is safe. The procedures,
design, manufacturing techniques, and everything else are still being
developed and certainly haven't been validated yet. Failures are expected.

It is a good thing that there is nothing inherently wrong with the design,
just because that would mean more re-design work for SpaceX. A procedural
issue (possibly this failure), a manufacturing defect (MK1 failure), a failure
due to a temporary hack (SN1 failure), etc are all understandable errors and
easily fixable at this point in time.

Starship isn't really being driven by the "boots on the moon by 2024" push by
the government. Starship is a private venture being developed on their own
schedule (and with their own money).

~~~
londons_explore
I worry that the people who ran this test either didn't know what they were
doing, or were too tired or didn't care.

The top tank was covered in ice, indicating it was nearly full, while the
bottom tank had no ice. That leaves three possibilities:

* The intention is the bottom tank was pressurized with gas:. There is no way that much gas at ~4 bar is safe. It would destroy all nearby buildings if it failed. That's a serious safety failing.

* The intention is the bottom tank was filled with LN2. Whoever was running the test didn't notice the tank didn't turn white, and didn't have any kind of level gauge or flow meter?

* The intention is the tank was unpressurized. The design could never withstand the top tank filled and the bottom tank unpressurized and empty. Whoever designed and ran the test didn't understand the basics of the design.

All of those 3 possible explanations point to more than a simple oversight,
and IMO suggest the team working at 3am might possibly not really understand
what they were doing.

~~~
gpm
> There is no way that much gas at ~4 bar is safe. It would destroy all nearby
> buildings if it failed.

Wat?

This is a launchpad, the safety assumption at play here is that this vehicle
(and in fact a much larger vehicle with this vehicle as a second stage) can be
filled with liquid methane and liquid oxygen, explode (with the energy of a
small nuclear bomb, a few kilotons), and no one will get hurt. The area around
the launchpad is kept clear during testing by local law enforcement for this
very reason.

Hours before the test that failed, in a different test, the same test article
was pressurized with room temperature nitrogen, likely up to ~7 bar (based on
previous tests for which we were given figures).

For comparison, this is the same location that the following test was
performed at, all the additional structures you see have been added recently
by SpaceX as ground support equipment and they are undoubtedly perfectly aware
and ok with the idea that they might be damaged in the case of a failure
[https://www.youtube.com/watch?v=bYb3bfA6_sQ](https://www.youtube.com/watch?v=bYb3bfA6_sQ)

As for the rest of your post, yes, if you assume the people running the tests
were incompetent then they would be incompetent. We can't fully rule out the
idea of course, but it's far from the only option. Most likely the failure was
not nearly so simple as "we forgot to pressurize a tank" \- particularly
considering that the bottom tank was clearly pressurized as it was venting.

The most reasonable suggestion for a proximate cause I've seen, for instance,
is that the pressure release valve on the lower tank iced over and leaked
until the pressure was no longer enough to support the above mass and it
failed. This is supported by the evidence that we know they were having valve
issues related to cryogenic liquids earlier in the day, and that the bottom
tank was inexplicably venting for roughly 13 straight minutes before the
failure. It doesn't quite line up with Elon's suggestion that this was
possibly a test configuration error.

------
Animats
Depends on the lead time for making new ones and the supply of money.

Von Braun's V-2 team built 700 rockets before hitting a target.

~~~
hef19898
And they were the very first ones to actually try. Quite a difference to a
company that can build upon 80 years of experience.

~~~
4gotunameagain
Most of this experience has not been made public though, but yes, it's quite
different

------
elisharobinson
i know that this sounds stupid but failure is par core for spacex. i remember
a time when landing rockets were impossible and then it changed to landing
orbital class rockets is very difficult to being surprised when a rocket
doesnt land successfully. hopefully we will remember these failures and
hopefully spacex comes through again with better designs and sees this project
to success.

------
chirau
I really don't understand how people try to defend Elon Musk and team on
anything and everything they do. It's nice they are pushing the edge, but
please, when things fail, let's acknowledge failure and shortcomings and learn
from them rather than try to spin everything SpaceX or Tesla does as faultless
and forward.

~~~
whatshisface
But failed test aren't failures. That's why you have tests, so that you get
failed tests and not dead astronauts.

~~~
fendy3002
Ha, say that to my previous PM...

------
rexarex
It’s a good thing they’re trying to break it on the ground, PR be damned.
Better they than hiding failures like Boeing

------
trhway
interesting that the visible deformation started at 01:56am and the test
continued until full failure 10 minutes later

[https://youtu.be/kkqgkccWKYI?t=130](https://youtu.be/kkqgkccWKYI?t=130)

While i've happened to be wrong anytime i've questioned Musk's
decisions/approach, looking at that and the previous pressure test failure and
how the body crumples, i wonder whether the way the Starship body is built
isn't faulty by design - like all this steel plates horizontally oriented.

Of course, fast and cheap is the only way to get to build hundreds of
Starship, the Liberty ship of our time, yet the current structural design of
the body seems to require re-work.

~~~
lutorm
All rockets are thin metal cylinders. How do you mean this is faulty by design
when it's fundamentally like any other rocket?

------
RivieraKid
I wonder what their financial situation is. The combination of Starship
failures and C19 would make it very difficult to raise money.

~~~
imglorp
They have a good sized future launch manifest and they have like 60% of the
market now. Maybe not hurting too badly.

[https://www.spacex.com/missions](https://www.spacex.com/missions)

------
erwinh
Is there a good web page somewhere that keeps track of all starship prototypes
currently being built?

~~~
mabbo
Reddit's /r/spacex is probably the most informative place about SpaceX and
what they're doing.

A good example, here's their info on Starship prototypes:
[https://old.reddit.com/r/spacex/wiki/starships](https://old.reddit.com/r/spacex/wiki/starships)

~~~
Robotbeat
The Reddit wiki is super nice; for the rest, I think NASASpaceflight is just a
bit better.
[https://forum.nasaspaceflight.com/index.php?topic=50522.0](https://forum.nasaspaceflight.com/index.php?topic=50522.0)

------
m0zg
Yeah, hardware is unforgiving like that. Made a mistake in calculations? Start
over and spend a few weeks making another widget. There better not be a
mistake in that, otherwise lather, rinse, repeat until you figure out how not
to make mistakes.

------
leesec
I honestly love Elon Musk and much of what he tries to do, but how are these
tests still running? Are rocket scientists and engineers immune to the
coronavirus?

~~~
jccooper
The whole rocket industry is considered relevant to national defense, and thus
"essential", so they don't have to stop if they don't want to. And Elon
doesn't want to.

------
jti107
with the amount of welds this thing has, it wouldnt surprise me to see more of
these failures. every single weld needs to be perfect and not leak

------
rurban
How stupid can arstechnica be?

They didn't loose any Starship, they are still testing tank weldings with
various pressure tests, which had a problem from the beginning. Then they
switched the manufacturer and the tank welding might still not be good enough.
Or as announced it was just a bad test.

Such iterations are totally expected, it's the third now (SN3, serial number
3), but no engine tests yet in integration with the spectacular new Raptor
engine.

------
Priem19
This is SpaceX: the fourth time's the charm.

------
D13Fd
Good thing they have plenty of backups!

~~~
ceejayoz
They've lost two of the three Starship prototypes.

Thankfully, they seem to be fairly easy to build, and the tests aren't burning
through the really expensive bits thus far, the engines.

~~~
de_watcher
You mean two not including this one?

------
eucryphia
Better than in orbit around Io.

------
tzfld
No way they will fly one of these to the Moon in 2022. Will see the Shuttle
effect, ie. introducing engineering workarounds that will greatly increase
costs.

~~~
nabla9
I saw somewhere a spreadsheet that keeps track of Musk's promised and real
timelines.

Musk seems to be off by 1.8 on average.

1.8 is also one Mars year in Earth years. Coincidence?

~~~
tenpies
Curious, does that spreadsheet include everything applicable from
[https://elonmusk.today](https://elonmusk.today) ?

Because I am not sure how you average out against an infinite time frame (E.g.
where are the country-wide summon, snake-charger, car carrier, etc?).

------
bryanlarsen
It appears likely that this was a test procedure failure rather than a
hardware failure. Which brings up the question, why were they doing it at 2AM?
Either they were using night crew who are presumably not their A staff or they
were using highly fatigued day crew.

Edit: everybody is taking this as an attack on night crew. It was poorly
worded -- I believe the latter is more likely, since Elon expects his
employees to work 80-100 hours a week. [https://www.businessinsider.com/elon-
musk-says-80-hours-per-...](https://www.businessinsider.com/elon-musk-
says-80-hours-per-week-needed-change-the-world-2018-11)

~~~
NikolaeVarius
Thank you for insinuating the night crew is not just as capable as the day
crew. It really makes it seem like you have never worked In a 24/7 industry

~~~
eloff
With engineers, that may well be a fair assumption. Night shift is less
desirable, therefore people with more options will be less likely to settle
for it, therefore the better people are in the day shift. Unless this is like
shift work where people rotate between day and night shifts - but that would
be highly unusual for a company like spacex.

Just playing devil's advocate, but the logic chain is plausible.

~~~
have_faith
Perhaps reclusive people have more time and energy to dedicate to their craft
increasing their skill level and propensity to work night shifts? I'm just
spit-ballin'.

Smart people seem to be very good at conceiving a plausible chain of logic for
almost anything. Something Star Trek demonstrated well with the the Vulcans. I
mean that as no snark towards yourself, it's just a subject I've been thinking
about recently.

~~~
eloff
Haha, yeah I can rationalize almost anything and make it sound plausible. It's
a talent of dubious merit.

I think the take-away here is there's no reason to assume that both shifts
would have equal skill - likely there will be some differences, and it's a
question of how much and why. It would be odd if it turned out there was no
difference at all.

------
dcabrejas
This is really bad. I don't know how much money they have left but I cannot
imagine they can continue blowing up prototipes that took weeks to build every
other week like this.

~~~
greedo
These are relatively cheap prototypes. No engines, just the fuel tanks and
body. Cheap stainless steel, fabricated in weeks. If this was Boeing, it'd set
them back years. The whole idea is to crank out a steady stream of cheap
prototypes to iterate design ideas and fixes rapidly.

~~~
papito
At most other companies you have a whole hierarchy of executives who want
their bonuses, so you have that risk aversion where they will spend tremendous
time and resources covering their asses first. Failure is failure, it's not
just a step to more improvement in that world.

