
SpaceX concludes anomaly investigation, sets return to flight target date - dtparr
http://www.spacex.com/news/2016/09/01/anomaly-updates/?
======
hoorayimhelping
_The accident investigation team concluded that one of the three composite
overwrapped pressure vessels (COPVs) inside the second stage liquid oxygen
(LOX) tank failed. Specifically, the failure was likely due to the
accumulation of oxygen between the COPV liner and overwrap in a void or a
buckle in the liner, leading to ignition and the subsequent failure of the
COPV.

The investigation team identified several credible causes for the COPV
failure, all of which involve accumulation of super chilled LOX or SOX in
buckles under the overwrap. The corrective actions address all credible causes
and focus on changes which avoid the conditions that led to these credible
causes. In the short term, this entails changing the COPV configuration to
allow warmer temperature helium to be loaded, as well as returning helium
loading operations to a prior flight proven configuration based on operations
used in over 700 successful COPV loads. In the long term, SpaceX will
implement design changes to the COPVs to prevent buckles altogether, which
will allow for faster loading operations.​

SpaceX is targeting return to flight from Vandenberg's Space Launch Complex 4E
(SLC-4E) with the Iridium NEXT launch on January 8._

Are COPVs a universal thing in rocketry? Or has SpaceX actually loaded fuel
into its rockets over 700 times?

~~~
schiffern
It's true that COPVs are rather common in aerospace (being the lightest
pressure tank technology available).

> Or has SpaceX actually loaded fuel into its rockets over 700 times?

This. SpaceX has done development test cycles, and for each launch performs
stage acceptance tests at McGregor Texas and both "wet dress rehearsals" (now
generally skipped) and static fires at their launch pads. All of these
operations involve loading and unloading helium and LOX.

~~~
astrodust
Given the nature of carbon-fiber composite manufacturing where it's usually
applied in layers, is the risk of a tiny bubble a lot higher when it's
subjected to repeated heating/cooling cycles?

------
the_duke
To save you from the endless bla bla bla:

...concluded that one of the three composite overwrapped pressure vessels
(COPVs) inside the second stage liquid oxygen (LOX) tank failed

...failure was likely due to the accumulation of oxygen between the COPV liner
and overwrap in a void or a buckle in the liner, leading to ignition and the
subsequent failure of the COPV.

.... corrective actions .. short term ... changing the COPV configuration to
allow warmer temperature helium to be loaded ... long term, SpaceX will
implement design changes to the COPVs to prevent buckles altogether

Next launch on January 8th.

------
ChuckMcM
Two things I learned from the report, one was that the helium tanks are
actually _inside_ the oxygen tanks. All the rockets that I can recall seeing
the insides of, had the helium tanks as a sort of 'ring of pearls' around the
end of the tank. Clearly from a volume perspective it doesn't matter if they
are inside or outside, and from a thermal perspective it means the LOX can
keep the helium cool (or vice versa I guess).

And the second is that their wrapping method allows for the creation of voids
between the inner aluminum tank and the carbon fiber wrap. I can certainly
imagine how that could happen with carbon fiber prepreg (it's a prewoven
fabric which maximizes strength by pre-laying out the various ways the fiber
goes through it) but if you were to spool on straight up fiber that would make
it easier to be conformal.

I wonder if other people do the 'tanks within tanks' architecture and if so
how they avoid voids and buckles in their overwrap.

~~~
taneq
Helium has a much lower boiling point (~4K vs ~90K) than oxygen, and so at the
same temperature would have to be at significantly higher pressure to stay
liquid. I'd expect that putting the helium tank inside the oxygen tank would
allow the helium tank to be of lighter construction, since it would only have
to withstand the difference in pressure between the liquid helium and liquid
oxygen. Scaling up the oxygen tank to compensate for lost volume would be
lighter due to the squared/cubed law.

I imagine the reason it's not done this way traditionally is that it'd be a
pain to fabricate.

~~~
kfrzcode
I think this might have something to do with it. SpaceX being SpaceX, they
have a significant advantage in the agility of their fabrication process(es).

------
throwawasiudy
This NASA article on COPV failure is enlightening.
[https://www.nasa.gov/offices/nesc/home/Feature_COPVs_Jan-201...](https://www.nasa.gov/offices/nesc/home/Feature_COPVs_Jan-2012.html)

Turns out COPV's have a tendency to explode semi randomly, with failure modes
that are not fully understood

~~~
manicdee
The failure modes are slightly better understood now, as are the boundaries
between success and failure!

------
brianshaler
It's surprising how much of a disruption this caused to their launch schedule,
but it's great that they got to the bottom of the issue before repeating the
process on any further rockets.

The worst kind of bug is the intermittent one that you can't reproduce or see
in logs. With only 35-55 milliseconds of metrics leading up to the event and
nearly obliterated evidence, it seemed nearly impossible they would be able to
nail down the exact cause.

Critics have pointed out risks in SpaceX's method of loading supercooled fuel.
Accidentally forming solid oxygen around carbon fibers from a buckling
overwrap is an example of this.

There are benefits, but they will need to find and prevent all these possible
edge cases exposed by the new method.

~~~
mikeash
This "anomaly" caused hundreds of millions of dollars in losses, and was an
unprecedented failure mode. It doesn't surprise me at all that it disrupted
the launch schedule for the better part of a year (edit: actually the lesser
part of a year, about four months total if they launch soon). If they manage
to return to flight this month, it strikes me as rather quick.

I wonder if supercooled propellants are really worth it. It seems like a huge
risk in exchange for a small performance increase. Hopefully this will be the
last vehicle loss they suffer from it.

~~~
Retric
My understanding is it effectively reduces launch costs by ~25% by getting
~30% more mass to LEO. Really if it increases risks of failure by less than
10% it's probably worth it for unmanned rockets.

PS: High launch costs set up a cost escalation as satellites are so expencive
they can't fail. Which means launches need a really high success rate. If you
could get 10,000kg to LEO for say 1 million but had a 40% failure rate you
would see a very different approach with much lower overall costs.

~~~
mikeash
With this explosion, there was a ~$200 million satellite destroyed as part of
a ~$60 million launch. We'll probably see much cheaper payloads to go along
with cheaper launches eventually, but for right now the payload loss
dominates, so you have to look at that as well.

For this particular problem, the risk to the payload will be eliminated by not
attaching it for the static fire. Any risk during the actual launch would
still affect the payload, though.

~~~
ricardobeat
Payloads are insured, most likely SpaceX itself takes most of the loss.

~~~
mikeash
Insurance isn't magic money. The cost of the satellite is recouped through
premiums charged to companies launching satellites. It becomes part of the
cost of launching one way or another. If gain X has cost Y, Y still factors
into the tradeoff even if it's insured.

To put it in concrete terms, if insurers decide that SpaceX is significantly
more dangerous to its payloads, then launch insurance for satellites using
SpaceX rockets will become significantly more expensive.

~~~
ricardobeat
Not sure what you're getting at with the condescending tone. My point is that
the satellite owner did not realize a $200m loss; SpaceX being at least
partially self-insured, from what I've read, is more likely to take a hit.
There were 85 launches in 2016, one rocket blowing up is not going to
significant change the landscape (heh).

Disclaimer: I don't know shit about the space industry, so don't waste your
time.

~~~
mikeash
We're discussing the tradeoffs involved in making launches cheaper but more
risky. My point here is that insurance is irrelevant to that tradeoff. One way
or another, the payload owners are ultimately paying for that risk.

------
dmix
Ouch it looks like the Isreali company lost their previous satellite too after
4 yrs of being in space:

[http://www.haaretz.com/israel-
news/business/1.687543](http://www.haaretz.com/israel-news/business/1.687543)

There is still no cause listed on Wikipedia.

> Industry experts described the total loss of contact with the satellite as a
> highly uncommon event.

This one was built by Russians for the first time. It looks like they went
back to a domestic built one with SpaceX and it blew up. That's some bad luck
:/

[https://en.wikipedia.org/wiki/Amos_(satellite)](https://en.wikipedia.org/wiki/Amos_\(satellite\))

~~~
dispose13432
That's why insurance isn't the end-all/be-all.

They've got enough money out of insurance to build themselves a new satellite,
the problem is that they need that satellite _yesterday_ , not today.

On the other hand, SpaceX isn't living off poor blokes who were hoodwinked by
a starry-eyed agent. Their customers are professionals with lawyers and
accountants taking care of things, so I don't feel that SpaceX is particularly
unethical.

------
ashnyc
Does anyone know if the public can watch when they launch at Vandenberg AFB
Space Launch Complex. I have always wanted to do this but never sure if it is
open to the public

~~~
wolf550e
Kinda. Read the details at r/spacex FAQ and the links there
[https://www.reddit.com/r/spacex/wiki/faq/watching#wiki_i.27m...](https://www.reddit.com/r/spacex/wiki/faq/watching#wiki_i.27m_at_vandenberg._where.27s_the_best_place_to_watch_the_launch.3F)

------
amorphid
What is the primary purpose of using liquid helium in a Falcon 9?

A quick google search for `liquid helium rocket` introduced me to pressure-fed
engines [1]. A pressure-fed engine uses liquid helium, and is a replacement
for a turbopumps. Since the Falcon 9 uses turbopumps, I don't understand what
the liquid helium is for.

If I had to guess, it sounds like a Falcon 9 has a pressure-fed engine AND a
turbopump. On a car, that'd be kind of like using both a turbocharger and a
supercharger, which sounds complicated.

[1][https://en.wikipedia.org/wiki/Pressure-
fed_engine](https://en.wikipedia.org/wiki/Pressure-fed_engine)

~~~
motoboi
Not only is pressurized gas used to avoid the tanks from crumpling from
negative pressure, they cannot sustain themselves without internal pressure.

This gas need to be as inert as possible, hence helium. Another noble gas
could in theory be used, but I think price is the reason.

~~~
InclinedPlane
Gases at the same pressure and temperature have the same molecular density, so
lowering molecular weight translates directly into lower total weight of
pressurant gas needed.

Alternately, if the propellants allow, it's possible to use those in gaseous
form for pressurization (by passing some of it through a heat exchanger). This
increases the weight of pressurant gas wasted but it saves the weight of the
Helium system, and the complexity, so ultimately it's a big win.
Unfortunately, this doesn't work with kerosene so the Falcon 9 needs a Helium
system.

~~~
kijin
Could they use oxygen gas for pressurization, since the helium tank in a
Falcon 9 seems to be inside the liquid oxygen tank? Or would it be too
dangerous?

~~~
snrplfth
You _can_ use oxygen gas for pressurization of oxygen tanks, but it introduces
a new set of problems, since you've got to get a stable pressure of oxygen gas
inside a container with a huge volume of sub-cooled liquid oxygen, and you
have to have a high-pressure tank in there anyways to pre-load pressurized
oxygen gas, so that you can get it into the tank fast enough. Also it means
you've got reactive oxygen gas all around, so you have to redesign things for
that.

However, this kind of oxygen-gas pressurization is exactly what's planned for
SpaceX's next-generation oxygen-methane rockets.

~~~
InclinedPlane
The Space Shuttle used autogenous pressurization for its main engines /
external tank, and the Delta IV is eventually planned to do so as well. And as
you mentioned SpaceX plans to use the same system with their next generation
of rockets. It is tricky to get working correctly and requires a larger up
front engineering effort, but the payoff in simplicity of operations is
significant. Additionally, it provides gaseous propellants that can be used
for other purposes, such as attitude control thrusters.

~~~
snrplfth
Yes, it's great if you can manage it. But helium is so pleasingly inert and
cold, I can see why it's not always the choice.

------
dispose13432
My question is:

1\. How often does this happen?

2\. Hoe many tests did SpaceX do _before_ the first FT launch?

Considering that most rockets don't use supercooled fuels, I would expect
insane tests (they can't rely on NASA's historical data).

~~~
ben-schaaf
All the well known rockets use cryogenic fuel at least in their ascent stages:
Space Shuttle, Saturn V, Soyuz. They certainly can't rely solely on NASA's or
Russia's data, but it will/has undoutably helped.

~~~
xorxornop
Cryogenic fuel is not the same is supercooled; supercooled is closer to
_freezing_ than to _boiling_

------
GizaDog
I've been trying to dig up the launch time. Does anyone know when that will
be? Thanks.

~~~
mabbo
[https://www.reddit.com/r/spacex](https://www.reddit.com/r/spacex) usually has
the most up to date information. They are saying 10:28/18:28 (PST/GMT).

------
nrjdhsbsid
Does anyone else think the timing is kinda funny? Today, at least in the US,
is probably the slowest news day of the year. Great time to unload anything
unfavorable :)

~~~
arijun
Wouldn't you want to unload something unfavorable on a fast news day, where it
would get buried by other, bigger news?

------
CydeWeys
That's multiple complete losses now that have been caused by issues with
COPVs. I remember reading some discussion on /r/spacex about SpaceX going with
really aggressive COPV technologies to shave as much weight as possible, but
they're definitely paying for it in reliability. Hopefully they can iron it
out, and maybe they'll switch to less aggressive COPV designs soon?

~~~
snrplfth
The previous loss was, as far as they can tell, related to a strut that broke
under strain and released an otherwise-fine COPV into the oxygen tank. As far
as they can tell it didn't have anything to do with aggressive design, just a
part failing under its rated strength.

~~~
JamesUtah07
I don't think it even failed under it's rated strength, it was that their
supplier for the struts provided sub-par/faulty struts and they couldn't
handle the pressure they were supposed to be rated for

~~~
snrplfth
That's what I meant. It was rated for a certain load, but there was a faulty
strut in the batch, and it failed well under that rated load.

------
dnautics
My suspicion was that SpaceX was rushing to validate their carbon fiber tank
plans, which is why they did it on stage 2 (process is lower scale and easier
to experiment). Well, at least they failed fast. I made this prediction in a
private twitter DM (yes, weaksauce, I should have been bolder with a public
post) the day that Musk announced their plans for giant carbon fiber tanks,
with no knowledge that the stage 2 tank was made of carbon fiber.

[https://drive.google.com/open?id=0B3uVSPZl-I22UjE4aElRcDFaaE...](https://drive.google.com/open?id=0B3uVSPZl-I22UjE4aElRcDFaaEk)

~~~
snrplfth
The Carbon-Overwrapped Pressure Vessels are quite different from the large-
scale carbon fiber tanks they're building now. The new large tanks are
designed to be able to contain large quantities of propellant at moderate
pressures, possibly without a liner, while the COPVs are meant to contain
pressurant gas at extremely high pressure, with a metal liner. SpaceX has been
using COPVs to hold helium pressurant since the first Falcon 9 back in 2010.
This is not a new technology. What is new is the large propellant tanks, which
have not been done before at this scale, and have not been used in Falcon 9s
at any point.

So no, you shouldn't have been bolder with a public post.

------
hackuser
Could we link to something less intrinsically biased, such as a news report,
and not SpaceX's press release? Here's one:

[http://www.nytimes.com/2017/01/02/science/spacex-launch-
rock...](http://www.nytimes.com/2017/01/02/science/spacex-launch-rockets-
explosion.html)

~~~
apendleton
The release is SpaceX reporting on an investigation they themselves conducted.
There probably aren't other sources besides SpaceX that are better equipped to
report on an investigation conducted by SpaceX or to independently corroborate
or dispute its findings; all we can do is take their word for it, biased or
not. Indeed, that NYT article is mostly just a regurgitation of the release.

~~~
hackuser
> There probably aren't other sources besides SpaceX that are better equipped
> to report on an investigation conducted by SpaceX

I disagree; SpaceX has a clear interest (and a well-established track record)
of presenting things in a way that bolsters their image. Outside experts could
provide critiques of the report: What represents bad luck, what is trivial,
and what is a serious error; where is SpaceX being straightforward and where
are they spinning things; where are the gaps, where are there alternative
analyses of the same data, and what seems solid.

That is a big reason why journalism is important: It provides context and
other voices.

~~~
andars
That's all fine and good, but the only source NYTimes article you suggested is
SpaceX, so you're not getting commentary from any outside experts. Instead,
you are just getting quotes and paraphrases from this and previous SpaceX
press releases.

~~~
hackuser
It's just one article of many possibilities. Also, they still can
contextualize info and filter out spin.

Incidentally, any time you post something that isn't fully saturated with the
SpaceX Kool-Aid, you get a strong reaction. Look at all the comments and the
downvoting over a pretty mundane detail and a common concern: Don't link to a
press release. I didn't even criticize the Great and Powerful Wizard or his
rocket ships.

To be clear, I don't care about the votes, but I think the overall response
says more than the comments.

~~~
jasode
_> downvoting over a pretty mundane detail and a common concern: Don't link to
a press release._

I didn't downvote your comments but I can see why you got them: you provided
an NYT article that wasted readers' time.

Yes, it's wise to be skeptical of "press releases" to avoid spin but your
rigid adherence to _avoid-press-releases-at-all-costs_ put you on autopilot
and caused you to link an article that was _worse_ than the press release.

Now, you think the downvotes are about disagreeing with your principles about
press releases when it may just be downvotes about that _specific NYT article_
that added no value.

Sure, when Apple makes a press release, readers would be better off skipping
their corporate marketing fluff and look at the Ars Technica writeup that will
be more critical. However, the NYT writeup of SpaceX's accident report isn't
an analogous example value-added journalism. (The newswriter Kenneth Chang is
not a rocket scientist and can't offer any expert counterweight to the press
release.)

~~~
hackuser
This comment attributes many things to my thinking that are the creation of
the commenter, some of which are contradicted by what I wrote myself, such as:

> you think the downvotes are about disagreeing with your principles about
> press releases

------
cnnsucks
>> This "anomaly" caused hundreds of millions...

It also produced a large quantity of nonsense. WAPO kicked that off with
speculation of "sabotage." [1] By the time that stuff had finished ricocheting
around the echo chamber it was SPETSNAZ snipers. Today WAPO has finally
grounded out it's "Russian's hacked the US power grid" fiction. [2]

[1]
[https://news.ycombinator.com/item?id=12617990](https://news.ycombinator.com/item?id=12617990)
[2]
[https://yro.slashdot.org/story/16/12/31/1533245/washington-p...](https://yro.slashdot.org/story/16/12/31/1533245/washington-
post-retracts-story-about-russian-hackers-penetrating-us-electricity-grid)

It would be nice if Elon could take a few minutes and install some grown-up
editors at his paper. It's turning into a tabloid.

* Oh dear, I've mixed up my celebrity billionaires... Yes, Elon would have to get Bezos to fix the paper.

~~~
aristus
Are you perhaps confusing Jeff Bezos with Elon Musk?

[https://en.wikipedia.org/wiki/The_Washington_Post#Jeff_Bezos...](https://en.wikipedia.org/wiki/The_Washington_Post#Jeff_Bezos_era_.282013.E2.80.93present.29)

~~~
duskwuff
Mind that Bezos also has his own aerospace company -- Blue Origin -- so it'd
in fact be _in his interest_ to report negatively about SpaceX. (If you
subscribe to the theory that he's personally interfering with WaPo's editorial
choices, which I do not.)

