It's a whole nuther ballgame for commercial airliners, for example. They undergo hundreds of test flights and all manner of subsystem tests before getting certification.
Of course rockets are single use. Most rocket designs will never fly even half the number of test program flights a commercial airliner has to undergo. The cost would be (hohoho) astronomical.
A lot of the reliability estimates of spacecraft are really high speculative, and that's a very polite euphemism. I know personally a few people at Nasa involved in human spaceflight, and they say, in unguarded moments at the end of a long day, things like 'well... what the hell does 'man rated' actually mean anyway? does anyone know?'. No one does. Obviously to say 'it will be 99.9% reliable' is very odd - who is going to pay for the several thousand tests required to make a statement like that with any confidence? So yes, take all talk of safety, especially quantitatively, with a pinch of salt.
Interestingly the Skylon (reusable) Spaceplane is down to be qualified to the same standards as a commercial airliner. That might yield some data.
Actually, the largest cost is building a whole new structure. With a fully reusable rocket (as SpaceX is going to use), the only costs are fuel and maintenance, which is more expensive than an airplane (~$50k per flight, IIRC), but negligible compared to the initial cost of designing and building the structure.
Could you imagine if airplanes were single use? There would no way Southwest could achieve a 45-minute turnaround time
And reliability numbers for man rated parts don't come out of nowhere. They come from extensive simulations and tests, which are then extrapolated. It's not the same guarantee as running thousands of end-to-end missions, but it's better than you imply.
> With a fully reusable rocket (as SpaceX is going to use)
This is the sort of thing where you need to be careful with what you infer, the the sort of thing that often causes engineers to be overconfident in performance and reliability estimates.In reality, they're doing some very initial experiments in vertical landing with a view towards exploring reusability. That is different to your implication that reusability is a done deal.
> They come from extensive simulations and tests, which are then extrapolated. It's not the same guarantee as running thousands of end-to-end missions, but it's better than you imply.
And that's the problem. Notice you're talking about man rated 'parts' and I'm very deliberately not. Many of the mission failures or anomalies in launch vehicles so far have come from parts that work fine on the bench as individual subsystems. It's the lack of full-scale, realistic tests of complete systems that cause problems. There's just not the money for it nowadays. For example, Orion's crew vehicle had budgeted 2 aeroplane parachute drop tests. Apollo's landing module had over 230. Interestingly, they recorded anomalies on over 210 of those.
As for simulations, well one of the catch-phrases in the rocket engine business is 'plumbing never leaks in simulations'.
As for extrapolation, as a datapoint related to a field I have worked in (parachutes for space systems), quite a few of the high profile parachute failures were colloquially summarised as 'they extrapolated without a license'. All the Mars landers the USA have landed so far have used disc-gap-band parachutes of the same design and size that were explored in a set of very expensive and extensive tests performed at high altitudes for the 70s Viking Lander. It's called the 'viking box' and people at JPL know you do not just 'extrapolate' out of it because they've seen what happens when smart, well intentioned engineers do. That's why they called it a box :)
Going back to simulation for a moment, I am familiar with the state of the art of parachute simulation (and fluid-structure interaction simulation in general), and so are they people in charge of the space missions, and that's why they stick to the Viking box. We can barely match that viking data in sims, let alone start wondering out of it into unexplored territory.
Finally a little anecdote from Charles 'Chuck' Lowry, the guy who designed the apollo landing systems, about testing. On Apollo 15 reentry, one of the 3 parachutes failed, the first and only recorded failure of an apollo chute during operations. It was traced back to being because the landing module thrusters had vented their fuel out before landing, but this had ignited on the still hot nozzles on the way out, causing a load of burning fuel to go fly up into the chute and destroy it. Thank god, he said, that it only caught the one and not a second one, else it could have ended very badly.
The parachute system tested perfectly, and the thrusters performed admirably during their entire qualification program and all previous flights. But the combination of these two systems, under real conditions, interacted in such that the consequences were a significant risk to life. 'You ain't tested it till you've tested it', he said.
HN is full of similar examples of outages of things like AWS due to an interaction of failures of parts, systems, and bob the technician not putting the circuit breaks back in exactly the right place after routine maintenance. The space of possible failures rises exponentially with the number of parts, when you consider all the ways they can interact. It's a hard problem to solve and the people at the top are under no illusions about the reliability numbers, they're made for congress and journalists.
It's the lack of full-scale, realistic tests of complete systems that cause problems. There's just not the money for it nowadays. For example, Orion's crew vehicle had budgeted 2 aeroplane parachute drop tests. Apollo's landing module had over 230. Interestingly, they recorded anomalies on over 210 of those.
I would argue that modern designs that learn from the mistakes of past designs require less testing. Maybe it takes 230 drops to understand the aerodynamics of a falling capsule, but once the knowledge is obtained, it only takes 2 to verify a new capsule works as good as the old one.
As for simulations, well one of the catch-phrases in the rocket engine business is 'plumbing never leaks in simulations'.
So why not simulate leaky plumbing? Computer modeling has come a very long way over time.
'You ain't tested it till you've tested it', he said.... The space of possible failures rises exponentially with the number of parts, when you consider all the ways they can interact.
True, but each new failure requires a series of events more complex than previous failures. You're using a bunch of examples of old failures to imply that new designs will fail in the same way, when the reality is that new designs have the benefit of learning from every single previous failure, and every subsequent failure further increases the reliability of the system.
I'm reminded of Asimov's essay, The Relativity of Wrong[0]. Despite your experience in the field, it seems you're too eager to assume that every new idea can be just as wrong as the previous one. Sure, new space vehicle designs like those from SpaceX may fail in ways we couldn't predict, but that is completely different from saying that they'll be less safe or less reliable than their predecessors, or that they have to fail in all the same ways as their predecessors first in order to prove their success.
First of all, what do you think you're arguing against? The argument isn't that SpaceX can't do better than existing designs. It isn't that they aren't trying. It isn't that they aren't going around this the right way. It is simply that we do not have data right now to have any real idea how well they actually will do.
Do you have anything to counteract that?
As for the rest, do you have any idea how ridiculous your position is? Based on a misinterpretation of a popular essay about basic science, you conclude that experimental data is less important now than previously. And you're doing this when arguing with an expert in parachute design who is well aware of the current limits of simulations, and several examples of what has happened when actual engineers tried to extrapolate from past designs and models to predict what would happen with a future design.
Furthermore you're doing this with willful ignorance of the fact that every area of technology where people actually achieve high reliability, it is done by people who place a lot of emphasis on actual data from experiment. Simulations are a supplement, not a replacement for that.
Finally your claim, new designs have the benefit of learning from every single previous failure is plain wrong. Anyone who studies this stuff will tell you that people keep making the same types of boneheaded mistakes over and over again. And, people being people, it is hard for us to recognize when we've made that particular type of error again. Therefore we create procedures to automatically catch errors that our organization has proven to have a tendency to make. Those procedures need to include live tests. Furthermore our expectation should be that we will continue to screw up in similar ways to what we have done before, and not that we've learned from the past and now only make more exotic errors.
All of that said, let me repeat. The people working at SpaceX absolutely know this. They seem to be on course to potentially do better than has been done in the past. But until they accumulate an accident record, we won't know how well they've done. (And at this point their designs are in sufficient flux that it will be years before we really can establish a good baseline.)
First of all, what do you think you're arguing against? The argument isn't that SpaceX can't do better than existing designs. It isn't that they aren't trying. It isn't that they aren't going around this the right way. It is simply that we do not have data right now to have any real idea how well they actually will do.
It seems we were using a different vocabulary and/or arguing along orthogonal axes. My initial impression of your and ballooney's comments was one of excessive pessimism, presumably to temper what you perceived as excessive optimism.
I'm just an interested layman trying to keep people from giving up on the idea of eventually sending people to Mars, because darn it, I really want to go ;), and I'm willing to accept "extensively simulated and unit tested with a few successful integration/flight tests" as good enough for me.
> So why not simulate leaky plumbing? Computer modeling has come a very long way over time.
You're not even close! There is no value in nebulous hand-waving statements about 'a long way'- what does that even mean? Did you not read the rest of my message which had very specific examples of how simulation isn't there yet?
And as for simulating combustion (when fuel leaks onto a hot pipe, say), academia are only just scratching the surface of simulating things like combustion instability in very toy problems, where they deliberately induce some perturbation x on a flow y and sample they system at some frequency that will just about tell them if there's a limit cycle going on. This is still so far away from actually being able to simulate a burning rocket engine properly.
Now of course I must encourage you to stop being so literal. It's not like rocket engineers say 'oh no you can't do simulation with rocket engines, because you can't model leaks properly'. That's preposterous and you're the first person I've ever come across who has inferred it so. What it means is that real actual hardware is very much not like a computer program where you can test something against all inputs and be deterministic about how it will respond because a computer is a comparatively simple, discrete thing. It's a vastly different problem to simulate a rocket engine. This saying speaks to the fact that you can't simulate every paramater of something like a rocket engine - it's just not computationally feasible, and there are plenty of people working on these problems who are familiar with the state of the art of estimation techniques too. It's hard regardless. Oh what I wouldn't give for a real world version of Haskell's QuickCheck!
Here's the thing about simulation. It's not, as you might imagine it is, a little local copy of the universe in your pc where you just arrange all the bits at t0 and say 'ok go!' and come back and see that it's worked so your design is fine. Instead you the engineer make the rules and propagate the system through your rules for a bit. If you haven't thought of a scenario, it's unlikely that your simulation will be able to show it. There are not a whole bunch of hidden states.
Now you can do universe-in-your-pc type simulation which produces very realistic looking results, but to simulate something as complicated as a rocket in flight would probably take longer than the age of the universe per second. And there are still lots of assumptions there.
> You're using a bunch of examples of old failures to imply that new designs will fail in the same way, when the reality is that new designs have the benefit of learning from every single previous failure,
This is the sweetest and most endearingly optimistic thing I've read all day. I imagine you ride into work on a unicorn. There's some validity to what you're saying, of course people say 'ok won't try doing it that way' but like with your simulation comment, I think you just don't know the reality of how these things actually work in practice.
I promise you there are still lots of Fuel Slosh Failures (an interaction of the control law and the fluid dynamics of the fuel tank causing the failure of Falcon 1 2nd flight) out there in the wild that you don't pick up till you actually fly the damn thing, despite I'm sure very thorough simulated control systems in computers on the ground by people who have a deep understanding of control theory. Static test fires never picked up the pogo effect ( http://en.wikipedia.org/wiki/Pogo_oscillation ) which blew up a few rockets and caused engine shutdowns in others. None of these are witchcraft, in that engineers perfectly understand them once they've seen them. My point is that you can't come up with them and swat them in advance in every case because there are just too many possible ways things can perniciously interact to cause you problems.
Likewise, 'every single previous failure' is not that many in rocketry, because there have not been that many rockets. It's not like rocket X blew up because it's failure mode Y was The Failure Mode for rocket X. Rocket X probably took with it to the grave several other possible failure modes, it was just failure mode Y that got there first.
The shuttle flew hundreds of time before that bit of foam broke off and put a hole in the wing's leading edge. We might never know that that particular bit of foam was a Loss of Life waiting to happen if something else had blown up the shuttle sooner. I went to the talk, one of the best talks I've seen in the whole of my career in engineering, by one of the lead investigators of the Columbia disaster. He handed around 2 identical bits of foam, about an inch diameter and 2 inches long. They looked a bit like that dense styrofoam you use in roof insulation. Anyway, these were the insulation foam on the shuttle fuel tank. He then showed us a 200,000fps video of their pneumatic cannon firing these samples at a bit of carbon carbon composite of the sort used in the leading edge of the shuttle wing. The first sample collided and then disintegrated into a cloud of dust, leaving the wing edge unharmed. The second sample collided with the wind edge and punched a huge whole straight through. The whole audience gasped. It turns out the 2nd type of foam was 'trivially' different in some small way, that no one thought would be an issue at design time, but that was the composition of the bit of the foam that broke off and put a hole in the shuttle.
Now, that whole audience was an audience of engineers, and we were all shocked. We all knew that if we were asked to simulate it, we'd say 'well, it's a homogenous foam. this kind of density. this kind of young's modulus. this kind of poisson ratio. this kind of hardness. ok that'll do' and simulated with it. But these 2 kinds of foam were basically identical in all these respects, yet their behaviour was vastly and tragically different. Unless you simulate down to the sort of molecular level, simulations just don't show you this stuff.
So I understand as an outsider [I am making an assumption that you are from your understanding, apologies if you are not] why you might think that 'surely' simulation 'should' be able to be good enough 'nowadays' what with Moore's Law and MCMC and so on. But really honestly no, not to the point you're going to catch the kind of outliers that cause problems.
P.S. I'm not arguing either way on whether or not SpaceX will be safer or not than some marker. I'm sure they'll have among the safest launch vehicles ever flown. But my points so far have been 1) Beware people putting numbers or otherwise strong claims on reliability and 2) I've been trying to kick the tyres of the mental tools and reasoning people from a software background bring to bear when trying to understand things like space hardware. There is a lot more to it.
> P.S. I'm not arguing either way on whether or not SpaceX will be safer or not than some marker. I'm sure they'll have among the safest launch vehicles ever flown. But my points so far have been 1) Beware people putting numbers or otherwise strong claims on reliability and 2) I've been trying to kick the tyres of the mental tools and reasoning people from a software background bring to bear when trying to understand things like space hardware. There is a lot more to it.
Now this I definitely agree with. I may have misunderstood your initial comment, but I was only trying to argue generally against the idea that Elon Musk is somehow out of line for saying that F9/Dragon will be the "safest, most advanced crew vehicle ever flown" because what else do you expect him to say? "We'll make it, uh, as safe as we can, I guess. Safety's a difficult concept, and rocket science is hard, y'know?"
You and I both have experience in space hardware (my guess is that you have more experience than I), so we both know how rediculous it is to put a number on things, but in this thread, we were arguing the same position from different directions. You against the sentiment of "of course it will be safe" and me against "they can't possibly know how safe it will be." Does that make sense?
> your implication that reusability is a done deal.
The plan is full reusability, except for a few parts that are lost as each stage separates from the rest, and fuel. There is no plan B. http://www.transterrestrial.com/?p=27574
And re: testing, I misspoke. By "part" I didn't mean component, I meant to say "system" but had in mind testing each part of the mission and contingency plan (in addition to end-to-end tests). I, and the engineers at SpaceX, are aware that most failures come from interactions between systems. And by "extrapolate", I didn't mean extrapolate from one part to the entire system, but from N tests to the N+1th test (because when you test as you fly and fly as you test, the actual mission is just another test).
Also, you conflate 'safety' with 'complete system success.' Apollo 15 had a safe, successful reentry, despite the single parachute failure, because there was designed-in redundancy. You can expect similar from SpaceX.
But you are right in that assigning any sort of reliability number is rediculous. On my project, we have analyzed every possible failure scenario we can think of, and come up with contingency plans on top of contingency plans, until we get to the point where so many things would have to be wrong in order for our plan to be used that it's not worth the effort, and we would think on our feet at that point. But still, nobody has bothered putting out a percentage chance of mission success. And nobody has asked the launch vehicle for a percentage chance of correctly inserting us into our desired orbit.
Actually one of the design goals for SpaceX is to develop reusable rockets. According to Elon in an ideal world, this would reduce the cost of a launch by something like a factor of 100. Elon claims that a realistic target is to reduce it by a factor of 10.
Of course as soon as you design a rocket to be reusable, the task of making sure it is still safe after a dozen or a hundred launches becomes much harder.
Of course rockets are single use. Most rocket designs will never fly even half the number of test program flights a commercial airliner has to undergo. The cost would be (hohoho) astronomical.
A lot of the reliability estimates of spacecraft are really high speculative, and that's a very polite euphemism. I know personally a few people at Nasa involved in human spaceflight, and they say, in unguarded moments at the end of a long day, things like 'well... what the hell does 'man rated' actually mean anyway? does anyone know?'. No one does. Obviously to say 'it will be 99.9% reliable' is very odd - who is going to pay for the several thousand tests required to make a statement like that with any confidence? So yes, take all talk of safety, especially quantitatively, with a pinch of salt.
Interestingly the Skylon (reusable) Spaceplane is down to be qualified to the same standards as a commercial airliner. That might yield some data.