
Boeing Starliner's flight's flaws show “fundamental problem,” NASA says - bookofjoe
https://www.nytimes.com/2020/02/07/science/boeing-starliner-nasa.html
======
mantap
The incredible thing about this is that when NASA gave the contract to both
Boeing and SpaceX, SpaceX was supposed to be the risky new startup and Boeing
was supposed to be the safe "nobody ever got fired for choosing IBM" backup
option. How times have changed.

~~~
Nasrudith
Really that pattern (technically antipattern) shows why things degraded the
way they are. If the "default option" is taken as granted there is no
incentive for it to stay adequate let alone improve and advance like it should
and a "profiteer" temptation to monetize the fixed demand grows until a
breaking point is reached.

Which is why in any smart organization should ironically fire someone for
"choosing IBM because nobody got fired for it". Not because it is the standard
but because the butt covering over actual quality is a self fufilling prophecy
of trouble. Essentially choosing IBM in itself is fine so long as you do at
least a surface evaluation to be sure it is actually a better choice instead
of cover which subverts the actual root function.

~~~
emn13
Also why anti-monopoly laws need to be reevaluated, and improved to ensure
nothing counts as a market with fewer than 10 practical competitors at any
given moment, and probably more like 100 for real competition (or some similar
set of thresholds to ensure flexibility).

We've sacrificed meaningful constant progress at the altar of relatively
speaking small economies of scale. That's a win... for the first few years;
and a loss for the hundreds thereafter.

There's nothing necessarily wrong with a segment where there's essentially one
one or two providers; some things may be natural monopolies. But if so - those
suppliers are essentially governments of their fiefdoms, and deserve similar
restrictions, including most particularly, no incentive-distorting profit
motive.

~~~
magduf
You can't have 100 aerospace companies doing big projects; the economy just
isn't that large.

For very mature industries, I believe studies have found that the optimal
number of competitors is usually 3 large companies. For immature industries
with a lot of change and innovation, that number is generally higher, but as
the industry matures, players either die off or merge. A good example of this
is the American auto industry.

~~~
wonderwonder
Generally I would say this is a good measure for companies that serve
individuals. Person A shops around and buys a car that most meets their
needs/budget. This goes out the window once you enter the world of government
contracts, especially large scale ones. You currently have a few large
companies; Boeing, Lockheed, Northrop, etc. that land all the major contracts.
The landing of these contracts though is for the most part based on government
lobbying, the assurance of a nice private sector job post government
retirement and a large helping of 'this is the way we have always done it'.
This has recently led to debacle after debacle where the end user (the US
taxpayer) is left holding the bag for massive global corporations that
continue to profit. When there is an incident large enough to cause people to
sit up and take notice, the ceo of the company gets a massive golden parachute
and the company says 'see we took care of it. wink wink'. Look at the F-35,
Littoral Battleship, and the subject of this article.

If the government started bidding out smaller real hardware contracts to new
companies, we would see a rise in new competitors and an accompanying rise in
quality and roi. To do that though lobbying would have to be severely
curtailed. Probably not going to happen.

~~~
magduf
Sorry to reply twice, but I think there's more here to address.

First, the thing about cars doesn't work even for individuals, because it's
not comparable. Buying an all-new spacecraft or aircraft isn't anything like
buying a car. Cars are engineered and designed for a market, and manufactured
in quantities of tens to hundreds of thousands or more. This is more like
getting an extremely custom house built for yourself (think something like a
Frank Lloyd Wright house, not a subdivision house): you have to find someone
who can architect it, and someone to build it. Neither of these is easy
because the house is entirely unique and not even like other houses. It's not
something you can just go pay someone to do and not think much about, like you
can with a car: you have to be actively involved in the design and
construction process.

>If the government started bidding out smaller real hardware contracts to new
companies

What companies have these abilities? Would you have a super-custom mansion
designed and built by some random guy you met on a street corner? Of course
not. You have to find someone who can actually do the work, and you have to be
careful because a bunch of people will _claim_ they can do it, but they really
can't, so you can't accept their bids, and that means you have to do a lot of
work in verifying they're capable of delivering.

Building a new spacecraft or airplane isn't trivial, and isn't something some
small company can just do, like you seem to believe. How is a new company
going to start up that can actually do these things? The only reason SpaceX
worked out is because a billionaire started it as a hobby project, and even
there, it's been at it for well over a decade now. Same goes for Blue Origin:
it wouldn't exist if it weren't for some billionaire who apparently had
nothing better to do with his time. Even going back to the old days, this is
what happened with TWA and Hughes aviation: Howard Hughes was a billionaire
who inherited his fortune and got bored with oil drilling bits. You can't just
award a hardware contract to some tiny company that has no ability to deliver
on the contract in a reasonable timeframe.

------
hwbehrens
> _Because of the clock problem, Boeing engineers started searching to see if
> there were other flaws in the software. On the evening before landing, they
> found one._

I feel like, maybe, some baseline QA would be a good idea for a multi-million
dollar rocket. If they were able to find this bug in the middle of the
mission, then presumably with some more robust testing tools, it could have
been caught in a relatively short time?

I wonder what the possible justification would be for skipping testing. As a
cost-saving measure for getting an MVP or website change pushed out the door,
I understand -- as the foundational control software for a difficult-to-
replace, very costly hardware artifact, it seems like the very definition of
penny-wise, pound-foolish.

~~~
bumby
There are plenty of QA-related requirements [1], [2]. To a certain extent, the
incentives (cost, schedule) may align with contractors trying to skirt as many
requirements as possible as long as NASA doesn't catch them:

> _Mr. Loverro acknowledged that NASA failed to identify the weaknesses in
> Boeing’s work. “Our NASA oversight was insufficient,”_

[1] [https://snebulos.mit.edu/projects/reference/NASA-
Generic/NAS...](https://snebulos.mit.edu/projects/reference/NASA-Generic/NASA-
STD-8739-8.pdf)

[2] [https://snebulos.mit.edu/projects/reference/NASA-
Generic/NAS...](https://snebulos.mit.edu/projects/reference/NASA-Generic/NASA-
STD-8719-13C.pdf)

------
rbanffy
It sure doesn't look good to have two software issues very late in the product
development cycle, on a live test. While the first one would have been averted
by a human crew, the second one would not be so benign.

Software is hard. Embedded software doubly so, but... Come on... This should
not be happening to Boeing.

~~~
throwawayjava
_> Software is hard_

Traditional engineering companies don't respect software. You can tell by the
fact that they pay well below market rates.

~~~
jillesvangurp
I think maybe the difference here is the fact that SpaceX probably treats
software as a first class citizen; just like Tesla does, and is likely to have
a modern view on how to build and evolve software. Additionally, software is a
key component for them that is very much determining the abilities of the
hardware. Tesla even ships pure software features that they charge money for.
That's unheard off in the car market. And I would expect, Elon Musk to drive a
similar strategy in SpaceX.

I suspect Boeing maybe treats software the same way as they've always done,
just one of many boxes that need to be ticked in a long waterfall like
process. They just throw more people at it hoping it gets done on whatever
time line they imagine it needs to be done. They are likely to be very
conservative with respect to tools, technologies, and practices (i.e. not much
has changed there in decades).

The fact that SpaceX is able to come up with a new design for Starship and can
then get it and the software working for several successful hovers & hops in
around 9 months, tells me they are able to iterate quickly and adapt to quite
major design changes. They launch a lot of rockets and they seem to be
learning a lot every time they do this. I don't see how they'd be able to do
this without very solid software development practices. I'd expect lots of CI
& test harnasses, probably lots of emulators and other tools, and I have a
hunch they probably use a lot of static code analysis etc. to prevent
preventable bugs from happening.

~~~
vonmoltke
It also helps that there is no union stopping SpaceX from driving it's
engineers and technicians to tens of hours of unpaid overtime.

~~~
madamelic
I really don't understand this line of thinking.

Any engineer or tech at SpaceX is likely worth their weight in gold at any
other company due to the company's perceived high bar. Additionally, I don't
hear complaints about their pay either, which is likely extremely good.

No one is locking them into cages and telling them they can't leave. I would
bet most everyone who is there strongly believes in the mission.

EDIT: Eh. They aren't paid fantastically but not slave wages either.
Additionally this is spread out over three very different cities.

[https://www.glassdoor.com/Salary/SpaceX-Engineering-
Salaries...](https://www.glassdoor.com/Salary/SpaceX-Engineering-Salaries-
EI_IE40371.0,6_DEPT1007_IP3.htm)

~~~
vonmoltke
> No one is locking them into cages and telling them they can't leave. I would
> bet most everyone who is there strongly believes in the mission.

That's true for a lot of other companies where people here have advocated the
employees would benefit from unionization.

For the record, I am conflicted on engineer unionization, and my original
statement was intended to be neutral.

As to this specific example, on one hand I understand the drive that comes
from having a real mission (as opposed to an MBA-concocted "mission") and how
a team who share that drive can execute with amazing speed; conversely, I have
seen how poorly teams of clock-punchers putting in the bare minimum can
perform.

On the other hand, I have also seen "the mission" used as a means to guilt
trip people into overwork or accepting substandard compensation.

Of the two, I lean towards the SpaceX approach being the better one. In the
absolute I don't particularly like either of them.

> EDIT: Eh. They aren't paid fantastically but not slave wages either.
> Additionally this is spread out over three very different cities.

> [https://www.glassdoor.com/Salary/SpaceX-Engineering-
> Salaries...](https://www.glassdoor.com/Salary/SpaceX-Engineering-
> Salaries..).

Those salaries look to be in line with what equivalent engineers at Raytheon,
Lockheed, and Boeing are making in those same locations, maybe a bit higher.
Thus, if the WLB reports from SpaceX are to be believed, their per-hour pay is
much lower. It's hard to tell, though, without a proper breakdown by years of
experience. Also, there are certainly teams within those companies where the
engineers put in just as much time and effort as SpaceX engineers. Boeing is
the only one I am aware of that has unionized engineers, though.

------
nickik
Boeing over the last 10+ years always got by far the most money for every
program (or were the only one selected) and the excuse from NASA is always
'they have greater schedule certainty'. Something that never actually worked
out in practice.

Starliner was selected over Dream Chaser for that reason. And Dream Chaser
would have been a much better selection.

Boeing was also selected for Experimental Spaceplane (XSP) from DARPA and
basically did nothing and dropped the program, while a company Masten Space
Systems couldn't survive even while they had way more innovative technology.

This human space flight stuff has the potential to cost Boeing a huge amount
of credibility, they got double the money compared to SpaceX and might end up
catastrophically under-delivering. They will never again be able to argue that
they should be preferred based on experience.

~~~
b34r
Between this and the 737-MAX they’re pretty much cooked. I don’t see many
airlines or governments trusting them much going forward.

~~~
magduf
Where are you getting that crazy idea? Why wouldn't the US government trust
them going forward? It doesn't matter if they don't deliver or their planes
crash and kill hundreds: what's important is that they know all the right
people, and they're "too big to fail". I see no indication that the government
will stop preferring them.

Of course, foreign government and airlines might very well stop doing business
with them, but that'll just cause the US government to prop them up even more.

~~~
b34r
Because SpaceX is delivering as promised. They’re getting the contracts going
forward and ULA will be dissolved.

~~~
nickik
ULA has nothing to do with this. ULA is a launch provider that will exist for
a long time yet.

------
heisenbit
> An investigation team has found the cause of the incorrect time problem that
> emerged during the December flight; Starliner gets that information from the
> Atlas 5 rocket that propels it to orbit. Because of the programming error,
> the spacecraft queried the rocket too early, before the clock had been
> properly set.

Anyone who has seen a space movie or airplane pilot movie knows these guys go
over checklists before doing anything major. Things are checked and double
checked independently. But looks like this culture has has been lost. Clock
was synced and not checked before launch. Mind boggling.

~~~
jakeinspace
Boeing didn't provide any information on their ground displays for this
particular clock, meaning that ground didn't have any way of verifying this
with a checklist. It's a value that should have been properly telemetered and
displayed, but was not. Don't knock JSC mission control, they're extremely
professional and don't make silly mistakes like that.

~~~
mzkply
But what's crazy is that once the capsule disconnects from the second stage of
Atlas, the only remaining part of the whole system... is the Starliner. So
what other clock could they be looking at? The Atlas V blows up in the ocean.

------
Nothing2SeeHere
There are a few points I'd like to make. First, no one writes perfect code on
the first go. That's why we have testers. Second, the testers don't want to
cut corners. Management pressures them to do so and/or hires people who don't
know what to do. I hear there was one manager over flight software and
avionics (fired a few months ago) that was directly responsible for most of
these problems. However, it isn't all his fault. His managers wanted good news
and he was happy to give it to them. Faster! Cheaper! Lay offs! Now all the
money "saved" and more will be spent redoing what should have been done right
the first time along with the cost of public humiliation.

But the flight software team had very smart people (those that were not driven
off by the aforementioned manager) and they did what they were supposed to do.
The rush to test and fly is a management problem, not a technical one.

------
iFred
I might be alone in this sentiment, but is anyone else out there with a
history working with Boeing or in the industry feel slightly irritated with
the armchair hot takes seen in the comments sections?

~~~
Analemma_
IME it's just the opposite: my uncle is a retired mechanic who used to work at
Boeing, and over Christmas dinner he opined that the press and armchair hot
takes are if anything not going far enough, and that internally Boeing is even
more of a disaster than we know.

~~~
perl4ever
People have been complaining about Boeing management for _years_ , and if it
seems like "years" to me, it's probably 20+. When people have been ignoring a
problem for a very long time, and it finally gets to a tipping point that has
dramatic public consequences, you can expect it to get worse for at least as
long as people are coasting on the inertia from their previous assumptions
everything was fine.

------
jussij
Back in 1992 Queen Elizabeth II, for various reasons, described her year as
_annus horribilis_.

I suspect Boeing could use similar words to describe the year they have just
experienced.

------
neonate
[https://archive.md/tft4l](https://archive.md/tft4l)

------
SPBS
Is Boeing still using Ada?

~~~
TkTech
Boeing uses pretty much everything.

Ada is still widely used and having decent Ada experience will get you a job
pretty quickly, especially if you know how to get things done with most of the
run-time disabled.

F-22 flight controls are all Ada for example. The new training tools are
C#/C++ (Unity). Inventory tracking is Java.

~~~
perl4ever
People on HN and elsewhere frequently slam Ada (on the rare occasion it comes
up) but I really liked it, in the one college course I took that used it. That
may have to do with becoming fluent in Pascal before doing much with C, and
considering the general appearance of similar languages to be more pleasant.

------
moneromoney
"move fast and break things"

------
foobarbecue
'“It is our belief we wouldn’t have found it if we hadn’t gone looking,” said
Jim Chilton, senior vice president of the space and launch division at Boeing'

You don't say.

~~~
pfdietz
"You can observe a lot by just watching." \-- Yogi Berra

------
hurricanetc
Boeing doesn’t know how to deliver on time and on budget any more. They have
spent too long being able to do whatever they want that when faced with a
nimble competitor they absolutely flopped.

And they finally found a circumstance that they couldn’t lie and bribe their
way out of.

And remember everyone... NASA gave Boeing $2 billion more than they gave
SpaceX.

~~~
tzfld
"NASA gave Boeing $2 billion more than they gave SpaceX."

I saw this argument in a lot of places. Comparisons like this doesn't have a
point without context. And with the context it's not something that you may
want to suggest.

Edit: In a nutshell, the starting point is not the same. Crew Dragon is
derived from the cargo Dragon created for COTS years ago. The Starliner was
made from scratch just for the crew transportation program.

~~~
kevin_thibedeau
Except that Boeing's justification is that they are a "legacy" contractor
needing more funding because of their bureaucracy. One which doesn't seem to
add much value.

------
Accujack
Oh, look... a paywall. (closes tab)

~~~
bookofjoe
[http://archive.is/tft4l](http://archive.is/tft4l)

------
b34r
Boeing is done. Trust in them has eroded severely and now they’re paying the
price for profiteering at the expense of safety.

Was nice knowing ya!

