

787 Dreamliner teaches Boeing costly lesson on outsourcing - pointillistic
http://www.latimes.com/business/la-fi-hiltzik-20110215,0,442445.column

======
gaius
_Sure, it's immoral to abandon your loyal American workers in search of cheap
labor overseas._

I know a lot of people who talk like this. However when it comes to spend
their own money, they always go for whatever's cheapest, whether it's clothes,
electronics, whatever. You can literally see these people ranting about
worker's rights while wearing Primark and talking on a Huawei OEM'd handset.

Me, I'm a capitalist, and proud of it, but y'know, I'm the one willing to
spend a little extra for a UK PLC product. Bullshit talks, money walks.

~~~
jhghjmhnbgv
It wasn't the foreign suppliers that were the problem - Mitsubishi delivered
the wings on time. It's was the excessive outsourcing to layers and layers of
subcontractors without enough oversight. So a one mall plating shop in
Albuquerque goes out of business and a critical path component is missing.

This is fairly common in defense, highly specialized components are subbed out
to small shops - there are lots around LA - the problem here was that there
was so many layers of sub-sub-sub contractors all being bidded out to the
cheapest supplier nobody had any idea who was making what. That works for
Walmart - it doesn't work for Boeing.

How they thought they were going to ensure40years supply of spare parts is
anybodies guess

~~~
quanticle
Just to elaborate on your point, the outsourcing model works for Walmart
because Walmart is selling numerous separate products, rather than an
integrated whole. It seems that Boeing forgot that lesson and started thinking
of their airplane as a "collection of parts flying in close formation".

~~~
jhghjmhnbgv
Also very few of Walmart's products are critical path - if they run out of
lemon scented cleanup squares the store still opens (gratuitous hitch hikers
reference)

------
pmorici
In Boeing's case wasn't there a secondary goal in outsourcing of drumming up
foreign demand? I seem to recall reading an article a few years back where
Boeing agreed to outsource the wing sub assembly to some Asian country and in
return that country agreed to buy some number of Boeing planes. I can't seem
to find the article now though.

~~~
notahacker
This is part of it. Its also part of the reason why Airbus/EADS quite
successfully builds bits of aircraft all over Europe (and even proposed
expansions of operations in Alabama as part of their bid for USAF tankers)

Also, it may surprise the article author to learn that there are outside
specialists who have capabilities that Boeing couldn't effortlessly replicate
in Seattle. Italy and Sweden certainly don't have low labour costs and they
don't have much of a domestic market for long range airliners either.

Ultimately Boeing's logistical cockups have more to do with the overall
ambition and complexity of the project than cost cutting.

~~~
jhghjmhnbgv
Airbus's distributed manufacturing is more about pork-barrel politics, it's
spread around the countries that own shares in the airbus consortium

Ironically Several of the components of the 7e7 are made in europe by Airbus
suppliers. The undercarriage, tail and bits of the fuselage are made in the
same plants that are building them for airbus.

AFAIK the only certified engines at launch are Rolls-Royce

------
microarchitect
The title should really be 787 Dreamliner teaches Boeing a costly lesson in
due-diligence. This sort of thing has nothing to do with outsourcing and
everything to do with poor decision making by management.

~~~
gnaffle
Here's the problem, though: Who is capable of doing the due diligence? By
outsourcing, you end up losing a lot of the in-house knowledge necessary to
spot problems early on.

~~~
CaptainZapp

      Who is capable of doing the due diligence?
    

Even if they're everybodies favorite wipping boy right now. I'd enter Nokia as
the winner in the due dilligence category.

There's a classic business case study about two handset manufacturers, a
Philips fab in Mexico and a small fire in that fab. Details can be found here
:

<http://www.ftpress.com/articles/article.aspx?p=1244469>

~~~
mncolinlee
In light of Nokia's latest struggle with iOS and Android, aren't they both a
success story and a cautionary tale for due diligence and proactive
management. Nokia had a second fire-- a burning platform.

By the logic of the book excerpt that I finally had time to read, they should
have read the tea leaves about the iOS and Android universes much quicker than
they did. Nokia's burning platform memo would not have been so dire if their
response had occurred one year earlier. Nokia was already considerably left
behind by the application universe at that point. iOS had momentum then and
should have been considered a direct threat even though the Android userbase
was only 1/10th of the size it is now.

I still find that corporate cultures tend to sideline Android to this day
because they underestimate the value of an open platform on commodity hardware
with infinite hackability that intrigues developers. The same incredible
growth occurred with Linux and Apache on the web as the 888% Android growth in
the last year. Android will win partly for the same reason Microsoft beat
Apple's superior UI in the 80s and 90s-- Apple is still tied to expensive
proprietary hardware and the need to completely control their platform.

------
Derbasti
This makes it sound like US companies were better than foreign countries just
because they are american.

Me, I feel slightly insulted by that. Just look at american cars. I kid.

There is nothing wrong with patriotism and there is nothing wrong with
outsourcing, if both is done with due diligence and proper planning. But there
is no way one company is any better than any other company just because it is
registered in a different country. The 787 did not have problems because of
outsourcing, it has problems because of bad planning--at the all-american
Boing HQ itself.

~~~
jasonkester
You're only insulted because you misread outsourcing to mean offshoring.
Boeing farmed work out to other companies, and it bit them. The fact that some
of those companies were based in other countries is not a point mentioned as a
contributing factor.

~~~
Derbasti
Oh! My bad, you are right! I repent!

------
pohl
_Among the least profitable jobs in aircraft manufacturing, he pointed out, is
final assembly — the job Boeing proposed to retain._

...and...

 _"I warned Boeing not to make the same mistake. Everybody there seemed to get
the message, except top management."_

If top execs aren't keeping an eye on where the profits are, you're screwed.
Are they compensated too well to care?

~~~
tomjen3
No. They are given stock options which means that they have an incentive to
pump the price up short term even if it means that long term the company would
loose more money.

~~~
mistermann
Precisely, and this explains a wide swath of seemingly nonsensical decisions
being made in western countries in the last decade.

~~~
acgourley
I feel like most execs are still concerned about their long term credibility
and legacy ove their ability to pump some of their stock at a higher value in
the short term. And I say that with full cynicism - it's simply suboptimal for
them personally to do so.

------
pgroves
I'm surprised no one is talking about the fact that this is one of the most
complex projects ever undertaken by human beings. It was being designed and
orchestrated in the only way we know how, and it broke down. I don't believe
bringing in the people 20 layers below the top management and giving them a
paycheck that says 'Boeing' on it would solve all the problems. They were
building something too complicated for their basic process (blueprints,
documents, proposals, etc.)

Many of the problems such as parts that don't fit and supply chain problems
could be solved by current machine learning techniques. Too bad Google and
Facebook hire all those people and sets them to work optimizing ad placement.

~~~
gnaffle
The surprising thing is that they managed to design and build the 747 in the
60s in record time, without access to machine learning or any CAD tools
whatsoever.

Although the Dreamliner is a completely new design using new materials, I'd
say the 747 was a far more impressive accomplishment at the time.

~~~
Terry_B
The problem I saw working on the 787 and other modern aircraft is that we now
bury ourselves in analysis and data in the name of accuracy. Just like in
programming, when the code gets complex and you have more and more
dependencies. One small change impacts lots of other things and you go round
and round in circles and iterations.

Back in the day they used to do more conservative, more basic calculations and
conventinoal designs that they knew would work.

If there was a small change they didn't have to go and revisit everything
because everything was still well within their original conservative
calculations.

It's basically Paretos Law, they got 80% of the results for 20% of the effort.
We are now chasing that other 20% of the results and the complexity to do so
has balooned.

~~~
pgroves
I appreciate someone in your position posting this comment more than you can
imagine. The research project I'm working on is an attempt to confront this
problem. As you say in another comment on this thread, GA's are good for small
problems, but don't scale well as more _people_ are involved on the problem.

The problem is more about software engineering and project management, in my
opinion. What would we replace CAD systems with if we could? What does the IDE
look like when everyone is trying to encode their expertise in such a way that
the final design is determined by a machine learning algorithm? That is, it's
conceptually easy to imagine those final iterations being done by a machine
learning algorithm, but how do we do that?

~~~
Terry_B
Geez, where do I begin... :)

It's a fascinating problem and on very small engineering projects with a small
team of people willing to experiment with new methods it would be great to try
and take things in this direction and see what works.

The reality of how aircraft projects and pretty much any engineering project
really, is undertaken is currently so far from lending itself well to this
type of thinking it's not funny.

Best that can be done is to chip away at little sub problems at a time tyring
to improve things.

In terms of CAD data, there is a shared workspace amongst the different teams
around the world. You can see what everyone else is up to.

The problem is that 90% of what goes into that design is not something that
lends it self easily to an optimisation problem of any kind. At least not
without gathering a ton of metrics that are currently in various documents
from the 40s, people's heads and difficult to quantify metrics like what
manufacturing options are currently available, what things people already have
experience at etc etc.

A huge part of engineering is almost black magic. It's peoples experiences and
judgements.

Also, many things are not done by analysis because we don't understand all the
factors involved fully, they are done based on emperical evidence of what
worked in the past. This emprical data is spread all over the place in tables
and documents or just some old timers brain.

Reducing much of it to an optimisation problem is an enormously difficult
task.

Then there is the analysis side of things. This is where it could be more akin
to programming and could be "solved" potentially.

However, the current processes involved in doing engineering analysis makes
life difficult. Basically, hundreds or even thousands of engineers produce
mountains of Excel spreadsheets, words docs, text files of data and hundreds
of other formats that all relate to one another somehow to tell a story of
whether the aircraft is safe to fly or not.

This is far from computer code where the variables all neatly reference one
another and a computer can understand it. Picking through it and figuring out
how the numbers in one file match the numbers in another is a nightmare at
times. I do not envy the senior engineers that have to check it and sign their
name on it to say it is correct.

As you say, if everyone was working in a common IDE to do this and forcing
things to have relationships the world would be a much better place for
engineers. The mountain that has to be climbed to get there in the aerospace
industry is enormous however.

So that I don't sound like a complete skeptic though, there is hope!

On the design side, things are most simple early on in a project where people
are working on the early conceptual designs. This is were the problems are
simple enough that things can be solved. It's once the conceptual design is
thrown out to the large engineering departments to fill in the millions of
details that the trouble begins.

And on the analysis side, the world is moving toward more and more finite
element analysis. This is basically the brute force approach to analysing a
structure. This is where it is in machine readable form and optimisation is
possible. However, the answers the computer gives you are always layered and
compromised from the engineers looking over them and having to turn them into
something that's actually practical, can be built and takes into account the
many requirements the computer does not know about.

~~~
pgroves
I agree almost 100% with what you're saying. I'll just add 2 comments:

1) Regarding the fact that at any given time, there is knowledge that people
have that hasn't been encoded in a form usable by the optimizer: I think this
can be addressed by generating a set of decent designs with machine learning
(on the pareto front, if you will) and then providing user interface tools to
allow the humans to pick the designs that best handle the knowledge that
wasn't optimized against. These user interfaces will often have to be quite
elaborate but that's a tractable problem.

2) When working with Genetic Algorithms, and any machine learning technology,
really, my experience has been that the first results only show what is wrong
with the problem setup. The designs that are returned by a genetic algorithm
are invariably nonsense at first and the then it's a matter of playing whack-
a-mole as the optimizer exploits inaccuracies in the problem statement, a fix
is made, a new exploit is found, repeat. This is often viewed as a deficiency
of genetic algorithms but in my opinion the fact that the metrics being used
can be gamed is valuable information that needs to be addressed as early as
possible.

Finally, I understand my original comment came across as somewhat naive. If we
ever get to a point where we can design physical products, I am imagining
starting with something along the lines of a mechanical pocket watch, or maybe
a nice desk chair. Airplanes are gonna be last.

~~~
Terry_B
Oh absolutely, if the problem is simple enough it's definately possible :)

Even in aerospace finite element optimisation techniques have led to some very
effecient structures.

The problem is when you pass the limits of what is manufacturable and have to
start breaking it up into simpler pieces joined together based on time,
budget, abilities, what you know will work etc.

Also when you have to meet hundreds of requirements like having to fit lots of
systems that people haven't designed yet and don't fully know what they'll
need yet. It's the problem of compounding unknowns etc.

An awesome chair is definately doable :)

------
spoiledtechie
Boeing has been late on almost all their projects. The Australia tanker is
behind by 2 years, the Japanese shipment is behind by 3 years and counting.

Its sad to see a company that is politically corrupt still win contracts by
the U.S. Government even though it delivers years behind schedule.

~~~
sanj
Unlike software, which is always on schedule.

~~~
calloc
Not if you are part of the highway bandits!

------
jdfreefly
I'll take planes I don't want to get in for 1000 Alex!

Seriously, think of software projects that took years longer than predicted
and went way over budget. What kind of product was finally delivered? I don't
want to be a seller of fear (I hate that about the US) but I won't be
surprised to see serious issues with this thing when it enters service. Anyone
here ever work on a 3 year long 1 year project? I have and its rare to see the
attention to quality at the end of that 3 year death march that you see in the
beginning.

------
sagacity
Off topic, but I'll keep this short:

At the time I post this, there are just 52 comments on the story at LATimes
site, while there are 75 here !!!

I guess that'll surprise a few.

~~~
T-hawk
Not too surprising. Any talk of outsourcing will hit a pretty personal nerve
among the hacker and programmer community.

~~~
sagacity
A small clarification: :-)

Being fairly new here at HN, I was not fully aware of the popularity of this
community and actually speaking, the only context in which I made my
observation was a comparison of general traffic/interest/engagement levels
between HN and LATimes.

------
motters
It depends upon how the outsourcing is managed. By outsourcing you're adding
extra complexity, unknowns and unmeasureables to the problem - increasing the
variety in cybernetic terms. Unless you have a sufficiently good way of
managing that additional complexity you're going to get into trouble, which in
business terms means overruns or in the worst case failure of the enterprise.
See Ashby's law for details.

------
ramkrishnapk
Aim to establish India in global supply chain: Boeing
[http://timesofindia.indiatimes.com/business/india-
business/A...](http://timesofindia.indiatimes.com/business/india-business/Aim-
to-establish-India-in-global-supply-chain-Boeing/articleshow/7458158.cms)

------
igrekel
Interesting viewpoint from hat retired employee who said the final assembly
has the least value. Kind of put the fact that the final assembly of
electronics such as the iPhone is outsourced while some of the component are
home mad or at least home designed.

~~~
lurker17
Why do you believe that components are home made? They are only home-designed.

Nearly all of the iPhone is built and assembled outside of Apple and outside
of the US. The (non-standard parts of the) hardware are "designed by Apple in
California", and the bulk of the software is created in Cupertino as well.

[http://texyt.com/iphone+manufacturer+supplier+assembler+not+...](http://texyt.com/iphone+manufacturer+supplier+assembler+not+apple+00113)

------
wazoox
Not much different from what happened to airbus only a couple of years before.

~~~
CaptainZapp

      Not much different from what happened to airbus only a couple of years before.
    
    

Uhh, no. That had much more to do with the structure of EADS (the airbus
parent) and how it's a joint company between multiple European countries.
Notably France, Germany, Spain and the UK.

Airbus also had French / German co CEO's at that time, which didn't help
decision making.

The problem stemmed from the fact that manufacturing is divided between
various plants in various countries. Notably between Toulouse (France) and
Hamburg (Germany). This is certainly not the same thing as outsourcing
critical components to external suppliers (which I'm sure Airbus does too to a
lesser extent).

Upon final assembly of the A380 in Hamburg they bungled the cabling which hit
them back another six month or so. They where other problems of course, which
there are always bound to be on such an ambitious project and with the
(likely) unrealistic time scale for delivery.

The A380 was two years late, which is probably not bad for such a project. I
wouldn't bank on it that the Dreamliner takes of in 2011. At least not
commercially.

~~~
jhghjmhnbgv
The airbus style big block integration actually worked quite well - engines
from RR, undercarriage from airbus and wings from Mitsubishi all worked well.

The problem was that they eg. farmed out the avionics to mega corp, who farmed
out the hardware to company A and software to B. Company B farmed out bits to
company C and company D. Nobody realised that company D was a single
contractor shop until they were late which went all the way up the chain.

There was an article on here at the time, Boeing literally couldn't find out
the name of the company doing a vital part of the avionics software.

