
Boeing's 737 Max software outsourced to lower-paid engineers - pseudolus
https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-737-max-software-outsourced-to-9-an-hour-engineers
======
codingslave
"In offices across from Seattle’s Boeing Field, recent college graduates
employed by the Indian software developer HCL Technologies Ltd. occupied
several rows of desks, said Mark Rabin, a former Boeing software engineer who
worked in a flight-test group that supported the Max.

The coders from HCL were typically designing to specifications set by Boeing.
Still, “it was controversial because it was far less efficient than Boeing
engineers just writing the code,” Rabin said. Frequently, he recalled, “it
took many rounds going back and forth because the code was not done
correctly.”

Boeing’s cultivation of Indian companies appeared to pay other dividends. In
recent years, it has won several orders for Indian military and commercial
aircraft, such as a $22 billion one in January 2017 to supply SpiceJet Ltd.
That order included 100 737-Max 8 jets and represented Boeing’s largest order
ever from an Indian airline, a coup in a country dominated by Airbus."

~~~
su30mki117
How are the points you have mentioned in your post relevant to the MCAS fault,
which the article is supposed to be about?

It is explicitly mentioned in the article that the Indian companies DID NOT
work on the faulty MCAS system - _Boeing said the company did not rely on
engineers from HCL and Cyient for the Maneuvering Characteristics Augmentation
System, which has been linked to the Lion Air crash last October and the
Ethiopian Airlines disaster in March. The Chicago-based planemaker also said
it didn’t rely on either firm for another software issue disclosed after the
crashes: a cockpit warning light that wasn’t working for most buyers._

This looks like a misleading article trying to shift blame away from Boeing
management. Yes, there are problems with outsourcing, which the article brings
up trying to show as reason for the problems with 737 Max.

~~~
supercanuck
You are missing the point. This information is necessary in making the
decision to continue flying on Boeing 737 Max Airplanes.

------
crorella
Money saved by firing experienced engs and outsourcing the development <<
money lost because they can't sell planes.

Why is that so hard to understand? Massive fail at risk management maybe?

~~~
ganeshkrishnan
The software did exactly as it's supposed to do: If the AOA sensors tells you
the plane is tilting too high up, pitch the nose down a little.

The main issue was that there was only one AOA sensor connected in standard
packages and the hardware for it failed.

The pitot tubes failed for the doomed Air France incident which caused the
software to read incorrectly and stall.

For what it's worth AirBus has a huge engineering department in India:
[https://www.airbus.com/careers/search-and-
apply/opportunitie...](https://www.airbus.com/careers/search-and-
apply/opportunities-overseas/vacancies-in-india.html)

Articles like this are just drummed up to stir negative emotions about
"others" (with "struggling to understand English" and "unable to code
fizzbuzz") as is evidenced in the rest of the comments below.

~~~
dTal
This comment is glib and misinformed.

Firstly, no the software did not do exactly as it's supposed to. It crashed
the plane by rendering it unflyable. It's not supposed to do that. And your
description of the MCAS specification is vastly over-simplified - it is not an
autopilot and its job is to gradually change the _behaviour_ of the plane in
different flight modes (which it must detect), _not_ to directly fly the plane
as you suggest (and even two AOA sensors would be insufficient for the type of
functionality you describe). The behaviour of the MCAS is absolutely in
question, in particular the way it kept "re-adding" the correction, with
cumulative effect, ending up fully pitched down.

Secondly, if by "the doomed Air France incident" you mean AF447, your
description of it is flat-out wrong. The pilot, not software, stalled and
crashed the plane by manually holding the stick back for 15 minutes. The pitot
tubes briefly iced over (they didn't "fail") which is not an uncommon
occurrence, and the only consequence of this was that the software guardrails
that would have prevented the pilot from crashing the plane were removed. All
instrumentation systems were fully operational at the time of the crash.

~~~
ganeshkrishnan
It kept re-adding because the flap of the aoa sensor was stuck in the
"aircraft is pitching up" position.

Boeing management did not add a second sensor to some planes and also pilots
were not aware of how to switch off mcas.

If you worked in large scale project you would know that most devs are
assigned components to work on and usually have no idea what other components
are or how they will interact.

~~~
gotdevops
This is where testing comes in and from the fact packet it appears that the
testing was also outsourced to another Indian firm?

------
toss1
>>"...recalled one manager saying at an all-hands meeting that Boeing didn’t
need senior engineers because its products were mature. "

Whatever manager that was, should never be working in an aerospace context,
where one design failure or fault can literally kill hundreds of people in
minutes (and in this case, did exactly that).

Clearly, Boeing has become infested with people who literally do not
understand the fundamental life-and-death consequences with which they are
dealing, and who treat everything as fungible commodities. The entire story is
essentially about how the outsourcing and commoditizing just expanded and
expanded until now, when there were fatal consequences for the passengers, and
perhaps even for the company (TBD).

Unless this changes, this is no longer a company who can be trusted with such
life-and-death engineering decisions.

------
chrismatheson
I’m not discounting the possibility of bad workmanship by Boeing. However
“temporary workers making as little as $9 an hour to develop and test
software” does sound to me like the reporter might have found some highly paid
mechanical Turk style employees given the job of battering against a solidly
built piece of software just because there would be a possibility of finding a
problem.

If the sentence was “the mean salary of $9/hr” then I would be thinking
differently.

~~~
danielg6
The article mentioned recent grads (working in Seattle for an India-based
company) who wrote code on specifications, not “mechanical turk”-style
employees. It was followed by a quote from a former Boeing software engineer
affirming this.

Is your issue just with the amount ($9) being paid?

~~~
throwawayMUSE
I worked at a Fortune 10 in the early 2010s for several years. We had a small
army of local and offshore Indian contractors through Larsen and Toubro
Infotech, an Indian IT company. They were paid between $10-20 an hour, many
couldn’t understand English. These guys have no CS background and would get
stumped writing fizz buzz. But hey, management got ball game tickets and other
favors from the L&TI reps. One of the directors admitted to getting a cut
under the table after he left to go work for a competitor.

To be clear, we paid $120-150 per developer and L&TI paid them nothing. You’d
see .NET code with a single class that has a few thousand lines of code, no
concept of unit tests, integration tests, manual or automated acceptance
tests, usability tests, etc.

Shelf life of most of this garbage was a few months if something ever did get
delivered. This went on for years.

The whole industry is corrupt.

~~~
ganeshkrishnan
You must have been a pleasant employee to work with.

~~~
throwawayMUSE
I sense sarcasm. Is it because I called out a legit issue? For my employer, it
didn’t matter because IT had a reputation of delivering garbage and so our
business partners weren’t shocked at what they delivered. If something like
this happens at a Boeing (I don’t know if it is), then heads ought to roll and
people should be going to jail.

------
la_barba
I think this is just rabble-rousing. For all the flack outsourcing gets, if
you hire an incompetent person, and they fuck up, you're still responsible for
hiring an unqualified person. I don't think it matters if that person was
outsourced or not. I haven't worked with any offshore teams, but friends tell
me you can do what are called "client interviews" where the client, in this
case Boeing, has the option of interviewing the actual people who will be
assigned to work on your project.

~~~
iamnotacrook
"but friends tell me you can do what are called "client interviews" where the
client, in this case Boeing, has the option of interviewing the actual people
who will be assigned to work on your project."

LOL! Ahem. You'll discover that they get someone competent in for the phone
interview and you'll later discover that the story is now that the person you
spoke to was representative - in terms of skills, experience and competence -
of the sort of person who'll actually doing the work, and is not the one doing
it. Later still you'll discover that this is not the case.

Still, they're very cheap, aren't they?

~~~
AnimalMuppet
If they're in offices right across from Boeing Field, you'd _think_ that
Boeing might be able to do actual face-to-face interviews with the _real_ (not
just representative) people. You also might think that anything less is
definitively a management fail.

~~~
nonamechicken
This is what typically happens to h1bs: if someone comes from India to work at
the client location (Boeing office in this case), and if the customer is not
happy with them for whatever reason, they get send back within a month or 2.

------
Havoc
Really wonder how deep this rabbit hole goes. I could live with the mcas drama
but these articles hinting at seemingly pervasive internal culture issue makes
me wonder whether I want to be on any Boeing jet at all

------
mehrdadn
When is the media going to get the message that this wasn't a software issue?

~~~
pseudolus
The final verdict isn't in, but it appears to have been a systems engineering
problem with a software component.

~~~
mikeash
The software performed exactly as designed. The problem is a faulty spec for
how it should behave. The software is pretty much incidental.

~~~
salex89
You are correct, however... I can't imagine a single engineer with any kind of
proper education (formal or not) implementing any piece of software or
hardware without thinking:"One sensor? Really? What's the failure rate on this
thing". That might have been enough to stop the chan of events.

~~~
CydeWeys
The engineers may not have realized they were relying on one sensor, or they
might be the $9/hour variety with no avionics experience who don't know any
better.

The main problem was higher up. The overall design was bad, and Boeing didn't
seem to have a handle on it.

~~~
anticensor
Or they have a "obey all orders even when it is blatantly criminal" clause.

------
dba7dba
Why worry about meddling of elections by Russians when large American
companies that make life/death products and one of the pillars of US economy
goes around doing stuff like this, effectively committing a corporate seppuku.

I used to watch PR videos from Boeing on new planes being tested and feeling
sense of wonder and awe. Now I feel like puking. Of course not the rank and
file. But the ones in corner offices.

------
rthornto
These low paid engineers have short tenure (high turnover) and graduate to
higher paying jobs. That is their goal and companies know that. So, in the end
they are honing their skills on projects where hundreds of peoples lives are
at stake. It used to be the design built by them and quality checks completed
by experienced engineers. In the quest for more profits (narrow margins), they
forgo this valuable process step. Most 'business' executives have no clue and
never will and more don't care. Be honest.

------
bayareanative
I know which is worse: it's not 737 NG's Ducommun's substandard critical
structural parts, underpaid subcontractor software engineers in India or
Boeing monetizing safety as optional extras... it's the systematic, systemic
corruption of safety regulatory protections capture allowed this to happen.
Planes didn't fall out of the sky 30 years ago because enforcement entities
actually had resources and tried to do their jobs.

The inescapable problem is that all newer planes from Boeing and likely other
manufacturers are suspect in uncountably tens of thousands of ways because
they cannot be presently assured (without engineering reviews, inspections and
documentation) to be fit for passenger travel due to the lackadaisical
regulatory climate in which they were developed and manufactured. The 787, 737
NG (-600 and up) and 737 MAX are just the tips of the known problems. And now
Boeing wants a folding wing plane too?

------
exabrial
Not trying to defend them, but this _was_ like every company from 2002-2012
until everyone collectively realized the asset is the people not the code.

------
olliej
I’m reminded of an article many years ago about when Boeing was outsourcing to
to improve return on capital or some nonsense and at some point someone just
said to maximize return on capital they should just have someone else design
and make the planes and put a Boeing sticker on them

~~~
durnygbur
Run an aerospace franchise. Let other design and produce the aircrafts. Wet
dream of cost cutting rent seeking managers.

------
dba7dba
I am so tired of software engineers that, patch that software, etc etc.
Ultimately it all comes down to $, $ that executives thought they could save
for the company AND make that yearend bonus.

It's not engineer that or this. It is MBA this and MBA that.

------
nonamechicken
I really doubt 737 Max's failure has anything to with outsourcing to India.
Usually, most of the work outsourced to India are low value work (what I call
as 'near brain dead' type). I don't think Boeing would outsource anything even
remotely critical to a company like HCL. From the article:

>Based on resumes posted on social media, HCL engineers helped develop and
test the Max’s flight-display software, while employees from another Indian
company, Cyient Ltd., handled software for flight-test equipment.

>Boeing said the company did not rely on engineers from HCL and Cyient for the
Maneuvering Characteristics Augmentation System, which has been linked to the
Lion Air crash last October and the Ethiopian Airlines disaster in March. The
Chicago-based planemaker also said it didn’t rely on either firm for another
software issue disclosed after the crashes: a cockpit warning light that
wasn’t working for most buyers.

Even inside US, Indian engineers are not allowed in critical software that
goes into defense and (most likely) airlines, only citizens can do that. So,
there is no way they would have outsourced safety critical stuff to HCL.

~~~
gotdevops
For DOD no foreign you are correct. Unfortunately 737 max is not a no foreign
DOD project. Please understand these outsourcing companies are extremely
powerful and influential. They bring In highly trained and connected sales
teams to negotiate big deals at the executive and political levels. In this
case 11bn deal to Boeing for 1.7bn in services contracts for India. Please
consider what level at Boeing that sort of deal is brokered at? Ask yourself
what the repercussions are for internal folks who call out the outsourcing
firms? As you suggest outsourcing firms often start out with non critical
tasks like operations and maintenance. Through this process they actually
consider those contracts as investments to gain access to learn the lay of the
land especially where the critical systems are and who manages them. Then
these firms often including architecture leads who are camouflaged sales
people do their trade craft of above board and from my perspective suspicious
below board tactics until they open the door to more critical projects and
higher paying contracts. HCL, TCS, etc work on critical system not only in
this industry but also in Medical, Infrastructure, Utilities,
Telecommunications, Broadcasting, etc, etc. I wish you were correct but sadly
I’ve seen the transition first hand and have been brought in to unwind what
outsourcing firms have done in the name of cost savings. At the same time I
was told and my teams were told to investigate, identify and fix the issue but
don’t blame the outsourcer or this will be your last contract. This i was told
was due to the level of the outsourcers executive Sponsership. Consider the
influence these firms have over the execs and the reputation at stake with
anything negative or other then thank you so much for the great work. In fact
we were told that the outsourcer was to be treated as the customer. Even
though months of war rooms and rework fell squarely on their ineptitude.

------
blinkingled
> Boeing said the company did not rely on engineers from HCL and Cyient for
> the Maneuvering Characteristics Augmentation System, which has been linked
> to the Lion Air crash last October and the Ethiopian Airlines disaster in
> March.

~~~
carlmr
That may well be, but that constant supervision and error correction takes a
lot of your experienced engineers attention away from the important things.

~~~
blinkingled
Assuming same engineers were doing the constant supervision and error
correction and not few dedicated ones. (My previous company had one guy whose
job it was to do this - that got good results because the guy knew what to
look for and how to get it fixed.)

------
stcredzero
How in the world does one hire $9 an hour software engineers?

~~~
zaroth
> _Still, for the 787, HCL gave Boeing a remarkable price – free, according to
> Sam Swaro, an associate vice president who pitched HCL’s services at a San
> Diego conference sponsored by Avionics International magazine in June. He
> said the company took no up-front payments on the 787 and only started
> collecting payments based on sales years later, an “innovative business
> model” he offered to extend to others in the industry._

~~~
sct202
How is that allowed? It is like basically the equivalent of dumping products
below market and production price like what China frequently does to steel.

~~~
kevan
Why wouldn't this be allowed? This is the company equivalent of a person
working on commission vs salary. Salary is more stable and predictable,
commission is riskier but has higher potential for upside.

~~~
leetcrew
it's also the equivalent of what they call "royalties" in the music industry.

------
WomanCanCode
You don't actually save a whole lot of money by outsourcing. "Engineers in
India made around $5 an hour; it’s now $9 or $10, compared with $35 to $40 for
those in the U.S. on an H1B visa, he said. But he’d tell clients the cheaper
hourly wage equated to more like $80 because of the need for supervision, and
he said his firm won back some business to fix mistakes"

------
echelon
Yet another shocking revelation in how little Boeing values lives when it
comes to the bottom line.

This should be a company-ending event for Boeing. These folks should not be
making aircraft used by living, breathing human beings.

The stock holders Boeing is beholden to, that clamor for this cost cutting,
should pay.

~~~
ericd
The stockholders are you and I, via retirement plans.

~~~
echelon
Our investment portfolios and retirement plans can be rebalanced. Those that
died in the two Max crashes are gone forever.

This isn't cheating on emissions. We have to send a strong message.

~~~
ericd
If you mean just lose money when you say the stockholders should pay, then
yeah, whatever. But if you mean that they should be held responsible, I think
that’s absurd. That’s what I meant by my response.

Losing 1000 people is a tragedy, and those responsible should be held
responsible. But a company like Boeing is a huge machine with many people
working diligently on extremely complicated products, the vast majority of
whom didn’t touch this faulty system at all. Which is to say you shouldn’t
throw the baby out with the bath water.

------
durnygbur
"These are the rates in India and Eastern Europe, we're paying well above the
market actually, so we are getting crème de la crème" \- some directors and
managers at Boeing, probably.

------
xerxex
I wonder if this is unchecked laissez faire capitalism at work? Nobody wants
to fly on Boeing 737 Max anymore but this has cost hundreds of lives to get to
this point.

~~~
leetcrew
there's very little "laissez-faire" about the relationship between Boeing and
the US government.

~~~
AlexCoventry
There was definitely regulatory capture which led to lax FAA oversight,
though. Maybe a free-market fundamentalist would argue that they would have
done better if they weren't dragged down by all that red tape, but it suggests
to me that a looser regulatory environment was more dangerous.

------
chimpburger
"Rabin, the former software engineer, recalled one manager saying at an all-
hands meeting that Boeing didn’t need senior engineers because its products
were mature."

Boeing is dead to me.

~~~
newsoul2019
Understand the sentiment, but can anyone be sure that Airbus isn't doing
something similar.

~~~
chimpburger
Irrelevant. If Airbus is doing the same, then they are also dead to me.

~~~
chappi42
Dead meaning you won't fly any more? (beneficial for the environment)

~~~
chimpburger
Dead meaning that, in my opinion, they have sold their soul to Satan. Boeing
was previously at the apex of engineering. The 747 was a respectable
achievement in 1968. The MAX debacle has exposed a catastrophic fall in
engineering standards.

------
danielfoster
There are some Boeing executives who should be making $0.35 an hour in prison.

~~~
dba7dba
I don't even want them in or near the prison. I just want the salary/bonus
they made while employed by Boeing, making the decisions that led to this.

And give them to the family of the victims, ON TOP of whatever punitive damage
Boeing will be paying.

Holy crappus, what was Boeing thinking?

------
usrusr
Is that so surprising? I have never worked in a domain like that, but I have
always assumed that flight computer software would be built with process so
deep that you could almost leave the implementation to trained monkeys,
implying that those $9/h workers could still be overqualified.

In my mental model of this kind of development (which could be wrong), you'd
absolutely not want the 10x hotshot coder, you'd want predictable worker bees
to faithfully transcribe from one specification format to another
specification format slightly closer to the metal (or to a verification tool)
until you eventually have a product that is working as designed. And from what
I have read so far the MAX software does work as designed, the problem already
started with garbage in. The article reads like an attempt at shifting blame
that I think is entirely uncalled for.

~~~
salawat
>In my mental model of this kind of development (which could be wrong), you'd
absolutely not want the 10x hotshot coder, you'd want predictable worker bees
to faithfully transcribe from one specification format to another
specification format slightly closer to the metal (or to a verification tool)
until you eventually have a product that is working as designed.

Which gets you exactly what happened here. The 10x "hotshot coder" isn't a
coder who vomits out 10x more code than another, but the coder that knows _the
right code for the job_ and also understands _the provenance of from whence
the data operated on comes from, and to whence it goes_.

Software engineering and computer science is about _so much more than coding_.
It's about knowing the types of problems an implementation can solve, what
boundary conditions have to be taken into account, and knowing what code _isn
't the right code_. All for any subject matter area they end up operating in.

You'll be amazed the speed ups you get from not having to write ten different
versions of code implementing the same functionality before finding the one
that plays well and reads well with the rest of the system. All that extra
time spent _not coding_ can be spent going over the operating environment to
make sure the specification you were handed is actually _complete_.

Unfortunately, I have the feeling Boeing would have hidden that information
regardless seeing as their design doesn't even necessarily comply with 25.173
as written without interpreting it as allowing a computer to magically make an
unairworthy frame airworthy. But that is beside the point.

Once your management starts assuming the process will spit out product as high
quality without your expertise, you can pretty much set up a timer to measure
the time until the first impedance mismatch from a false assumption not rooted
out rears its ugly head.

------
inamberclad
Was just looking at their job listings recently - a lot of the software jobs
are in India. I don't know if they're flight software positions or just
database/website stuff, but it still surprised me.

On a side note, if anyone is looking for all-around engineers, including
software/electrical/mechanical stuff, feel free to shoot me a message.

~~~
tomcam
How? No contact information is given.

~~~
inamberclad
Right here, I suppose.

~~~
tomcam
The convention on Hacker News is to put your email address in your user
profile.

