Hacker News new | past | comments | ask | show | jobs | submit login
Boeing's 737 Max software outsourced to lower-paid engineers (bloomberg.com)
287 points by pseudolus 18 days ago | hide | past | web | favorite | 149 comments

"In offices across from Seattle’s Boeing Field, recent college graduates employed by the Indian software developer HCL Technologies Ltd. occupied several rows of desks, said Mark Rabin, a former Boeing software engineer who worked in a flight-test group that supported the Max.

The coders from HCL were typically designing to specifications set by Boeing. Still, “it was controversial because it was far less efficient than Boeing engineers just writing the code,” Rabin said. Frequently, he recalled, “it took many rounds going back and forth because the code was not done correctly.”

Boeing’s cultivation of Indian companies appeared to pay other dividends. In recent years, it has won several orders for Indian military and commercial aircraft, such as a $22 billion one in January 2017 to supply SpiceJet Ltd. That order included 100 737-Max 8 jets and represented Boeing’s largest order ever from an Indian airline, a coup in a country dominated by Airbus."

You have got it the other way around. TCS etc don't have that much power over Indian military contracts. Instead Boeing had to comply and open an Indian development center to win those contracts.

I talked to couple of people at the Indian center and they tell me because the center was created due a compulsion and not strategy the quality of work is terrible. Most of them are paid well but have no work to do.

How are the points you have mentioned in your post relevant to the MCAS fault, which the article is supposed to be about?

It is explicitly mentioned in the article that the Indian companies DID NOT work on the faulty MCAS system - Boeing said the company did not rely on engineers from HCL and Cyient for the Maneuvering Characteristics Augmentation System, which has been linked to the Lion Air crash last October and the Ethiopian Airlines disaster in March. The Chicago-based planemaker also said it didn’t rely on either firm for another software issue disclosed after the crashes: a cockpit warning light that wasn’t working for most buyers.

This looks like a misleading article trying to shift blame away from Boeing management. Yes, there are problems with outsourcing, which the article brings up trying to show as reason for the problems with 737 Max.

You are missing the point. This information is necessary in making the decision to continue flying on Boeing 737 Max Airplanes.

Money saved by firing experienced engs and outsourcing the development << money lost because they can't sell planes.

Why is that so hard to understand? Massive fail at risk management maybe?

For one, in part because many managers don't really understand software or software projects and think it's mostly just people * man-hours. "Through this cost-saving measure, we've tripled the amount of people working on the project and expect to reduce the amount of time it'll take to complete by two-thirds without incurring any additional costs." In their minds, it's an easy win.

And they were probably regularly reassured by various internal and external entities that the code was high-quality and thoroughly tested and very safe. Obviously they're not going to outsource if they think it's likely to cause a safety hazard; they just didn't place the odds of this outcome anywhere near what the actual odds ended up being.

Which means they have no business managing. It’s to tiresome to repeat because it should be obvious , but engineering is not a commodity.

The end product may be, but not the design problem process nor the manufacturing process.

For me the solution is quite easy: I don't invest in companies where I don't see technical excellence happening.

The world is changing fast enough that old style management won't get investment after some time.

someone needs to throw the mythical man month at these managers.

There would be no englightment happening.

Most managers only think short-term. In the short term it saved money. In the long term is cost Boeing their reputation.

But in the long term the MBA manager has been likely been promoted and is working elsewhere, killing another company, while his or her CV boasts of "reducing inefficiencies".

As long as they can get a RSU package and get a promotion it's s win.

The software did exactly as it's supposed to do: If the AOA sensors tells you the plane is tilting too high up, pitch the nose down a little.

The main issue was that there was only one AOA sensor connected in standard packages and the hardware for it failed.

The pitot tubes failed for the doomed Air France incident which caused the software to read incorrectly and stall.

For what it's worth AirBus has a huge engineering department in India: https://www.airbus.com/careers/search-and-apply/opportunitie...

Articles like this are just drummed up to stir negative emotions about "others" (with "struggling to understand English" and "unable to code fizzbuzz") as is evidenced in the rest of the comments below.

This comment is glib and misinformed.

Firstly, no the software did not do exactly as it's supposed to. It crashed the plane by rendering it unflyable. It's not supposed to do that. And your description of the MCAS specification is vastly over-simplified - it is not an autopilot and its job is to gradually change the behaviour of the plane in different flight modes (which it must detect), not to directly fly the plane as you suggest (and even two AOA sensors would be insufficient for the type of functionality you describe). The behaviour of the MCAS is absolutely in question, in particular the way it kept "re-adding" the correction, with cumulative effect, ending up fully pitched down.

Secondly, if by "the doomed Air France incident" you mean AF447, your description of it is flat-out wrong. The pilot, not software, stalled and crashed the plane by manually holding the stick back for 15 minutes. The pitot tubes briefly iced over (they didn't "fail") which is not an uncommon occurrence, and the only consequence of this was that the software guardrails that would have prevented the pilot from crashing the plane were removed. All instrumentation systems were fully operational at the time of the crash.

Agree with everything you wrote, with the possible exception of referring to the stick-holder in the AF447 crash as a "pilot".

> Firstly, no the software did not do exactly as it's supposed to. It crashed the plane by rendering it unflyable. It's not supposed to do that.

This belongs in the true-but-trite category. The decision to depend on a single, unverified AofA input was not a programming error, and neither was the increase in power in the second version. These decisions were made by Boeing, and endorsed by the FAA (to the extent that Boeing informed the FAA of them.)

It kept re-adding because the flap of the aoa sensor was stuck in the "aircraft is pitching up" position.

Boeing management did not add a second sensor to some planes and also pilots were not aware of how to switch off mcas.

If you worked in large scale project you would know that most devs are assigned components to work on and usually have no idea what other components are or how they will interact.

This is where testing comes in and from the fact packet it appears that the testing was also outsourced to another Indian firm?

> Articles like this are just drummed up to stir negative emotions about "others"

The negativity seems more targeted at the leaders who arranged this than at the developers. Both the examples you cite are from the same comment, and even that comment is more about resentment towards management corruption than about the incompetence of the developers hired as a result of the corruption.

> The software did exactly as it's supposed to do

I gather new issues with the software have come to light since the accidents, as a result of the heightened scrutiny, and they seem likely to keep the planes grounded for longer than Boeing hoped.

See the post below yours lol.

Not sure which one you were referring to. The ordering of comments changes over time. Could you post a URL or identifying substring?

That’s the joke... You can scroll up or down and find such a comment.

You sound suspiciously like every other outsourcer I’ve worked with who breaks something critical. A million reasons why it wasn’t their fault. They did exactly what they were told right and didn’t take one second to validate the assumptions surrounding the systematic nature of the solution. The goal, the mission, the end state solution. That was another departments concern right. That was QA’s concern, that was designs concern, that was the testing center of excellence’s concern. Blame them. The perfect circle of siloed fiefdoms and no feedback blame. The problem is that lives were at stake here on this one and the entire house of cards is coming crashing down if this is determined to be true. There will be public outcry and no matter how many excuses you have the fact is families and lives have been ruined FOREVER, NOTHING ON THIS EARTH CAN REPLACE THEM and something must be done.


I do think it’s weird that the parent post started out by saying the sensors were designed to spec, but is he wrong? The article even said as much. (Not MCAS though.) I didn’t read it as an excuse.

It seems like you’re saying that outsourced devs should have as much accountability as home-based devs or the entity Boeing itself. I don’t agree with this. The salary does not reflect that. The work that’s assigned does not reflect that. Anecdotally working with outsourced devs and based from the comments here, no one honestly thinks that.

It seems like you were triggered by the “excuses” part of the post. But what about the rest of it? He made a valid point (which I think you proved) about people using the article to discriminate.

I guess I’m just wondering why you’re making generalizations. I’ve worked with many devs (who weren’t outsourced) who blame everyone else but themselves. I learned to use ‘git blame’ very early on.

It’s a very interesting point. I’ve worked with some very skilled devs from HCL. I’ve also seen them turn over to other firms because of salary and other firm related issues. I don’t think that piece has anything or very little to do with the fact that they are Indian. However the organizational culture is definitely flawed and it does appear to be more prevalent in large fixed bid outsourcing arrangements. I’ve witnessed it first hand being brought in to resolve issues created by the outsourcer on projects as large as 1/2bn. They difference between those projects is the mission. Those project were critical on a financial basis. This was and aircraft with peoples life’s on the line. No one dev or otherwise can say they didn’t know this code was going on an airplane. No matter their salary.

Thanks. I thought your first post was very emotional and didn’t align with the truth of the situation, which is why I felt the need to reply.

You’re right in that the primary focus of the article was not the outsourced devs being Indian. The issue comes from how people respond to such articles, no matter the article’s intention.

I’m not saying that outsourced devs shouldn’t be held accountable for any faults. The point is that there are different degrees of accountability for all parties involved.

The article only chose to focus on one of these groups, which is fine. What’s not fine is HN comments that seem to not realize this.

No matter the salary or experience or contract, outsourced devs/companies shouldn’t be in a position of power to break critical systems on an airplane. (I’m not saying this is what happened with the Max.)

Could you please not cross into personal attack? and also please not post in the flamewar style? These are things we're trying to avoid on HN. The site guidelines explain, and if you'd read them and stick to the rules, we'd be grateful: https://news.ycombinator.com/newsguidelines.html.

In your world, Boeing designed (and the FAA endorsed) MCAS to use only one sensor and made it dangerously powerful, yet the crashes are the coders' fault for being the ones who did not anticipate the problems this would cause.

Experienced software engineer would NEVER code something like that. Both physically and logically it makes no sense to get same AOA reading after many iterations of trimming. That clearly means something is wrong. And probably the sanity of data being fed is in question. It’s the dumbest mistake in critical system I’ve ever heard. Even in non critical software these types of logical mistakes are rare.

AofA is a function of a number of factors, such as control inputs -- data not available to MCAS.

It would be disturbing if the coders on aircraft control systems were modifying the specifications on the fly, and on the basis of misconceptions like that displayed here.

Responsibility for the dangerous capabilities of MCAS lies with Boeing, which chose to fix an aerodynamic problem this way, and the FAA, which endorsed it.

The problem is that developers also need to understand what they are coding, and fight back when they notice that the design is fundamentally flawed.

In this case, the lack of redundancy should have been a red-flagged.

This is why domain knowledge is at least as important as coding skills.

In highly process-based environments, which outsourcing centers are, fighting back gets you harrased by the zealot managers out of the workplace.

There were multiple redundancy built in. Multiple sensors, voting mechanism, manual override. Many of which meant re-training the pilots and Boeing wanted to avoid it.

I think southwest airlines had multiple sensors

The 737 Max aircrafts that crashed relied on a single AOA sensor.

A second AOA sensor is an option.

Which means code was written to account for both configurations.

Many, many things must happen and change to avoid allowing management to make decisions unchecked.

Because this is a case of bargain-bin engineering and tunnel-visioned management ruining a once-reputable company.

Software was there when we went to the moon and back to ensure hardware failures are handled and communicated. Why can't it do the same for these planes?


Ask Boeing management your question wrt to the hardware failure. Not pile up on software devs

It’s the hardware people’s fault right. Oh no it’s managements fault. It’s everyone else’s fault. I see a pattern forming here.


I have seen this kind of problem before (never in a case where lives were on the line), and I called it "pathological optimism". If you are in a corporate culture of the sort that lionizes "can-do" thinking, then saying "that won't work" is going to look defeatist.

I have to wonder if seeing SV startups go to multi-billion dollar companies on a regular basis, has given CEOs in other industries a case of envy. But more likely, it's just that pathological optimism works, really well, right up until that moment when it doesn't. Taking risks in order to lower costs makes the bottom line better, which seems to call for more of the same, right up until the moment when the risk explodes in your face.

It is not unlike the attitude towards software security, in many industries. People harping about security risks seems like just all cost and no benefit, right up until the point where you have a big problem.

> I have to wonder if seeing SV startups go to multi-billion dollar companies on a regular basis, has given CEOs in other industries a case of envy.

If so, they're learning the wrong lessons. SV unicorns become valuable because of explosive growth, not because they underpay their engineers. Just the opposite; a lot of engineers at SV startups are paid quite handsomely (especially the early ones at successful unicorns). SV startups don't win on cost-cutting.

I think you are right on with this. Most startups also leveraging an organizational culture that goes against silos or fiefdoms. In the startup culture I’ve been a part of fast feedback and early identification of flaws in the process is a cornerstone of the development process. Troubleshooting issues in production is the worse case scenario. In this case production is literally a flying airplane.

The risk management is being TBTF. In a nationalist era, is the US going to buy planes from Airbus, or are they going to bail out Boeing?

For others wondering: TBTF = too big to fail

Can't sell planes, or fly the ones that exist, and pay out huge untold sums in death lawsuits.

Yeah, that too, after the fact seems so stupidly obvious.

Unfortunately it seems other critical industries are affected, reminded me of the EPR nuclear reactor fiasco https://en.wikipedia.org/wiki/EPR_(nuclear_reactor)#Olkiluot...

Olkiluoto (Finland)

> In May 2006, construction delays of about one year were announced, following quality control problems across the construction. In part the delays were due to the lack of oversight of subcontractors inexperienced in nuclear construction.

> At the end of June 2007, it was reported that Säteilyturvakeskus (STUK), the Finnish Radiation and Nuclear Safety Authority, had found a number of safety-related design and manufacturing 'deficiencies'.

> Initial cost estimates were about €3.7 billion, but the project has since seen several severe cost increments and delays, with latest cost estimates (from 2012) of more than €8 billion.

Flamanville (France)

> In April 2008, the French nuclear safety agency (Autorité de sûreté nucléaire, ASN) reported that a quarter of the welds inspected in the secondary containment steel liner are not in accordance with norms, and that cracks have been found in the concrete base.

> In April 2015, Areva informed the French nuclear regulator ASN that anomalies had been detected in the reactor vessel steel, causing "lower than expected mechanical toughness values". Further tests are underway.[64] In July 2015 The Daily Telegraph reported that Areva had been aware of this problem since 2006.[65] In June 2015, multiple faults in cooling system safety valves were discovered by ASN.

> Why is that so hard to understand? Massive fail at risk management maybe?

Managerial bonus or promotion for saving money this quarter >> money lost more quarters out than I'll be around

This is really it. Board of directors set goals, say reducing costs, they offer a big bonus, you meet the goal, get the big bonus. Problems become someone else's proble. You buy a big car, house, go to fancy non-profit's $500 dinners.

This is definitely one major issue. At the top. The challenge at the top is that these outsourcing contracts are for many millions of dollars. They are often written in such a way that the parent cannot dictate how the outsourcing firm conducts business. The parent fills out a ticket and the outsourcer assigns the ticket to a resource and marks the ticket complete. The issue here is that this is not a bug related to email or Login credentials. This is an airplane with folks relying on it for safety of flight.

The managers who saved money would have been promoted well before any accident happens.

>>"...recalled one manager saying at an all-hands meeting that Boeing didn’t need senior engineers because its products were mature. "

Whatever manager that was, should never be working in an aerospace context, where one design failure or fault can literally kill hundreds of people in minutes (and in this case, did exactly that).

Clearly, Boeing has become infested with people who literally do not understand the fundamental life-and-death consequences with which they are dealing, and who treat everything as fungible commodities. The entire story is essentially about how the outsourcing and commoditizing just expanded and expanded until now, when there were fatal consequences for the passengers, and perhaps even for the company (TBD).

Unless this changes, this is no longer a company who can be trusted with such life-and-death engineering decisions.

I’m not discounting the possibility of bad workmanship by Boeing. However “temporary workers making as little as $9 an hour to develop and test software” does sound to me like the reporter might have found some highly paid mechanical Turk style employees given the job of battering against a solidly built piece of software just because there would be a possibility of finding a problem.

If the sentence was “the mean salary of $9/hr” then I would be thinking differently.

The article mentioned recent grads (working in Seattle for an India-based company) who wrote code on specifications, not “mechanical turk”-style employees. It was followed by a quote from a former Boeing software engineer affirming this.

Is your issue just with the amount ($9) being paid?

I worked at a Fortune 10 in the early 2010s for several years. We had a small army of local and offshore Indian contractors through Larsen and Toubro Infotech, an Indian IT company. They were paid between $10-20 an hour, many couldn’t understand English. These guys have no CS background and would get stumped writing fizz buzz. But hey, management got ball game tickets and other favors from the L&TI reps. One of the directors admitted to getting a cut under the table after he left to go work for a competitor.

To be clear, we paid $120-150 per developer and L&TI paid them nothing. You’d see .NET code with a single class that has a few thousand lines of code, no concept of unit tests, integration tests, manual or automated acceptance tests, usability tests, etc.

Shelf life of most of this garbage was a few months if something ever did get delivered. This went on for years.

The whole industry is corrupt.

Why would anyone pay $120-$150 per hour for .net code in India? Most American firms pay $25-30 per hour to the Indian companies. Also, do airplanes run on .net?

>You’d see .NET code with a single class that has a few thousand lines of cod

If we are blaming entire nationalities, I have seen code written by an American. His idea of refactoring was copy paste the function, append a '2' to the function name and use it. There were at least a 50-100 functions like this in the project that had like 30-50 source files.

I wasn’t blaming a nationality. Not sure how you’re drawing that conclusion. There’s a legitimate problem of offshoring programming work to “coders” in India who produce garbage only because management gets kickbacks.

If one could improve on this process, it seems like a huge opportunity.

I have one person in our Mumbai office out of a 8 person team I manage. He had been absolutely fantastic with the work he does.

You must have been a pleasant employee to work with.

I sense sarcasm. Is it because I called out a legit issue? For my employer, it didn’t matter because IT had a reputation of delivering garbage and so our business partners weren’t shocked at what they delivered. If something like this happens at a Boeing (I don’t know if it is), then heads ought to roll and people should be going to jail.

Actually, he sounds very good. The biggest problem in corporate environments is the amount of BS being spouted and people going along with said BS. When someone is actually honest and confronting issues, problems get fixed before they blow up. That's good for everyone.

Of course generally the reason people go along with the BS is that either they're incompetent or mediocre and BS suits them just fine, or they've given up and are just happy to be swept along and collect a paycheck.

throwawayMUSE is clearly right. These firms are playing the same game as used car salesmen--selling you junk that won't make it off the lot, all for $9/hr. It's true even for non-critical .NET apps that the business is getting ripped off. In mission critical systems, it's literally life and death. Sounding the alarm doesn't warrant dismissive sarcasm.

That Indian company is HCL. Ive hired and fired people that have worked in HCL here in Mexico. The culture seems veeery bad, as engineers coming from there have a culture of "do the minimum". And no ownership at all.

As an extreme example, one ex-employee stopped working one day because his wireless keyboard did not have batteries (a kB connected to his work MacBook Pro). Colleagues told him to just buy some AA batteries while I was back (I would have refunded them to him) and he did not want to do it.

Nowadays if I see HCL or Tata in resumes here in GDL, it is an automatic pass.

I think this is just rabble-rousing. For all the flack outsourcing gets, if you hire an incompetent person, and they fuck up, you're still responsible for hiring an unqualified person. I don't think it matters if that person was outsourced or not. I haven't worked with any offshore teams, but friends tell me you can do what are called "client interviews" where the client, in this case Boeing, has the option of interviewing the actual people who will be assigned to work on your project.

"but friends tell me you can do what are called "client interviews" where the client, in this case Boeing, has the option of interviewing the actual people who will be assigned to work on your project."

LOL! Ahem. You'll discover that they get someone competent in for the phone interview and you'll later discover that the story is now that the person you spoke to was representative - in terms of skills, experience and competence - of the sort of person who'll actually doing the work, and is not the one doing it. Later still you'll discover that this is not the case.

Still, they're very cheap, aren't they?

I don't like Indian IT companies because they pay us less and some managers exploit us a lot. But one thing they deeply care about is the customer. They would go to any length to keep the customer happy. There is no way they would risk loosing business playing such games.

If they're in offices right across from Boeing Field, you'd think that Boeing might be able to do actual face-to-face interviews with the real (not just representative) people. You also might think that anything less is definitively a management fail.

This is what typically happens to h1bs: if someone comes from India to work at the client location (Boeing office in this case), and if the customer is not happy with them for whatever reason, they get send back within a month or 2.

"client interviews" don't happen for such massive projects. Most of the time the "host" even lacks the technical capability to interview and it's time consuming.

In all my years consulting for the companies I have never been "client interviewed"

The article you just read told you that the teams working on critical software have to be cheap, not good. That's not "just rabble-rousing." That's a scandal.

As you say, you're commenting without having any experience with the issue, but, as you can imagine, there's very little overlap between cheap and good.

Actually, the article just broadly mentioned that they worked on some aspect of the software. There is no evidence (in the article) that they were working on critical software, and it was their piece of software that caused the current problem. You can't blame the notepad team because the windows kernel crashed.

In any case, even if we accept your premise, my point was that assigning blame to the person you hired rather than your process of hiring that let a large team of unqualified people work on your critical software is odd. (Not saying you personally are blaming them, but that was my general point)

>but, as you can imagine, there's very little overlap between cheap and good.

Why? Even outside of software, for e.g. my wife is a post doctoral scientist and she gets paid a pittance for working 9 sometimes 10 hour days doing critical research. Somebodys wage being associated with the quality of their output only happens in a small cross-section of industries. I work in pharma and while I'm happy with my compensation, its a fraction of SV startup salaries. I know.. this might come across as super salty, but I don't think their wage is in anyway co-related with the quality of their output.

Absolutely rabble rousing. Given that it was the angle of attack hardware sensor which was stuck in the wrong position and giving out the wrong reading.

The software got the reading that the nose was too pitched up and to prevent stall it pitched the nose down.

The software could handle reading from multiple sensor which Boeing sold separately.

It's an American management decision to make more money, not a $9 an hour software engineer's job to decide how to fly the plane.

Coming up after the break: How EA loot boxes are programmed by $10 an hour cheap software engineers from India. Grab your pitchforks on aisle #3

"Rabin, the former software engineer, recalled one manager saying at an all-hands meeting that Boeing didn’t need senior engineers because its products were mature."

Boeing is dead to me.

Understand the sentiment, but can anyone be sure that Airbus isn't doing something similar.

Irrelevant. If Airbus is doing the same, then they are also dead to me.

Dead meaning you won't fly any more? (beneficial for the environment)

Dead meaning that, in my opinion, they have sold their soul to Satan. Boeing was previously at the apex of engineering. The 747 was a respectable achievement in 1968. The MAX debacle has exposed a catastrophic fall in engineering standards.

Er, that does not leave too many options if you wish to fly...

Yes it means I would be forced to fly on a shitty plane built by junior engineers. Living life in the danger zone.

Interesting - a completely hypothetical whataboutism

LOL - yeah what about China? If they were making commercial aircraft they might be doing the same thing!

They are, such as the C919 and ARJ21.

With Boeing trashing its reputation and Airbus sitting on an enormous backlog I can envisage Chinese airliners making inroads in Africa and Asia. But to do so they need to make solid, dependable products; they can't scrimp or cut corners in pursuit of short-term profit.

Really wonder how deep this rabbit hole goes. I could live with the mcas drama but these articles hinting at seemingly pervasive internal culture issue makes me wonder whether I want to be on any Boeing jet at all

When is the media going to get the message that this wasn't a software issue?

It will eventually fly with only software fixes, which will reinforce that picture.

The final verdict isn't in, but it appears to have been a systems engineering problem with a software component.

The software performed exactly as designed. The problem is a faulty spec for how it should behave. The software is pretty much incidental.

Software is an integral part of modern flight. Somewhere up the chain, there’s a highly-paid software engineer/manager who helped design and signed off on the spec.

Would a highly paid software engineer be expected to understand the potential consequences of runaway trim? I’d think that would be up to the aerodynamics folks, who most likely said that it was not a problem.

You are correct, however... I can't imagine a single engineer with any kind of proper education (formal or not) implementing any piece of software or hardware without thinking:"One sensor? Really? What's the failure rate on this thing". That might have been enough to stop the chan of events.

The engineers may not have realized they were relying on one sensor, or they might be the $9/hour variety with no avionics experience who don't know any better.

The main problem was higher up. The overall design was bad, and Boeing didn't seem to have a handle on it.

Or they have a "obey all orders even when it is blatantly criminal" clause.

I suspect they were told that runaway trim wouldn’t be a serious problem, and they believed it since they wouldn’t have the expertise to analyze the whole thing from first principles.

Or may be they pointed it out but got reply "just work as said, you're just $9 engineer"? Or they were never told about the exact purpose?

You get $9 for hypothetical blood splattered on your hand or you get worse than nothing for nothing

If the issue was in the specs, then it is software. Producing code is only really one piece of software development.

Software was definitely part of the issue.

But it wasn't a software issue! /S https://philip.greenspun.com/blog/2019/04/08/boeing-737-max-...

Any competent engineer would raise moral and ethical questions when working on software that is "mission ciritical". Causing a passenger plane to continually go into a nose dive because of data coming from a single sensor is absolutely ludicrous. Just because the software was horribly designed, doesn't mean it isn't a software issue.

It is a hardware design issue. There's a concept of N+1 redundancy in safety-critical systems, a hardware design decision. This basic philosophy was missing from the MCAS system[1]. You can argue that the software could just ignore bad sensor readings, but that's just software making up for hardware deficiencies.

1. https://www.seattletimes.com/business/boeing-aerospace/a-lac...

But but but... there's a computer! It has to be a software issue!


Why worry about meddling of elections by Russians when large American companies that make life/death products and one of the pillars of US economy goes around doing stuff like this, effectively committing a corporate seppuku.

I used to watch PR videos from Boeing on new planes being tested and feeling sense of wonder and awe. Now I feel like puking. Of course not the rank and file. But the ones in corner offices.

These low paid engineers have short tenure (high turnover) and graduate to higher paying jobs. That is their goal and companies know that. So, in the end they are honing their skills on projects where hundreds of peoples lives are at stake. It used to be the design built by them and quality checks completed by experienced engineers. In the quest for more profits (narrow margins), they forgo this valuable process step. Most 'business' executives have no clue and never will and more don't care. Be honest.

I know which is worse: it's not 737 NG's Ducommun's substandard critical structural parts, underpaid subcontractor software engineers in India or Boeing monetizing safety as optional extras... it's the systematic, systemic corruption of safety regulatory protections capture allowed this to happen. Planes didn't fall out of the sky 30 years ago because enforcement entities actually had resources and tried to do their jobs.

The inescapable problem is that all newer planes from Boeing and likely other manufacturers are suspect in uncountably tens of thousands of ways because they cannot be presently assured (without engineering reviews, inspections and documentation) to be fit for passenger travel due to the lackadaisical regulatory climate in which they were developed and manufactured. The 787, 737 NG (-600 and up) and 737 MAX are just the tips of the known problems. And now Boeing wants a folding wing plane too?

Not trying to defend them, but this was like every company from 2002-2012 until everyone collectively realized the asset is the people not the code.

I’m reminded of an article many years ago about when Boeing was outsourcing to to improve return on capital or some nonsense and at some point someone just said to maximize return on capital they should just have someone else design and make the planes and put a Boeing sticker on them

Run an aerospace franchise. Let other design and produce the aircrafts. Wet dream of cost cutting rent seeking managers.

I am so tired of software engineers that, patch that software, etc etc. Ultimately it all comes down to $, $ that executives thought they could save for the company AND make that yearend bonus.

It's not engineer that or this. It is MBA this and MBA that.

I really doubt 737 Max's failure has anything to with outsourcing to India. Usually, most of the work outsourced to India are low value work (what I call as 'near brain dead' type). I don't think Boeing would outsource anything even remotely critical to a company like HCL. From the article:

>Based on resumes posted on social media, HCL engineers helped develop and test the Max’s flight-display software, while employees from another Indian company, Cyient Ltd., handled software for flight-test equipment.

>Boeing said the company did not rely on engineers from HCL and Cyient for the Maneuvering Characteristics Augmentation System, which has been linked to the Lion Air crash last October and the Ethiopian Airlines disaster in March. The Chicago-based planemaker also said it didn’t rely on either firm for another software issue disclosed after the crashes: a cockpit warning light that wasn’t working for most buyers.

Even inside US, Indian engineers are not allowed in critical software that goes into defense and (most likely) airlines, only citizens can do that. So, there is no way they would have outsourced safety critical stuff to HCL.

For DOD no foreign you are correct. Unfortunately 737 max is not a no foreign DOD project. Please understand these outsourcing companies are extremely powerful and influential. They bring In highly trained and connected sales teams to negotiate big deals at the executive and political levels. In this case 11bn deal to Boeing for 1.7bn in services contracts for India. Please consider what level at Boeing that sort of deal is brokered at? Ask yourself what the repercussions are for internal folks who call out the outsourcing firms? As you suggest outsourcing firms often start out with non critical tasks like operations and maintenance. Through this process they actually consider those contracts as investments to gain access to learn the lay of the land especially where the critical systems are and who manages them. Then these firms often including architecture leads who are camouflaged sales people do their trade craft of above board and from my perspective suspicious below board tactics until they open the door to more critical projects and higher paying contracts. HCL, TCS, etc work on critical system not only in this industry but also in Medical, Infrastructure, Utilities, Telecommunications, Broadcasting, etc, etc. I wish you were correct but sadly I’ve seen the transition first hand and have been brought in to unwind what outsourcing firms have done in the name of cost savings. At the same time I was told and my teams were told to investigate, identify and fix the issue but don’t blame the outsourcer or this will be your last contract. This i was told was due to the level of the outsourcers executive Sponsership. Consider the influence these firms have over the execs and the reputation at stake with anything negative or other then thank you so much for the great work. In fact we were told that the outsourcer was to be treated as the customer. Even though months of war rooms and rework fell squarely on their ineptitude.

> Boeing said the company did not rely on engineers from HCL and Cyient for the Maneuvering Characteristics Augmentation System, which has been linked to the Lion Air crash last October and the Ethiopian Airlines disaster in March.

That may well be, but that constant supervision and error correction takes a lot of your experienced engineers attention away from the important things.

Assuming same engineers were doing the constant supervision and error correction and not few dedicated ones. (My previous company had one guy whose job it was to do this - that got good results because the guy knew what to look for and how to get it fixed.)

How in the world does one hire $9 an hour software engineers?

Without wanting to sound disrespectful, most of the engineers outside US and some EU countries, get paid a lot less from what you’re used to and from what you read in major tech sites. I know a lot of good engineers in many countries (Greece, Croatia, south-east EU, India...) that get paid less than 9$/hr (myself included). The bad thing with all the big companies outsourcing positions and working with external partners is that it’s difficult to distinguish the good engineers from the copy-paste developers, self-taught coders and really bad engineers that give some countries a bad name. I have first hand experience of really bad engineers, working for top companies, who get paid low wages and don’t care about the quality of their work. I guess that’s how the system works...

> self-taught coders

That need not be a disadvantage, I've seen awesome self-taught coders (and really bad ones too).

> Still, for the 787, HCL gave Boeing a remarkable price – free, according to Sam Swaro, an associate vice president who pitched HCL’s services at a San Diego conference sponsored by Avionics International magazine in June. He said the company took no up-front payments on the 787 and only started collecting payments based on sales years later, an “innovative business model” he offered to extend to others in the industry.

How is that allowed? It is like basically the equivalent of dumping products below market and production price like what China frequently does to steel.

Why wouldn't this be allowed? This is the company equivalent of a person working on commission vs salary. Salary is more stable and predictable, commission is riskier but has higher potential for upside.

it's also the equivalent of what they call "royalties" in the music industry.

It’s more like becoming a partner or working for shares. Rather than getting a fixed price up front you take a (presumably larger) payment at a later date but accept the risk that there may be no payment.

You just described Uber.

How common is this, payment based on sales, and years later!? I guess you only do that with established companies? Still seems like a risk.

I'm positive they are more "developer" than "engineer".

9hr = 72 day = 1440 month = $28800 MXN a month. Buys you 1 software engineer between Jr and mid level here in Mexico my

$9 an hour works out to about 100,000 rupees a month, which allows one to live relatively comfortably in India.

India. Billing rate is around $30

Go to India?

That's usually the case, though in my experience from doing a bunch of re-writes of code outsourced to India - you really do get what you pay for.

Had to run an outsourced team out of India a few years back. Had to sign off on the time / pay paperwork. I believe their Senior-level folks were costing us $20 - $25 per hour. Not sure what they made after the agency took their cut. $9 sounds like very Junior-level staffing.

As for code quality, this team was working on a Spring / Java application. They were instructed to use Spring JDBC templates instead of Hibernate. So of course they build their own query-builder that was just concatenating strings. Naturally we paid them to clean up their own mess.

> Naturally we paid them to clean up their own mess.

Heh, basically funding external developers too learn without the benefit of retaining them then.

You don't actually save a whole lot of money by outsourcing. "Engineers in India made around $5 an hour; it’s now $9 or $10, compared with $35 to $40 for those in the U.S. on an H1B visa, he said. But he’d tell clients the cheaper hourly wage equated to more like $80 because of the need for supervision, and he said his firm won back some business to fix mistakes"

Yet another shocking revelation in how little Boeing values lives when it comes to the bottom line.

This should be a company-ending event for Boeing. These folks should not be making aircraft used by living, breathing human beings.

The stock holders Boeing is beholden to, that clamor for this cost cutting, should pay.

The stockholders are you and I, via retirement plans.

Our investment portfolios and retirement plans can be rebalanced. Those that died in the two Max crashes are gone forever.

This isn't cheating on emissions. We have to send a strong message.

If you mean just lose money when you say the stockholders should pay, then yeah, whatever. But if you mean that they should be held responsible, I think that’s absurd. That’s what I meant by my response.

Losing 1000 people is a tragedy, and those responsible should be held responsible. But a company like Boeing is a huge machine with many people working diligently on extremely complicated products, the vast majority of whom didn’t touch this faulty system at all. Which is to say you shouldn’t throw the baby out with the bath water.

"These are the rates in India and Eastern Europe, we're paying well above the market actually, so we are getting crème de la crème" - some directors and managers at Boeing, probably.

I wonder if this is unchecked laissez faire capitalism at work? Nobody wants to fly on Boeing 737 Max anymore but this has cost hundreds of lives to get to this point.

there's very little "laissez-faire" about the relationship between Boeing and the US government.

There was definitely regulatory capture which led to lax FAA oversight, though. Maybe a free-market fundamentalist would argue that they would have done better if they weren't dragged down by all that red tape, but it suggests to me that a looser regulatory environment was more dangerous.

There are some Boeing executives who should be making $0.35 an hour in prison.

I don't even want them in or near the prison. I just want the salary/bonus they made while employed by Boeing, making the decisions that led to this.

And give them to the family of the victims, ON TOP of whatever punitive damage Boeing will be paying.

Holy crappus, what was Boeing thinking?

Probably, but not likely to happen.

I guarantee you a lot of the legally binding design and testing documents were signed by engineers, maybe some product manager types. A few QA folks? But that's the extent of it as far as people who have the clear cut legal liability. You want to get anyone else? You're likely looking at a massive investigation, and spending years or even decades looking for incriminating documents or emails. And unless I miss my guess, the government is just not terribly likely to find any.

Maybe so, but please don't post unsubstantive comments to Hacker News.

Is that so surprising? I have never worked in a domain like that, but I have always assumed that flight computer software would be built with process so deep that you could almost leave the implementation to trained monkeys, implying that those $9/h workers could still be overqualified.

In my mental model of this kind of development (which could be wrong), you'd absolutely not want the 10x hotshot coder, you'd want predictable worker bees to faithfully transcribe from one specification format to another specification format slightly closer to the metal (or to a verification tool) until you eventually have a product that is working as designed. And from what I have read so far the MAX software does work as designed, the problem already started with garbage in. The article reads like an attempt at shifting blame that I think is entirely uncalled for.

>In my mental model of this kind of development (which could be wrong), you'd absolutely not want the 10x hotshot coder, you'd want predictable worker bees to faithfully transcribe from one specification format to another specification format slightly closer to the metal (or to a verification tool) until you eventually have a product that is working as designed.

Which gets you exactly what happened here. The 10x "hotshot coder" isn't a coder who vomits out 10x more code than another, but the coder that knows the right code for the job and also understands the provenance of from whence the data operated on comes from, and to whence it goes.

Software engineering and computer science is about so much more than coding. It's about knowing the types of problems an implementation can solve, what boundary conditions have to be taken into account, and knowing what code isn't the right code. All for any subject matter area they end up operating in.

You'll be amazed the speed ups you get from not having to write ten different versions of code implementing the same functionality before finding the one that plays well and reads well with the rest of the system. All that extra time spent not coding can be spent going over the operating environment to make sure the specification you were handed is actually complete.

Unfortunately, I have the feeling Boeing would have hidden that information regardless seeing as their design doesn't even necessarily comply with 25.173 as written without interpreting it as allowing a computer to magically make an unairworthy frame airworthy. But that is beside the point.

Once your management starts assuming the process will spit out product as high quality without your expertise, you can pretty much set up a timer to measure the time until the first impedance mismatch from a false assumption not rooted out rears its ugly head.

Exactly what salawat said. The 10x coder isn't the "hotshot" type. They're 10x because they build systems well enough to deliver 10x the features, 10x the reliability, etc., etc.

What isn't well understood is that software is never fully spec-ed out. No matter how hard some businesses may try, only the coder is in the position to find corner cases, unspecified cases, cases that "shouldn't happen" (but can), and just bugs in the spec. It's unrealistic to build a quality system without a feedback loop between the people writing the spec and the people coding the spec. Both sides have to be smart and engaged.

There's certainly nothing wrong with the article. I didn't see it taking a point-of-view on who is and isn't to blame. It's telling factual information, which is relevant to the story of what happened, and you can make your own judgments.

Was just looking at their job listings recently - a lot of the software jobs are in India. I don't know if they're flight software positions or just database/website stuff, but it still surprised me.

On a side note, if anyone is looking for all-around engineers, including software/electrical/mechanical stuff, feel free to shoot me a message.

How? No contact information is given.

Right here, I suppose.

The convention on Hacker News is to put your email address in your user profile.

If you can do low-level software, particularly assembly, sure:


Registration is open for Startup School 2019. Classes start July 22nd.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact