The article seems to conflate engineering salaries in a low cost of living country with the quality of the software.
As I understand it, the software performed exactly as designed, forcing the nose of the plane downwards to counter the upwards torque produced by the engines being offset relative to the centre of gravity of the plane.
Mentour Pilot had a great episode on the two tragic accidents. It appears that Boeing did not expose the way the software was designed to behave, and the system silently turned itself back on even after the pilots disengaged it.
My experience is that contract work like this ends up being done by ordertakers; the contractors execute the design, no matter how stupid.
Whereas, the best performing companies hire engineers who are empowered to understand the goals, understand the problem, and push back when appropriate.
The results are much better, because the people working on each system, understand it deeply and care.
A low-cost contractor isn’t allowed to care. They aren’t even allowed to talk to somebody who matters.
The theme throughout has been cost cutting and poor oversight for the sole benefit of shareholder profits.
A proper in house team would probably say, hey what happens when the sole sensor feeding our software is obstructed by a birthday balloon? Maybe we need an exponentially backoff instead of every 30 seconds. Maybe nose down actions shouldn't be automatically executed at low altitude.
What we see playing out is the result of delegation and "not my problem", outsourcing safety critical systems to the lowest bidder...
I understand that it may be more likely for this to happen with an in-house team, but my experience working in large companies makes me think it would still be unlikely. "Not my problem" is necessary armor at a company with 15 levels of management between ICs and the C-suite and a rich weft of (official and unofficial) "dotted lines", where every interaction outside your immediate team is fraught with political implications. Code monkeys, in-house or not, might not even know that they are working on software for an airplane, let alone a Boeing airplane, let alone in a safety-sensitive system. Even if they do, and understand that the overall plan they are contributing to is folly, there is often no linkage by which they can make that insight known to someone who can change things (often there is literally no one, including the CEO, who could change course even if they want to). Even trying to do so is often a career risk.
Large companies are mechanisms that run on their own. People have limited agency in a lot of situations but for big decisions ("our company's only real product is fundamentally dogshit, lets start over") there literally are no people on earth who can actually make them. They are pre-made by the structure of the company, capital markets, and historical accident.
Yes. It happens. In a recent news clip a pilot was interviewed about the MAX and he raised the risk of bird strike and birthday balloons as something that “happens all the time”.
GP is (quite reasonably) using a deliberately slightly silly example as a rhetorical flourish. I would do the same. Think of it of a stand-in or totem for all of the possibilities (from silly to very real) that leap into horrifying life in an engineer's brain when they learn that a critical system has a single point of failure.
The biggest problem with offshore engineering teams is their dedication to delivering exactly what you ask for. One of the qualities of a good software developer is to take the requirements and interpret what needs to happen then deliver appropriately. Delivering exactly what's written without thinking about the implications is a huge problem and one of the biggest issues I've run into working with offshore teams.
I agree with you on the problem with offshoring, having done this myself in a previous role in a regulated industry.
However, the offshore team are usually not intended to act as domain experts. In fact, they were very likely explicitly proscribed from interpreting the specifications handed to them to guard against them trying to act as domain experts and delivering something different from expectations.
As such, they were (likely) not the ones who specified that MCAS should silently turn itself on after a pilot turned it off. That misjudgment probably lies with the engineering team who made that design decision, and it had tragic results.
> However, the offshore team are usually not intended to act as domain experts. In fact, they were very likely explicitly proscribed from interpreting the specifications handed to them to guard against them trying to act as domain experts and delivering something different from expectations.
Finally! Thank you for stating this explicitly.
In safety critical systems specifications are everything and is always done by a team of domain experts to an enormous amount of detail (with optional formal methods verification). The actual coder has to just use the chosen implementation language carefully to meet the specification with no personal interpretations; everything has to be explicit and documented.
And yet, the specification is not what executes. If the code can be written by a domain expert, why compromise? If nothing else, more time spent working on the problem by a domain expert gives more opportunity to identify deficiencies in the specification, or the ability to meet the specification within the constraints of the actual execution environment.
Never mind that this idea of an offshore team diligently implementing the spec to the letter hand-waves away the software engineering, as if it’s a mere implementation detail not intimately connected to the system delivering the desired safety and performance characteristics.
>more time spent working on the problem by a domain expert gives more opportunity to identify deficiencies in the specification
So much this. This is happening in other industries, too. Domain experts are also becoming less expert because of lack of proper feedback about their designs.
It's great to have a spec, but you should understand what's happening with the spec and how it applies to the actual code that executes.
With software we can get software that is very close to being an executable spec. By adding a layer of indirection we make everything harder.
Developing Safety Critical Systems is nothing like developing "ordinary" systems. The stakes are so much higher that absolute rigour at every step is demanded and enforced. This is why "Software Engineering Process" methodologies were invented and enforced using "Formal Methods". When this rigour is lost you have catastrophic system failure like in the "current-day" Boeing company.
> Never mind that this idea of an offshore team diligently implementing the spec to the letter hand-waves away the software engineering, as if it’s a mere implementation detail not intimately connected to the system delivering the desired safety and performance characteristics.
If the "offshore" team does not have the requisite Domain Expertise (which seems to be the case here), then it is Boeing's job to provide rigorous specifications and more importantly have safety checks/verification/tests/etc. in place to guarantee "correctness to spec." Problems in the specifications itself are the responsibility of the Boeing Design/Engineering team.
Domain Expertise is an absolute necessity to come up with Rigorous Specifications and in particular; for Safety Critical Systems.
If they can be done by a single person (whatever be his role name) all the more better but usually for Safety Critical (and highly specialized) Systems that is not possible and hence you need two (or more) people.
Coming from one of those countries where a lot of engineering gets offshored to, the best and brightest engineers in those countries DO have options (no, not just immigrating to a rich country) and won't be a code monkey for multinationals.
I have been to IBM offices in São Paulo a few times and let me just say, although they paid reasonably okay salaries it wasn't the first choice of work for most engineers...
And that is on top of the problems you are describing.
Now imagine a constraint written in management legales. "The system has to always augment flight characteristics to avoid system envelope violations. Should the pilot choose to deactivate the system temporarily, the system must return to enhanced flight characteristics as soon as the pilot stops signaling the intent to fly without the system. "
> As I understand it, the software performed exactly as designed
This is very likely correct.
People on HN don't seem to understand how software is created in regulated industries outside the tech bubble.
In a complex system such an aircraft, the behavior of the system is modeled and detailed requirements are generated.
These models are created by the system engineers.
The models and requirements are then handed off to the software engineers to implement the modeled behaviour in code.
The software is then tested to see if the behaviour matches the models.
So it doesn't matter how much the SWEs were paid, if the software met the requirements, and implemented the models as designed, then the software engineers did their jobs.
The software engineers job is also to think about what happens in corner cases.
What if sensor malfunctions?, how should software act?, will it crash the plane?
If yes, then raise problem and refuse to implement / avoid killing people.
Then you have QA which should have the same questions and test plan to make sure corner cases are covered. QA is also responsible.
Then you have management which is supposed to make sure there are people skilled enough doing the above and the initial systems parts and encourage this environment.
Then you have business people to ensure sufficient resources to hire competent people above.
So what is happening is a massive problem everywhere...
> If yes, then raise problem and refuse to implement / avoid killing people.
Often, software engineers are pipped, written up, berated, stack ranked - and churned via various management processes for doing exactly as asked for.
Until the culture of the company aka ALL levels of management changes to full accountability without blaming the actual engineers, nothing will change. The current corporate structure just isn't set up to incentivize good work.
Perhaps software engineers looking for accountability would like the same level of liability that is enjoyed by their peers in civil, mechanical, mining, and electrical engineering?
> Perhaps software engineers looking for accountability would like the same level of liability that is enjoyed by their peers in civil, mechanical, mining, and electrical engineering?
If it comes with the ability to say NO, or DELAYED, without management song and dance, it will get traction.
Problem is software engineers are replaced with juniors/contractors that probably do not know any better and are only one part of a system that has failed in its entirety.
What sane person would willingly put up code that can kill others on purpose... so we're left with incompetence.
Accountability, if implemented, should be company wide.
Right now issues have passed design stage, implementation, qa... with this sort of failure you have management to blame for not putting up processes in place so its caught / hiring and keeping competent enough people.
This is sort of expected as they optimized cost to the detriment of everything else.
Not that you intimated this, but I don't know that it's necessarily a problem with the people writing the code. What I've seen is that organizations that try to save money by offshoring are not providing adequate domain knowledge or usable requirements to the developers.
Throw in a massive timezone difference and every misunderstanding winds up living in your system.
Culture issues do exist. IME different cultures have very different approaches to communicating e.g. “I don’t know” or “I made a mistake”. Not everyone in a particular culture acts the same, but it definitely affects the average outcome.
That could be. However I think the language barrier is also a significant factor. But of course there is nothing inherently problematic about offshore developers.
I've had to deal with offshores my entire career and have even flown out to train them up. They are good people, but almost universally, they contract for U.S. companies via large body shops (you know who they are) that have awful pay, poor treatment of developers, and just a general culture of half-assing work. I've told various great offshore developers personally to leave as fast as possible and join literally any other organization for way more pay. Good developers have zero business working at those body shops that all the U.S. corps utilize.
Yep, the problem with the software wasn't the implementation, it was the spec: to operate contrary to the expectations and experience of pilots.
The problem was that they didn't outsource the design spec. I doubt an externally developed design would be so contrary, with a flippant we'll document it and include it in the short brush up training.
The umpteenth comment trilling on as though MCAS was built by contractors when it was explicitly stated in the article that it was developed in-house by Boeing employees.
I think you are comparing apples and oranges here.
An individual contributor is typically given a tangible task (eg: deliver a product that does x in y months for a cost of z). There are several qualified people they could choose from to achieve this task, which drives down the cost. I worked for a company with 10,000 engineers. Oversimplifying for effect, but their skills are not unique.
A leader is given a far more intangible task (eg: gain 30% market share in z years for your division OR deliver 9M of product per day through 202X), and they are taking a huge risk. The team under them may not have the slightest clue how to do this. The interest rates can change. Currency fluctuates. They are paid to calmly chart a path through the chaos and deliver a result. At this level, there were about 60 of us, and the atmosphere gets rarer as you rise further in the ranks. They made absolutely certain we were kept super happy because we could create tangible value orders of magnitude above the average employee, and they REALLY did not want us working for a competitor.
As I understand it, the software performed exactly as designed, forcing the nose of the plane downwards to counter the upwards torque produced by the engines being offset relative to the centre of gravity of the plane.
Mentour Pilot had a great episode on the two tragic accidents. It appears that Boeing did not expose the way the software was designed to behave, and the system silently turned itself back on even after the pilots disengaged it.