
Boeing seeking to reduce scope, duration of physical tests for new aircraft - pseudolus
https://www.reuters.com/article/us-france-airshow-boeing-certification-e/exclusive-boeing-seeking-to-reduce-scope-duration-of-some-physical-tests-for-new-aircraft-sources-idUSKCN1TH0A3
======
torpfactory
In my experience as a mechanical designer physical testing, especially system
level integrated testing, is the only way to find errors in the “unknown
unknown” category. In a complex system (mechanical, electrical, software,
biological, basically all of them), there usually exist interactions that are
very difficult to predict. Software-based simulation can help, but is only as
good as the models which are used to describe the underlying physics. In my
experience, these models aren’t nearly good enough, and without testing, you
can’t even say what is wrong with them.

Full-up testing isn’t always possible, but it shouldn’t be abandoned only
because it is costly or difficult.

~~~
im_down_w_otp
One of the things I continue to find disturbing when interacting with the
markets we work in (e.g. self-driving & other highly-automated systems) is the
growing hope that simulation will somehow bail people out of having to do real
world validation, let them take significant shortcuts, or make assurance
claims that can't really be made.

We deal in formal verification for some aspects of what we do, and that lead
to a conversation that went roughly like this:

AV Exec: "Can we use formal methods to prove that our simulator is as good as
reality?"

Me: "No. You can use formal methods to prove your simulator implementation
more closely adheres to your simulation model."

AV Exec: "Isn't that the same thing?"

Me: "No. Your simulation model is definitely wrong."

AV Exec: (looking at me both disappointed and despondent)

Me: "Look. It's important to understand that simulation isn't creating an
approximation with a known correspondence to reality. Simulation is
fabricating an entirely new reality with an opaque correlation to our
reality."

AV Exec: "Then how am I supposed to use simulation to prove our system is
safe?"

Me: "You can't."

Despair on both sides of the table ensues.

~~~
microcolonel
I'm lucky that I'm generally met with people who trust me when I say the
simulation is not the reality.

------
samcday
Here's the thing. I don't doubt that since the controversies with the MAX
earlier this year, a great deal many people at Boeing are taking a hard look
at everything around them and figuring out how to do better.

I'm also sure that this article could be embellishing the facts. It's entirely
possible that the very smart people who think about aviation engineering all
day are supremely convinced that some of the digital tests can absolutely
replace physical tests that were in place for decades. I'm not even remotely
qualified enough to offer an opinion on that, so I won't.

So here is my question. Boeing is a very large company beholden to
shareholders, and kept in check by a declining number of other checks and
balances. Should we, the public, trust that Boeing will pay close enough
attention to maintaining safety as an utmost priority? Or should we be
demanding that more oversight and regulation are put in place?

~~~
AnthonyMouse
> It's entirely possible that the very smart people who think about aviation
> engineering all day are supremely convinced that some of the digital tests
> can absolutely replace physical tests that were in place for decades. I'm
> not even remotely qualified enough to offer an opinion on that, so I won't.

This is really the foundation of the problem. Members of Congress are not
aeronautical engineers, so they're in the same boat as you. Aeronautical
engineers work for aircraft makers, so they have a conflict of interest.

Then, because they don't really know what they're doing, they tend to require
things that _reduce_ safety, either by requiring measures that aren't cost
effective and thereby blow your entire budget including the money that could
have been spent on other safety measures with better cost/benefit, or are
overly rigid in requiring the specific solution to a problem which was the
state of the art three decades ago when the rule was enacted even though safer
alternatives are known today.

So why doesn't the government higher some regulators with some relevant
industry experience? Well, they tried that, and then we got all of this
revolving door nonsense where, to use an example closer to home here, former
Verizon lawyer Ajit Pai is now running the FCC as a wholly owned subsidiary of
Verizon Communications.

Then we get proposals to prevent regulators from going back to industry when
they finish, to try to prevent that. But the government _already_ has trouble
attracting talent when they're paying substantially less than private industry
does, and the jobs tend to only last for one administration until the next one
comes in and replaces them with their own people, so how are they ever going
to get anyone good to do a job that will a) pay less than they're making
already and b) by law end their private career even though their tenure in
government is likely to be less than a decade?

It's possible that we're better off not specifying _how_ to make airplanes
safe but instead imposing significant liability on companies and individual
engineers who make ones that aren't. (This works in industries where the
smallest company has a multi-billion dollar market cap. It's obviously less of
a deterrent when the manufacturer has no exposure to the jurisdiction of your
courts or is small enough to file bankruptcy every time there is a problem,
but then we're no longer talking about Boeing and Airbus.)

~~~
samcday
You make a compelling case as to why regulation can be problematic! I can't
find any part of your assessment that I disagree with.

What do you think about reforming how we regulate though? You point out that
the revolving door problem stems from the fact that attracting the right
people is hard when you're not offering them the right salary and career
security. Surely there's ways to make that possible in the public sector!

Given that regulatory bodies really ought to be non partisan and focused on a
particular field or industry of activity, perhaps the institutional heads
could be selected via a more democratic process (rather than just whichever
buddy helped out an elected representative win their campaign) and given
guaranteed minimum terms?

And as to the problem of turnover throughout changes of political
administration, isn't that perhaps also merely a policy problem? Why does the
FCC or HUD need to be gutted and repopulated during a transition of political
power? Those departments still have to report to the executive and
congressional branches of government, so are their staffing choices something
the White House should just be able to change on a whim?

~~~
AnthonyMouse
> You point out that the revolving door problem stems from the fact that
> attracting the right people is hard when you're not offering them the right
> salary and career security. Surely there's ways to make that possible in the
> public sector!

It's actually pretty hard. The first problem is that if you paid them the
amount it would take to attract good people, it's easy for opponents to score
cheap political points by charging you with wasting tax dollars or comparing
the high salary to that of unskilled workers. Meanwhile a high salary
increases the perverse incentive to fill the position with cronies as a reward
for political support, which is the opposite of helpful.

There may be ways around it if you're creative, like making a benefit of the
position that you and two generations of your descendants get free healthcare
and college tuition / loan forgiveness forever (provided you do the job for at
least two years). Then that's worth potentially quite a lot of money and
provides a suitable incentive to attract good people, but makes it harder to
take cheap shots because the total cost is indeterminate in both time and
amount. Meanwhile you attract just the sort of experienced greybeard you want,
who can do the job for a few years as the final leg of their career before
retirement and in so doing provide financial security for their children and
grandchildren.

At which point it becomes sort of a meta problem, because to bring that about
you need honest and creative people in the legislature to make good rules,
which was kind of the original problem -- how do you get good and honest
lawmakers and exclude incompetent and corrupt ones, when the people choosing
them have insufficient time and expertise?

> Given that regulatory bodies really ought to be non partisan and focused on
> a particular field or industry of activity, perhaps the institutional heads
> could be selected via a more democratic process (rather than just whichever
> buddy helped out an elected representative win their campaign) and given
> guaranteed minimum terms?

Making them directly elected positions might help, but it's more like trading
one set of problems for a different one. Now it's an elected position, but are
random airline passengers going to care about who gets elected? (Boeing will,
of course.) How do you get honest specialists who know what they're doing to
even run, and how does a random voter tell the difference between that person
and an industry shill prior to putting them in office? How do you prevent the
foreseeable result when there is a disaster not long before an election and
people elect a strongman to Do Something About It, only to ruin everything
with bluster and incompetence?

> And as to the problem of turnover throughout changes of political
> administration, isn't that perhaps also merely a policy problem? Why does
> the FCC or HUD need to be gutted and repopulated during a transition of
> political power?

It's because they make policy. If a Democrat replaces Trump they're certainly
not going to keep his head of the EPA. Giving them guaranteed terms that are
of similar length to the President's doesn't strike me as something likely to
change much in a helpful way -- and you still have the same issues in choosing
who to select to begin with.

~~~
samcday
Yeah, my suggestions were just spitballin'. I have exactly zero experience
formulating good public policy :) I find it fascinating though since I have no
illusions that even the best policy nerd is ever gonna come up with an optimal
solution, but I'm positive there's plenty of low hanging fruit!

> how do you get good and honest lawmakers and exclude incompetent and corrupt
> ones, when the people choosing them have insufficient time and expertise?

I think this is the crux of the problem, but I also think it's actually a
problem no matter what. Because if you don't regulate, you're just basically
crossing your fingers and hoping shitty people don't climb the corporate
ladder at Boeing / AT&T / Volkswagen / whatever.

> How do you get honest specialists who know what they're doing to even run,
> and how does a random voter tell the difference between that person and an
> industry shill prior to putting them in office?

Well I guess in my head I figured that if it becomes an enviable electable
position, then you're creating healthy competition for it. In the same way we
sort of all collectively hope that competent and benevolent people will run
for Congress, the presidency, etc. By making the positions individually
electable and clearly demarcated, I would hope that then the public is making
decisions for that role more based on competency, rather than charisma. Put
differently, I'm not a huge fan of representative democracy because I think it
inevitably descends into cult of personality, but I don't necessarily have
something I can hold up as a better way ;)

> If a Democrat replaces Trump they're certainly not going to keep his head of
> the EPA.

Yeah but that's my point. The people making policy in the airline industry, or
the telecom industry, or whatever, should ideally be formulating public policy
in the most bipartisan way possible. If you made those roles independent from
the current ruling party, then it would mean that those organization heads
would need to maintain healthy ties to the major parties, since they wouldn't
be able to effectively carry out their role if they're only pals with Bernie
and are universally hated by all conservatives. I'm probably just hopelessly
naive, but I think it should be the moderates who are making decisions about
how to maintain airline safety, not some hyper partisan cronie of the DNC or
RNC.

------
elil17
It sounds like Reuters is trying to draw a connection between this and the 737
Max crashes, but I don’t think there’s any reason to believe that better
testing in silico would not have caught that issue just as well as physical
testing

~~~
alkonaut
And let’s not forget that too strict requirements for _new_ airframes is part
of what led Boeing to keep modifying the original 737 design instead of
building something that would require new certification for both planes and
pilots.

------
dreamcompiler
It's time for the Feds to insist that Boeing return to developing aircraft in
an adversarial manner, where dedicated teams try to break everything the
primary engineers build. Adversarial engineering is the only proven reliable
way to build safety-critical systems. It costs more initially, but it's less
expensive than the penalties and lawsuits later.

~~~
dsfyu404ed
So what you're saying is they need competition?

~~~
dreamcompiler
They have competition from Airbus. But this is too coarse-grained in time to
provide the necessary incentives to improve safety.

------
contravariant
I'm not really against testing something in a simulation, but in this case
would the simulation be using the same software that designed the thing in the
first place? Because in that case you're not so much testing it with software
as just removing any testing altogether.

------
mhb
Boeing 777 wing loading test:
[https://youtu.be/Ai2HmvAXcU0](https://youtu.be/Ai2HmvAXcU0)

------
conistonwater
> _expanding the use of digital analysis over costlier physical testing_

Oh no.

> _For example, when vibrating a fuselage on an enormous platform to expose
> weaknesses - known as fatigue testing - the vast majority of the time the
> tool itself breaks instead of the airframe, according to a person with
> knowledge of past tests. Such work is costly and has reliably confirmed
> engineers’ expectations, he added._

Is this just hubris, or is this real? That's a lot of confidence to place in a
software model.

~~~
chroem-
It's not software: it's mathematics. You don't build a bridge by successively
creating bridges with more and more structural reinforcement until the bridges
stop falling down. Instead, you calculate the appropriate size of the columns
using a model and only build a single bridge. This is no different.

Finite element methods are such a known quantity that we're able to design
nuclear weapons without ever testing them in meatspace. If it's good enough
for nuclear weapons, I am strongly convinced it's good enough for something as
simple as stress analysis.

~~~
tempguy9999
I'm neither an aero engineer nor a nuke weapons designer but I am pretty sure
that nuclear weapons are a whole lot more complex than a plane.

As for not testing nukes, I'm damn sure they test the relevant non-nuclear
parts such as the lensing (conventional) explosives.

> ...for something as simple as stress analysis

Of _entire wings_? From the article "..such as using machines to bend the
wings to extreme angles and shaking the fuselage until it cracks"

Your job isn't in this area I guess. I'm sure of it.

BTW the model is only as good as its inputs so if you have a defective batch
of components (and wasn't there very recently a case where a bunch of
deliberately below-spec aluminium parts were delivered to various aerospace
companies which cost lots to rectify) then your model had better refelect that
defect. But how are you going to find out? With pure maths?

~~~
chroem-
> Your job isn't in this area I guess. I'm sure of it.

I'm a mechanical engineer. My point is that stress analysis is the oldest,
simplest, and most well understood application of finite element analysis. I
can't even begin to imagine the multiphysics that goes into simulating a
nuclear weapon, but if we are willing to bet our entire nuclear deterrent on
it, it seems reasonable to skip some additional rounds of physical tests that
are mostly a formality at this point.

Also, defective materials are caught much earlier in the manufacturing process
through quality assurance methods, before they become airplanes. And then you
apply a factor of safety on your calculations for added assurance.

~~~
kanaba
> I'm a mechanical engineer. My point is that stress analysis is the oldest,
> simplest, and most well understood application of finite element analysis.

I am not a mechanical engineer.

My understanding is that some composite materials used in modern aircraft
manufacturing can fail in ways unlike classic aircraft aluminum (i.e.
suddenly, without prior visible signs of stress/wear).

Is that relevant here?

~~~
goatinaboat
I am a mechanical engineer, and yes it is. The composites are simply too new
to have all their properties adequately modelled yet. And they are a moving
target.

------
taneq
Well, in theory there's no difference between theory and practice...

~~~
noir_lord
I think (since the topic is software testing of physical things) that this
classic applies.

> Beware of bugs in the above code; I have only proved it correct, not tried
> it. - Knuth.

------
throw2016
This is another instance of market failure with the mantra of 'freedom', self
regulation and 'good intentions' spectacularly coming undone.

Boeing's CEO is incredibly still in office inspite of damning evidence of
incompetence which is a straight indictment the whole concept of 'shareholder
interest' and accountability.

Can anyone provide one instance where shareholder interest has ensured some
kind of accountability of management? Why shouldn't Boeings top management be
fired for seriously damaging the company and the brand?

~~~
astrange
Boeing's share price hasn't gone down, therefore they haven't actually damaged
the company.

~~~
throw2016
If this doesn't affect Boeing's share price what will?

This raises even more questions about the stock markets accurately reflecting
business sustainability, revenue pipelines and brand damage given 737 Max
orders are now essentially over. [1][2]

[1]
[https://www.bloomberg.com/news/articles/2019-03-14/boeing-s-...](https://www.bloomberg.com/news/articles/2019-03-14/boeing-s-600-billion-
in-max-orders-at-risk-as-airlines-retreat)

[2] [https://www.wsj.com/articles/boeing-737-max-deliveries-
fell-...](https://www.wsj.com/articles/boeing-737-max-deliveries-fell-in-
march-11554826316)

------
kjar
Boeing cost cutting as the Max runtime defects pile up - aircraft groundings,
returns, and cancelled sales. My confidence in this rationale is non-existent.

------
anticensor
They are asking for more trouble by doing this.

------
solarkraft
Brilliant PR. Let's have more Boeings crash.

