
A short fable of software engineering vs. regular engineering - theaeolist
http://researchblogs.cs.bham.ac.uk/thelablunch/2016/01/a-short-fable-of-software-engineering-vs-regular-engineering/
======
lucozade
Automotive engineering is antediluvian wrt process compared to software
engineering.

We can manufacture billions of copies of our product with error rates that are
almost too small to measure. Six sigma? Hah!

Our design processes are far in advance of the automotive industry. Continuous
design is normal practice. It's only the most careful (or most backward) shops
that'll take months to update a design.

The automotive industry, by comparison, regularly takes years to update a
design even by a minor version. The best shops at this i.e. the F1 race teams
can fix bugs in days, minor releases in months and, if this season is anything
to go by, struggle horribly with major version upgrades.

TL;DR it's fallacious to compare manufacturing to programming. The latter is a
design process even if it feels superficially like an assembly line sometimes.

One final point. I would have very much liked to have read an analysis
regarding the different approaches of the engineers vs the craftsman. This is
an interesting topic and does have crossover between disciplines. Oh well...

~~~
OopsCriticality
> Automotive engineering is antediluvian wrt process compared to software
> engineering.

I couldn't disagree more strongly.

> Our design processes are far in advance of the automotive industry.
> Continuous design is normal practice. It's only the most careful (or most
> backward) shops that'll take months to update a design.

I'd say the latest trends in software engineering processes are actually
adaptations of what has been practiced in the automotive sector (and other
manufacturing sectors) for a long time, but outside of regulated industries,
practiced with far less rigor. I don't think the claim that development
processes in software are "more advanced" holds water (what does that even
mean?).

It's impractical to practice continuous design on a physical product like a
car so I think that's a poor point, but _kaizen_ (which I will argue serves an
analogous role) is certainly used throughout manufacturing and has origins in
the TPS.

> The automotive industry, by comparison, regularly takes years to update a
> design even by a minor version. The best shops at this i.e. the F1 race
> teams can fix bugs in days, minor releases in months and, if this season is
> anything to go by, struggle horribly with major version upgrades.

Ignoring regulatory and documentation concerns, part of the reason it takes
years to update a design is that the costs associated with producing a
physical product are completely different from a pure software product. It's
more cost effective to do good design work and heavy V&V of your design
_before_ you order tooling, rather than trying to modify your tooling (if it's
even possible) after an error is discovered.

~~~
stale2002
Exactly! The auto industry works like it does because it HAS too, not because
it is better. Software design is much better and more advanced, because our
industry allows us to choose better practices.

~~~
redneck_
"better"

What does that word even mean?

~~~
catnaroek
Number of errors fixed. Software engineers fix more errors in their designs,
ergo, they must be using better practices! /s

------
nathas
First, I appreciate this story, but it seems the author compares the build
process of software to the production process of the car line.

It's comparing the build stage vs the literal produce-the-thing stage.

I also think the stoppage of the production line happens in software.

If my code breaks while in production, I grab as many log files and stats as I
can, roll back the code, deploy, and then study exactly why everything failed,
and if we recovered post rollback.

That seems like the similar case on the car production line - haul the faulty
model aside (get the logs/metrics), figure out exactly what happened, fix it.

Who knows how the actual assembly line was put together. Maybe they iterated
hundreds of times.

I don't expect my code to fail in production, otherwise I wouldn't have put it
in production, but I understand it _may_ fail. That's why I put in logging and
metrics collection, the same way those engineers hauled aside the car.

~~~
msandford
A more accurate comparison would be "creating software" which is all design,
and "the process by which the production line is spec'd, planned, designed and
created"

Executing a program is analogous to running the production line.

It's intellectually dishonest to compare design with production.

~~~
sbensu
I agree completely with your last point and think that all Programmer vs
Engineer[1] discussions should at least take it into account.

[1] [https://bensu.github.io/abstracting-engineering-
away/](https://bensu.github.io/abstracting-engineering-away/)

------
curun1r
I hate these articles that compare software engineering to traditional
engineering because the typical software engineer has very different
requirements wrt failure and budget than does the traditional engineer. There
are examples [1] of software being written up to the standards of traditional
engineering. It can be done, it's just very expensive and it almost always
makes more sense financially to write software that's more likely to have
bugs. When that doesn't make sense, more disciplined practices are used.

[1] [http://www.fastcompany.com/28121/they-write-right-
stuff](http://www.fastcompany.com/28121/they-write-right-stuff)

~~~
catnaroek
With respect to the linked article, that software development team might as
well live in another planet. Sure, they develop software for a living, and so
do many of us here, but their approach is so different from ours that they are
effectively practitioners of a different profession. It is dishonest to
insinuate that any programmer could perform well at a job consisting in
developing error-free software.

------
Rexxar

        [...] a whole bunch of engineers tried to figure out not how to
        fix the problem but why the problem happened in the first
        place.
    

This is exactly what debugging is.

    
    
        My friend was quite obviously an accomplished hacker.
        His hacker mentality would have fared quite well in
        software production, where debugging is part of the
        normal workflow [...]
    

Finding a workaround in production is not debugging even if it's useful.

~~~
Someone
_" This is exactly what debugging is"_

Yes and no. Debugging (in the sense that programmers do it) stops when you
learn "X wasn't initialized" or "that buffer isn't long enough for the
trailing zero".

This is root cause analysis
([https://en.wikipedia.org/wiki/Root_cause_analysis](https://en.wikipedia.org/wiki/Root_cause_analysis)).
It goes way deeper into "why the problem happened". The buffer may be
undersized because the programmer isn't educated well enough, because (s)he
was too tired to think properly, because the spec specifying the maximum size
was wrong, etc.

Each reason leads to a different fix, often through more iterations of asking
"why?"

The programmer's lack of education can be caused by a problem in hiring
practices, by low quality of internal courses, etc.

The programmer may be tired because of long working hours, because of troubles
at home, because (s)he goes out too much, etc.

That buffer size might be a typo, an OCR error, somebody declaring the maximum
size to be N because, otherwise, the data couldn't be fit in the current
hardware, etc.

Each explanation comes with a different way to prevent the problem from
occurring again (something that traditional debugging does nothing about.
Initializing that variable makes the problem go away, but in itself does
nothing from you writing code with initialized variables in the future)

------
amazing_jose
Humans in advanced production lines are "actuators". They are not there to
think, but to replace an hypothetical machine that it's not possible to build
yet. We have those magical machines in software, we call them compilers,
interpreters, operative systems, file systems, etc.

I still have memories of my pass by Uni and how a Software Engineering
professor constantly refered to programmers/developers as "coders" or
"monkeys". He never stopped of repeating to us, day after day, that we
shouldn't program ever. Our job as engineers was to plan, design, assess and
manage those "monkeys".

More than 15 years of professional career later, my initial suspicion have
been confirmed one time after another: those professors from Uni need a good
deal of fresh air and to get in touch with the reality, instead of spend the
day pontificating like celibate catholic priests about what you can and cannot
do in your bedroom.

I know that many people here love Dijkstra and people like him, and even enjoy
to feel their intellectual whipe in their mind, reading their essays and
feeling bad about themselves as the essays repeat constantly how that
everything is wrong and broken in the software profession.

For those people, I suggest to read "To Engineer is Human" by Petrosky and
learn about how real engineers really work.

~~~
krschultz
> Humans in advanced production lines are "actuators". They are not there to
> think, but to replace an hypothetical machine that it's not possible to
> build yet. We have those magical machines in software, we call them
> compilers, interpreters, operative systems, file systems, etc.

Unfortunately this view of people that work in manufacturing is pervasive, but
it is also incredibly naive. That view is essentially the reason why American
car companies (along with many other industries) got their clocks cleaned by
the Japanese car companies over the last 3 decades. I've been involved in a
lot of companies where we worked really hard to correct that mentality and the
results were always obviously positive, but incredibly difficult to achieve.
Engineers just love to look down their noses at everyone else.

There is nothing worse than some new kid that just graduated with a BS in
engineering pretending like he's smarter than a guy that has been building the
product we're designing for 30 years. Unfortunately some people never grow
beyond the 'new kid' stage.

~~~
amazing_jose
I don't get your point.

My comment tried (possibly not very clearly) to talk about the differences
between the other engineerings and software. People keep thinking that
programmers are like factory employees. And that's so wrong. Each engineering
endeavour is different. In software, our factory workers, are the tools that
do the job. What in other places is designing in software is just programming.
In my view (and I'm not alone in that) people keep applying the manufacturing
metaphor to software for the wrong reasons.

In construction, the distance between design and implementation is so huge,
that the construction workers save or doom construction projects all the time.
Spend some time in a big construction site, in their meetings and you will
discover why those projects take so long to finish. Hence, they are important
because what the designer design is just an specification for the project,
it's a map not the territory. An ideal brick in a CAD program is not a brick.
Your perfect design for the AirCon machine in that room got destroyed by an in
place decision by a worker of changing the wiring in a wall, ignoring the
plan. Through tremendous effort during centuries construction has accumulated
more or less accurate models for bricks, but the workers do so many things
that the architects doesn't know how to do that the distance persist, and my
times just ignore them.

In manufacturing, you have the Ford model of car manufacturing. It has being
employed in nearly every single factory in the world. Yes, the Japanese
empowered their employees more than in other countries but hey, they took note
and introduced some of the practices. As of today American companies are as
competitive as the Japanese ones without all the mumbo jumbo and black zen
magic.

My point here is that each engineering is different. Different factors forced
us to discover what work for them and what doesn't. That knowledge was
accumulated in the industry and depending on the engineering, the feedback
loop from the industry into the University could vary _a lot_. In software,
the distance between the two is huge, due to the lack of transparency and
secretism that most software companies exhibit.

~~~
kpil
I guess that treating factory workers as "code monkeys" is also wrong :-)

I'd like to work in a flat(ish) meritocracy. Maybe I have to start my own
company...

------
wrong_variable
Comparing the production line for F1 with your average software shop is an
unfair comparison.

If you are ready to pay the same amount of money - you can get the same level
of quality in software too.

The problem is most people do not even think software should cost anything.

~~~
pmelendez
Not only that... I don't think they will "refactor" the engine in the middle
of the production, or include new features in the transmission before a
deadline.

~~~
VLM
The boss saw an article in "racing weekly" about using square wheels and our
salesman told him you'll have it done by tomorrow.

------
a3n
Software is created once. If it runs on more than one computer, it's just
copies of the actual result of that one time creation. If it's improved with
new features, then that's a new unit, itself created once. The effort goes
into the creation of the one thing, not the copies. Version 1.1 is not version
1.2.

This observation is often used to point out that "software is different" from
cars and other hardware. Cars and hardware have lots of chances for defects,
but software copies are essentially perfect. Assembly line products are not
copies like a package or disk is a copy; each finished unit has lots of effort
in it.

But if you still want to use the hardware example, find what the equivalent
products are. A lot of effort goes into physically assembling a car, and then
the car cranks out transportation.

A lot of effort goes into physically assembling a software team, and then the
team cranks out software.

And while of course there are profound differences between a car an a human
team, it might be useful to put engineering resources into designing and
producing a team.

------
vinceguidry
I try to take this kind of approach with my own work. My workload is really
light, so every task I get, I take the opportunity to dig into the task and
the business needs surrounding the tasks so that I can come up with a solution
that works, is stable, can be extended when needed, and doesn't require a lot
of maintenance.

When an issue comes up, the first thing to do is to put together a workaround,
a way for business to continue in absence of a proper fix. I then, without the
need to rush, dig into the issue, figuring out a way to reproduce it, then
isolate it, then fix it in a way that makes the whole system more robust and
not less.

Having worked on the codebase / infrastructure this way for two years, my
workflow is pretty-well refined. For my current issue, I'm devising a command-
line tool to download and parse the log files to give me exactly the info I
need to run down web issues. I can yak-shave to my heart's content.

------
AdeptusAquinas
Always thought formal methods were an attempt to achieve this.

However the biggest obvious difference that prevents dev becoming like an
engineering assembly line is that the engineers on the line are not surrounded
by car customers changing how the car should or shouldn't work/look/cost every
two weeks.

Not that cleaning up the actual process of development so its more easily and
naturally verifiable wouldn't be a good thing.

------
blokeley
I run a company which develops medical devices, many of which combine
mechanical, electronic and software engineering. It's well known (and a
running joke) that software engineers from non-medical fields think it is
acceptable to ship their product with serious defects. It takes a long time
and a lot of training to get them up to the level that we need, more so than
for mechies and sparkies.

~~~
noir_lord
Do you have any solid books etc for the kind of processes you use?

~~~
blokeley
For software in medical devices, the process that's accepted by the regulatory
authorities is ISO 62304
[http://www.iso.org/iso/catalogue_detail.htm?csnumber=38421](http://www.iso.org/iso/catalogue_detail.htm?csnumber=38421)

That's a good start

------
BEEdwards
I think the author misses the point of their own story, this isn't an software
vs regular engineer thing, this is a problems with monotonous static jobs.

His friends job is to fit one small body panel on the chassis, over and over
again. So when he gets one that doesn't fit his instinct is to make it fit as
opposed to asking why it doesn't. His job isn't questions, his job is to fit
it on.

When he was working in his own body shop this hacker wouldn't have stood for
his tools putting out misprints and mistakes, but then his job was managing
the whole shop.

Static monotony gets static monotonous thinking.

~~~
analog31
The engineers probably realized that cutting and bending the body panels to
make them fit, weakened the structure and invited corrosion. That's why they
spent days trying to figure out how to correct it.

------
antod
This is more about production lines vs one offs than engineering vs hacking.

Civil engineering is real engineering but jobs are usually unique. Problems in
civil engineering projects are usually 'debugged' and 'patched' on site
without going all the way back and revisiting the original design.

~~~
OopsCriticality
Are you suggesting that "rolling with the punches" on a civil engineering
project is happening without going through some sort of an engineering change
order process?

~~~
zwischenzug
There's an interesting story in Atul Gawande's The Checklist Manifesto about
fixing a skyscraper problem that threatened the whole project - if memory
serves it was moving too much. That was definitely a 'one-off' solution. It's
used in the book as an example of how the automation of the building process
freed up the real engineers to solve the specific problems to that site.

~~~
jloughry
It was the Citicorp building in New York City. In 1978, a year after it was
built, Diane Hartley (a student at the time) discovered that aerodynamic
calculations of wind loads on the skyscraper had been vastly underestimated.
She contacted William LeMessurier, the structural engineer, and he re-did the
analysis considering quartering winds. When it became clear the building would
structurally fail in the next big storm, emergency repairs were done, welders
working in secret at night, on almost every floor, installing extra cross
braces. The building was saved and became an engineering ethics example.

Diane Hartley got no credit for her rôle in saving the building until nearly
twenty years later.

~~~
zwischenzug
Thanks! My copy was lent to someone a while back...

------
bb88
Engineering is at its heart is both an applied science and a methodology for
it's application. So yes, software engineering is a valid form of engineering.
We're taking computer science principles and applying them to a practical
application. We don't write papers, but we use the knowledge to drive
industry.

So there's this topic of failure analysis, and we don't always do this well in
the world of software engineering. We don't like to analyze our failures
probably as much as we should, possibly because we find them embarassing, or
perhaps they might be bad PR if they were known. So we just want to get the
fix done and move on.

But the thing is for most software it doesn't matter how we deal with bugs,
and that's because they cause minor problems. But when it comes to loss of
life or money, bugs do matter. Doctors and lawyers have professional boards
that oversee actions and disputes. Airplanes and Trains have the NTSB that
oversee accidents and engineering failures. Civil engineers study the collapse
of bridges as part of an effort to make bridges safer.

Failure in those cases are bad.

If my software solution is off by a few bugs, meh, so what? It doesn't cost
the company money, usually. And it probably didn't kill anyone, and it
probably just fixed some small css issue that looked great on firefox, but bad
on chrome...

------
chrisco255
Software engineering isn't an assembly line. In our case, the assembler, is
quite literally a program that virtually never makes mistakes. When a program
won't compile, as would be analogous to this McLaren case, we do stop
everything and figure out why it won't.

------
kbenson
> I wonder how many years until software production will reach the level of
> maturity where we don’t just fix bugs but, when bugs happen, we stop in our
> tracks, shocked that bugs happened at all in the first place

This happens all the time in software engineering. Less because we're shocked
that bugs _could_ happen, and more because when you are working from the
assumption that what has happened should not be possible, when it happens,
that means your mental model of what is happening within the system is wrong
in some way.

There are the bugs you understand (or think you understand), and those you can
safely allow workarounds for in most cases, because you understand the scope
of the problems. Then there are the bugs that you don't understand, and it's
important to be very careful and provide them the attention they deserve,
because while they may be simple fixes, they could just as likely be an
insidious case of data corruption or loss that you weren't ware of.

------
Animats
The big difference is that the software industry still gets away with the
consumer paying for defects. This is where the auto industry was in about
1952.

------
leereeves
An assembly line isn't comparable to software engineering - it's closer to a
compiler.

~~~
krschultz
Having done both mechanical engineering & software engineering professionally,
I think the assembly line is a lot closer to software operations.

If you take the same code and put it through a compiler over and over, you
will get roughly the same result. If you run the same code in production day
after day, some days you will get different results because of the interplay
between various systems or conditions.

If you take the same design and put it through a computer controlled machine
you will likely get the same results unless something breaks. If you take the
same design and put it through an assembly line, you will get different
results on different days because of the subtle interplay between various
variations in processes.

The distinction is critical, because while you generally qualify a single
operation in an assembly line much like we would run tests on a piece of
software, you need to monitor an assembly line just as we need to monitor our
software in production.

~~~
leereeves
Yes, that is a better analogy.

------
jameshart
I've been in dev shops where, when a NULL turned up in a database column where
it wasn't expected, the site was rolled back to a previous version, people
were called in on the weekend, and a group of engineers spent hours trying to
figure out why that NULL got there and how to stop it from happening again.
It's not like a DBA couldn't have just run a quick UPDATE query to clean up
the null in a matter of seconds...

I see this as indicative that debugging production issues happens in non
software engineering contexts as well, in spite of the fact that we're always
told that their processes and procedures are so much more mature and robust
than ours.

------
ryanSrich
"He builds his castles in the air, from air, creating by exertion of the
imagination. Few media of creation are so flexible..."

Software engineering is so far removed from other facets of engineering that
comparisons to poetry make for better discussion.

------
jagtesh
I think the author is mistaken. No two car companies or software companies are
the same.

The software written for mission critical systems, such as nuclear reactors
and even cars (think direct injection, ABS, AWD and now auto-cruise control
systems) goes through an extensive series of evaluations. Same goes for the
software that helped New Horizons reach Pluto and beyond. It's just not
possible to achieve such feats without a thorough software development
process.

Agreed that the same cannot be said for startups, but comparing a startup to a
multi-billion dollar car company with tonnes of regulatory and compliance
checks is not a fair comparison to say the least.

------
venomsnake
I think that this also happens in every place where mission critical software
is developed.

But here is our dirty little secret - none of the software 99% of us write is
mission critical. And probably 75% of it is never used.

------
kpil
This was actually "debugging the program", as the running program is the
manufacturing process, and the coding and development is the engineering and
design.

Still, software systems are both more fragile and can easily be more complex
than real-world systems, so the naïvity is actually trying to apply
engineering processes with rigid "design" phases and handoffs to complex
software processes.

(As seen in the software, in just about every car.)

------
redneck_
As a civil engineer I am seeing a lot of ignorance on the part of the hackers
towards the physical sciences, especially the author.

Software debugging has 2 parts:

a) find the bug - lucky for you a machine does this step automatically

b) fix the bug - once the bug is found this is self explained

Industrial debugging has 2 parts:

a) find the bug - requires going on the production floor and measuring things

b) fix the bug - once the bug is found this is self explained

How are they different?

~~~
Sir_Substance
I'm a little confused on what you're trying to say. If I read your post
correctly, you think software bugs fix themselves because computers find them
automatically?

Neither is true.

~~~
assocguilt
Dick Smith screwed customer with gift vouchers - you used to work for them -
that should be a career ending move for you as you worked for such a scummy
organisation!!!

~~~
Sir_Substance
Wow, have I stung such a bee with you that you're following the comment up a
second time, in the wrong thread?

I've answered you in the other thread.

------
lordnacho
Seems like the hacker wanted to patch the problem, whereas the engineers
wanted to fix the root cause. I don't think either is wrong; you might want to
patch the issue first and then fix the root cause. Or you might suspect the
root cause is simple enough to fix and not spend extra time on the workaround.
Or you may never find the root cause.

The reason why there's a lot of bugs in software is that software is soft.
People know that changes can be tested with relative ease, and they also have
a good idea of how wrong things will go if they do go wrong, and quite often
an error is tolerable. This means we can often play a bit more loosely with
correctness and reliability in order to gain some speed.

There's plenty of software problems where this isn't the case, and you end up
with a very different process.

------
otoolep
The OP is mistaking manufacturing for design. The manufacturing phase of
software - compilation - is hugely efficient, reliable, and cheap. The
industry certainly does "drop tools" and see what has happened when a compiler
like GCC screws up, which is a better analogy here.

~~~
otoolep
More on this idea, a talk I came across a while back.

[https://youtu.be/9IPn5Gk_OiM](https://youtu.be/9IPn5Gk_OiM)

------
pkaye
When issues occur in my code, I do identify the root cause but then try to see
if it may occur elsewhere in my code and how to prevent similar issues in the
future. And I know lots of coworkers who do the same. I don't think this is a
major engineering insight but common sense.

------
jondubois
Some big software companies operate like the engineering companies described
in the article but it's often not a good thing.

I've worked for a software company where it would take almost a week to add
one new property to a JSON object because there was 3 different sets of tests
I had to write/update across multiple code repos and it involved the
coordination of multiple teams across multiple cities.

I've also worked for startups were it would take me an hour to make a similar
change on my own but with lighter (albeit just as effective) test coverage.

It all comes down to company processes and approach to risk.

I think the lean approach is much more efficient if you have a
good/experienced engineering team which you trust and who actually care about
the product.

------
al2o3cr
The assembly-line story would be way more like software dev if it had a
prelude where the installer repeatedly pointed out that the panels didn't fit
and was told "well, we bought them from the vendor so use them anyways". :)

------
codeshaman
> I wonder how many years until software production will reach the level of
> maturity where we don’t just fix bugs but,

If Google tried to figure out why the hard drives fail in every server, how
many years before we could type McLaren in the search box ?

Instead they embraced a different truth - things fail, so we'll build our
systems to handle that.

And I think this is the reason why we can get away with bugs in software -
because we can isolate and prepare for components failure or imperfections.

So instead of trying to absolutely NOT introduce any bugs, we can instead
focus on making our systems fault-tolerant and introspective and handle all
errors in a graceful way.

~~~
catnaroek
How exactly do hard disk failures support your argument for tolerating bugs in
_software_? Seems like a non sequitur.

------
Kinnard
Cryptocurrency bugs can be quite expensive. Maybe that will make software
engineering more like other types of engineering.

~~~
Xixi
Formal proof in software engineering has been a thing for a very long time, no
need for cryptocurrency for that. Control systems for rockets, autonomous
subways [1], and many life critical devices are built using such
methodologies.

In a startup environment you are first and foremost trying to figure out what
your customers want. Formal proof is not going to help solve this problem. I'm
sure McLaren is also not spending days trying to understand why a panel is off
by a few millimeters when they assemble a prototype.

[1] Paris subway (line 14) is controlled by a software written using a formal
proof method called the B-Method:
[https://en.wikipedia.org/wiki/B-Method](https://en.wikipedia.org/wiki/B-Method)

------
pron
The problem is that unlike with "regular" engineering, there are mathematical
theorems that basically say that writing bug-free code is somewhere between
prohibitively expensive and impossible. The cost of proving a non-trivial
program property has a proven lower bound of O(S) where S is the number of
states the program may take; alternatively, we say that proving correctness is
_at least_ PSPACE-complete (if not EXPTIME) in the size of the program. Not
saying we can't do our best and use various tools, but this cost cannot -- in
general -- be reduced to anything acceptable, and therefore we will never be
shocked when a bug does occur.

It is easier to achieve string AI than to write bug-free code, and even that
won't help: a super-intelligence will still have bugs in the code it writes.

------
sgt101
This is what happens if the compiler glitches or the os fails; people dig in
to find out why. Normal code is much more like doing a service - if the breaks
don't fit it's because the person fitting them has screwed up, and you take
them off and fit them (or a new set) back on again. Maybe there is an inquest
in the team about using spanners properly, maybe the old fella in the shop
tries to communicate some wisdom about checking things first.

At the end of the day it's about money and risk.

But, the interesting difference between software and mech eng isn't the
attitude to faults, it's the fact that software isn't designed to work within
tolerances. We don't build software around dimensions or stresses, we expect
it to work _always_ like maths, but it isn't, quite.

~~~
catnaroek
What's a meaningful definition of "tolerance" for a discrete system?

------
nomercy400
Don't fix a production issue by 'hacking' in the solution live on production.
Figure out what caused it and fix that. At least, that's what we do for our
software.

~~~
rurban
Having worked as software dev for formula 1 and as hardware engineer ditto, I
can verify this.

You see all. You see people hacking a bugfix in one day on the F1 site, but
this fix will not survive. Some shops restore from their backups nightly, you
will certainly face a lot of meetings and reports to get your one-line fix
back into production. Even on R&D problems, which mostly describes F1.

This case reported here was on the assembly line, there are different rules.

If a strange misproduction happens you certainly want to be able to evaluate
the riscs properly. The analogy is a Heisenbug in the production compiler. How
often does it happen, how expensive is the problem, how expensive is the fix.
There's not one car on the risc, there's the whole line in question. The new
robot? The SW or the HW? Wrong measurement, wrong planning, broken sensor,
broken motor? Or a real Heisenbug, which appears in HW much more often than in
SW. Magneto/electric cabling problem. Or sometimes even light. The cable guy
bending an overlong fibre cable <30cm radius. A fibre with a bad cut at one
end causing all sorts of crazy effects, and crazy hackish fixes. Missing CAN
terminators (analogy to missing SCSI terminators on harddrive chains) in
certain F1 cars. It might work, but I wouldn't want to sit in this thing. This
new 140.000 rpm high-speed rotor behind my back, making the driver
uncomfortable, who is starting making bad mistakes out of fear, and rather
pulls this thing out. Relations to completely crazy effects from outside, such
as a cleaning lady turning on a bad fan, causing a jitter or spike, having
some effects elsewhere. A bug, a real bug, in the system, esp. in Asia or
Australia.

I see not much differences from SW and HW engineering. But the biggest
difference I know about is the methology.

Agile in car manifacturing or HW engineering at all only happens at Toyota,
nowhere else. Properly planned waterfall, with excellent and high powered
middle management is the norm. If in Asia, Europe or the US.

There do exist some very small HW shops, with 1-4 engineers and 2
managers/sales people, and 2 support. There the development is of course
engineering driven. Or there are some exceptions, such as Toyota or VW with
crazy assembly optimizations, or e.g. Honda where their boss is one of the
best engineers (they are lucky). But they eventually they ran out of luck with
their funding and had to sell their very best test site, which was then taken
over Daimler. That's why their are leading the F1 currently.

Another difference is proper training. These good engineering companies train
their people a lot. Maybe 10x more than in normal SW engineering.

------
shitgoose
if software were like that hacked jeep, it wouldn't be so bad. lot of software
looks is like this: [https://benahrens.com/wp-content/uploads/2014/03/how-to-
be-m...](https://benahrens.com/wp-content/uploads/2014/03/how-to-be-more-
productive.jpg)

------
lectrick
One not-terrible way to start on your bug-free crusade would be to switch to
functional programming languages and styles and "pure" functions.

Having done 10 years of OO before I began coding in a functional style and
realized all the benefits, this is my experience talking.

------
mattexx
Hacking != Software Engineering

