
The Toyota Way at Codeweavers - fagnerbrack
https://www.infoq.com/news/2017/10/toyota-way-codeweavers
======
OpenDrapery
I've been around long enough to see software development methodologies come
and go. I worked for GE for a decade, and they tried to force Six Sigma on
everyone. Devs put through training. Lean. KanBan. Single piece flow.

The thing that businesses don't want to hear is that software development is a
skill game. You win by having better players. You cannot put blind process
around every facet of it. I know you'd like to, so that you can then pay very
little for unskilled labor. But the fact remains that you cannot, even today.

Wysiwig html editors and drag and drop UI builders for rapid application
development were trending for a while in the 90's. Then everyone realized that
giving these tools to monkeys resulted in a big pile of poo. So the trend
reversed back to hand coding UI layouts using markup, separating business
logic from display concerns, etc

I guess what I am trying to say is that if there is any hope in putting a lot
of process rigor around software development, it will be enabled by tooling.
Until then, skill and experience matter. And they aren't cheap.

Put another way, if the Cleveland Browns stole the New England Patriots
playbook, they would still suck.

~~~
narrator
One interesting thing about software methodology that most people seem to
like, such as testing and version control is that it prevents bad coders from
screwing up a project too badly.

I also think that languages like Java have become popular in big enterprises
filled with lots of coders because the language makes it hard to write
difficult to maintain code by doing away with any features that would make it
easy to obfuscate the intentions of code. This concern also had a big
influence on the design of Go and all the features it left out.

The upshot of these coding methodologies that stood the test of time is that
they protect the project from bad coders and they prevent really smart people
from writing code that is too clever such that noone else can maintain it.

~~~
lastofus
Instead the really smart people write abstraction upon abstraction until they
can achieve the level of cleverness they wanted in the first place. I'm not
sure all that extra code is really that much better.

------
tootie
I have to laugh at this. I know this is in reference to their manufacturing
process, but I have worked with Toyota's IT department and they don't follow
anything like this. They are a typical stodgy, waterfall, design by committee,
ivory tower architect kind of IT department. We pushed agile on them, but they
still insisted on signing off on 100-page BRDs before work started. Maybe I
just had a bad experience, but I recall constantly wondering why we had to
work so hard to sell agile on the organization that practically invented it.

~~~
dragonwriter
The Toyota Way is, as I understand it, much more closely related to Plan-Do-
Check-Act, empirical, metrics-based approaches in Lean Manufacturing, Six
Sigma, and other optimize-a-figure-of-merit methods (a lot of which are
anchored in approaches championed by W. Edwards Deming and other contemporary
advocates of empirical/statistical process control.)

Agile philosophy is _loosely_ similar in that it has a “do what works without
dogma” core, but has much weaker focus on actual concrete measurements and
empirical validation, which is one reason why despite its fundamental (in the
Manifesto) _opposition_ to canned, one-size-fits-all process that isn't
adapted to the particular context, it devolves into a lot of that—without a
cultural value of empirical measurement of outcomes, social proof and “this is
the way to do it because this is what other people who succeeded were using”,
or “this is the way to do it because it comes from <authority figure> who
makes <abstract argument> and/or gave <subjectively appealing presentation>”
become the bases available to make decisions on how to implement the vague
ideals of the Manifesto. And also why it can be particularly hard to sell to
an organization that does believe in changing process based on a specific,
testable hypothesis about improvement in a key measure.

------
hencq
> Whilst on the surface it is all about manufacturing cars if you dig a little
> deeper, particularly into Taiichi Ohno’s work, you soon start to see how
> these rules can be applied to all kinds of manufacturing and we saw many
> parallels in what we were reading about the Toyota Production System and the
> way we delivered our software.

I found this really interesting. I completely agree that the principles behind
the Toyota Production System can be applied to many other disciplines than
just car manufacturing. I wonder if Toyota itself actually applies its own
principles when it comes to software development. When their software got
audited as part of the unintended acceleration case [1] it was described as
spaghetti code and having generally poor engineering practices.

I'd be interested to know why this seeming discrepancy exists between Toyota's
quality in hardware vs software.

[1] [http://www.safetyresearch.net/blog/articles/toyota-
unintende...](http://www.safetyresearch.net/blog/articles/toyota-unintended-
acceleration-and-big-bowl-%E2%80%9Cspaghetti%E2%80%9D-code)

~~~
uncle_bob
Hardware can usually be spec'ed reasonably accurately from the start. "The
Toyota Way" can incrementally optimize building against a specification, from
the bottom up, starting with smaller details.

Software however is all but impossible to spec from the beginning, or perhaps
ever at all... so with only working with the smaller details, the larger
picture is lost. At any time, any team, is instructed to stop a problem and
immediately fix it, it's band-aids on top of band-aids.

Having multiple teams working on overlapping projects, with little
communication leads to the spaghetti code that was mentioned.

~~~
otakucode
You can spec hardware, but when the company goes behind the engineering teams
back and lies and saves a few pennies per car by not putting error-correcting
RAM on the things they say they will... you're kind of left hanging on that.
And that's how Toyota works or at least worked when building the Corolla (just
one of the best selling cars ever).

Much of the code being dealt with by Toyota (at least that subject to the
lawsuit and so made available for analysis) was embedded code which is a bit
more amenable to very deliberate design, testing, and verification. NASA, for
instance, has a long history of producing extremely reliable software, even
proving the software correct at times. Would any private corporation in the
world ever resort to following such a practice? Never. It would quadruple
their software spending and, maybe worse, they would have to listen to their
engineers. When the business wants to ship a project, if the engineer says
'its not ready yet', if its a bridge and they go ahead someone in a suit goes
to prison. If it involves software, everyone goes home happy except the dead
customers.

------
csours
The cornerstone of Lean Manufacturing is removing waste.

One of the largest historical sources was inspection and repair: you built
something and then inspected it and fixed any problems. The inspectors and
repair people didn't really talk to the production people much.

In Lean Manufacturing, the main purpose of inspection is to drive process
improvements back into production. (This is the simplistic explanation, but
I'm only using this as an analogy)

Thinking about data management - there's a lot of dirty data out there, and a
lot of data munging to fix it up. Lean Data Management would drive process
improvements into data acquisition.

Of course on the other hand, it's extremely cheap to fix data, on a per-unit
of data basis. It's rule management that's expensive.

~~~
keithnz
the cornerstone of lean is "flow" removing of waste is to the end of achieving
flow (time it takes to get sales to delivery).

Toyotas system is predicated on the idea that for manufacturing THEIR product
the touch time is way less the than the buffer/quueue times ( ie, the time
spent adding value is small compared to the the overall time to get an order
and get it delivered. Eliminating waste is a means to an end to cut into
unecessary buffers and queues and increase flow based on the constraints of
the system.

In software there's a lot of applicable ideas, but it's not exactly the same
and requires careful thinking around what your revenue model is and how you
deliver software based on that. But the focusing idea is not the practices of
lean but how to increase flow and build (reuse) practices around that.

~~~
csours
Right, as I said, that's the simplistic view, and just some of my thoughts.

------
idoh
I have read the books on the Toyota Production System and it does not apply to
software, except in a very loose general sense. Manufacturing is not at all
similar to software because with manufacturing you are making the same
physical thing again and again, while with software you are only making a
thing once, and it is intangible.

There are benefits for examining the flow of deliverables and information, and
trying to eliminate waste, but much of the TPS is geared around the
manufacture of physical objects and it is a stretch to apply it elsewhere.

~~~
walshemj
Exactly as A senior Guy at the BSI said where do you hang the defect ticket
:-) I was working of a Project with the BSI that fed into BS 7570 or ISO 9000

------
davidbanham
I think the core flaw here is thinking of software development as
"manufacturing". It isn't at all. It's a design process.

This Toyota methodology is about taking things that have already been designed
and then incrementally improving them or figuring out better ways to put them
together.

Building software is more akin to the process of taking a blank sheet of
paper, a knowledge of what kinds of parts already exist, then designing a new
car to fit a new market segment.

~~~
hencq
Yeah, true. Though there are definitely parts of the development process that
are recurring. E.g. processes for code review, testing, etc. These are the
areas the article focuses on as well. Since ultimately the software is
developed by this design process, improving this design process _should_ lead
to better software as well (whatever way you define better).

For the actual design of individual pieces of software, I wonder if something
like Toyota Product Development System is a better fit.

------
dnos
I worked at a software company that went through a "lean transformation" where
management effectively idolized Toyota's production philosophy and wanted it
used in each and every department.

While it helped our customer service department quite a bit as-is. The Toyota
Way is definitely not something you can directly copy/paste and use in
software development. Building a Camry 5000 times is not the same as shipping
5000 different software projects, so a lot of the analogies and examples you
find in text don't always apply. I always found it amusing that you never see
or hear examples used from the R&D and engineering of the actual automobiles
-- just once all that hard stuff has been figured out and the manufacturing
starts, all the principles are magically, easily applied.

To be honest, a lot of it is just "common sense" or things that just naturally
transpire if you have a good team (especially if Agile dev and ideas like
continuous delivery are used anyway.)

With the negatives aside, the problem-solving culture part of it is something
that can really change things for the better and is worth researching. If
nothing else, I found just being able to have a framework for solving problems
as they come up is huge -- surely regardless of what a company does. For
instance, integrating a standard to collectively write out the "Five Why's"
and clearly define the problem statement when a team is gathered to discuss
solutions to a problem can save so much time. How many times have you been
dragged into a meeting, go through a huge discussion, design a solution, only
to realize later that it wasn't really solving what needed to be solved?

Outside the culture aspects, it mostly just felt forced, frustrating devs, and
really made things feel almost like a cargo cult waiting for the goodies to
drop from the sky.

------
otakucode
Is this the Toyota that put together cars that killed multiple people? The
ones that were shown in court to be ASTOUNDINGLY incompetent? The ones whose
developers didn't even have access to a bug tracker? The ones who followed
only 4 of 90+ automotive industry standard 'required' or 'suggested'
practices? The ones who didn't use even basic static analysis tools?

I'll pass, thanks. I know it's all fine for Toyota because the courts ruled
essentially "Its computers and nobody knows how they work! We can't convict
them of criminal negligence!" and it saves them billions by not hiring
experienced people or giving engineers the tools they need or training them or
anything like that... but I actually have a conscience and don't mind if I
make a little bit less money if it means there won't be the dead bodies that
are my responsibility like Toyotas got.

------
polskibus
If anyone would like to read more about the real lean techniques in software
(not just kanban), I recommend Lean Software Strategies book.

[https://www.amazon.com/Lean-Software-Strategies-
Techniques-D...](https://www.amazon.com/Lean-Software-Strategies-Techniques-
Developers/dp/1563273055)

The West has a tendency to cherry pick techniques from holistic approaches
(for example quick path to mindfulness vs meditation and Buddhism). I suppose
it's a cultural difference. In my opinion, it is really worth it to try to
understand Lean as a whole instead of looking at kanban only.

~~~
csours
> The West has a tendency to cherry pick techniques from holistic approaches

I agree that it's worth it to spend some time to understand the holistic
approach... but there's also a lot of bullshit and over-reliance on form over
function.

~~~
kitotik
An excellent example of the difference in approach!

The “bullshit” emphasis on form before function is a pretty standard part of a
3 stage learning method - naive and unquestioning memorization of rules/form,
unique combinations of the memorized forms, and finally throwing it all out
the window and inventing new forms with the newly acquired knowledge and
skills.

------
Jtsummers
If you want to try Lean/Toyota Way in your sofware business, in addition to
reading the suggested texts at the end, check out Mary & Tom Poppendieck's
books. They offer a lot in terms of already translating lean production
concepts into lean software design/development concepts. They also offer
exercises that can help you move towards being lean. Because being lean isn't
a state, it's a goal and a process.

------
mmjaa
I've been in software since the 80's. I've watched all the fads and cults rise
and fall .. From General Electric to Kanban and back again.

Honestly, over the decades, there are very specific patterns in all the ism's,
and its the _recognition_ of these patterns which each new generation of
ignoramus decides to make their own and turn into the Hot New Cool Shit.

So I feel its really necessary to point out that while, with the Toyota
Method, there is very definitely a refinement of techniques that have been
well-worn, and proven, indeed in many markets around the world - there are
still parts of the human psyche of productivity which remain _unexplored_.

One of those aspects, I feel, is the obfuscation and neglect which occurs with
every wave of computer culture.

We kind of don't really work on the reasons for obfuscation - because
obviously this is not productive. Productivity means, getting the shit done,
in a way that increases value - for the individual producers as well as the
owners of the means of production.

Toyota _works_. The proof is in the fact that it is one of the biggest and
best brands in the world.

But what doesn't work is the occlusion of methodology that occurs, which makes
the revelation here in the 21st Century of time-worn, centuries-old
methodologies, relevant.

We get this way because we decide to occlude, forget, neglect, and ultimately
.. reject .. the ways of the masters. How many of us read this article, gave
it a cursory glance, and decided to close the tab, for whatever reason: "its
too preachy", "its a culture thing", etc. And what number of us actually
decided to incorporate the Ohnno way into our thinking, for the simple fact
that HN gave us the clue?

Really, what I would like people to understand is that there is a desultory,
occluding factor, to these different 'ways'. The moment the marketing people
take over a working technology, they decide to brand-differentiate, and this
means forgetting the roots of it all. "Nobody thinks of the sugar; all we see
is the cola" can only happen if we actually act on the basis of ignorance.

The point being, the Toyota way seems - like so many other ways - to have the
principle purpose of fighting ignorance. Ignorance of oneself, ignorance of
others, of the customer/user, of the managers, and so on. Ultimately, the Zen
approach is, remember that you will always be forgetting, and express your
current reality in such a way to defeat that. No successful brand is ever
forgotten; no successful methodology can happen in a vacuum. Yet: it requires
such a vacuum before you can find your own handle on it, and ultimately _apply
it with success_.

"Applying it with failure", on the other hand, is a sure-fire to re-invent
something else, 'brand new', again and again.

Don't you feel there is something inherently intrinsic to Toyota Method? Well,
there should be a list of these feelings, by now, for those of us who have
been paying attention for decades ...

~~~
kitotik
Fascinating commentary.

The paradox is that the more you attempt to tie practical/tactical things like
Lean, Kanban, meditation, diet, excercise, etc to a broader perspective, the
more likely the average person will actively resist/ignore both.

~~~
karljtaylor
it is a delight.

though I'd be remiss if I didn't point out that team jack hollis is on a path
to do that yearly. if they follow through on the user-based stuff, they'll end
up doing exactly what you were hoping for, @mmjaa.

[https://www.youtube.com/watch?v=ULdBWNOskoY](https://www.youtube.com/watch?v=ULdBWNOskoY)

