Hacker News new | past | comments | ask | show | jobs | submit login
People Working in DevOps Should Read the Toyota Way (zwischenzugs.com)
89 points by zwischenzug on Nov 9, 2019 | hide | past | favorite | 36 comments



Toyota, as a manufacturing organization, makes some of the best cars in the world.

Toyota, as a software writing organization, fucks it up so bad that people die. There was a single procedure in their ETCS system that was 1300 lines of C with no unit test plan and global variables out the fuckin wazoo. Whole ETCS had 10,000 globals.

https://users.ece.cmu.edu/~koopman/pubs/koopman14_toyota_ua_...


Jesus, reading that was completely horrific. It's total software gore and terrible practices, in critical car systems..

Accountability and proper practices just go out the window when dealing with invisible code, I guess. It's 2019 and we're still dealing with things like storing passwords in plaintext or overusing global variables. Sometimes I wonder what's the point of even being a programmer?


personally it helps with imposter syndrome, its like thinking youre a terrible writer and then reading some godawful bestseller like 50shades of gray and realizing "hey, i guess i cant really be that bad"


Yes, once you realize the level of imposterism and mimcry going on in software you realize 'hey I can do this too' it's empowering in a twisted way -- I suggest doing it from a position of first principles and reinvention rather than feeding the frenzy


Organizations that blindly and naively follow their interpretation of the Toyota way also write crap and drive their people like a factory gone haywire

Hey let's run our shop like a factory


Enter motorola’s six sigma trash.


Anyone have a book to recommend about how to do this sort of programming safely?


How about Nintendo? They make some of the best most innovative software yet nobody mentions them and their hardware isn't too bad either hmmm


I thought the Toyota Way/DevOps analogs had been done to death to a degree that they became practically a cliche. (Though admittedly I was attending a lot of DevOps events at one point.)

In general, this is a worthwhile read though. The lessons can be overapplied in part because a lot of lean manufacturing approaches do have a lot of focus on inventory which mostly doesn't apply in a software development context. However, as the author notes, there are also philosophical aspects to the the Toyota Way which are relevant.


It’s sometimes hard to see how the core principles can apply to situations in which there doesn’t seem to be an assembly line with individual steps that produces a gear, a motor, a door panel, etc.

I like The Phoenix Project because it specifically addresses this by showing how they apply in a warehouse and in a cube farm.

I don’t really think you need to dogmatically apply all 5 ways to every environment every time, but it’s helpful to remember them and try to see which might apply any time you run up against a problem.

Things like removing a roadblock, even if it slows you down even more to cross-train someone else in dealing with whatever task is causing the bottleneck is, in the end, in everyone’s interest.


> inventory which mostly doesn't apply in a software development context

I heard a guy (a Factorio youtuber who was manager in his dayjob, haha) say that it did apply indeed, if you see inventory as "what has been produced but not shipped, stuff for which the value has not been delivered for", in software context that being unreleased features. It made a lot of sense, and ties all nicely into the notions of short iterations and release early, release often. That was interesting.


I don't disagree. Hence my "mostly" qualifier. One can certainly imagine processes that lead to a lot of unintegrated components, code that should be "done" but won't pass tests, etc.

It's still rather different from manufacturing inventory management but it's not totally unrelated either.


Yeah if we are comparing software to manufacturing, the books I prefer are “High Output Management” and “The Principles of Product Development Flow”


I stopped believing in TPS when I saw how companies did terrible things with workspace/office organization and supply management under 5S methodology.

Basically, workers got uncomfortable, de-personalized workspace; no spare parts and materials and less inventory.

Maybe it is aestherically pleasing for the management, maybe its easier to audit, but no worker like to work in such system and many of the dogmas actively hurt productivity.


I would prefer the Honda way. Local autonomous teams of engineers led by engineers. Even the ceo has to be an engineer - and famously could field strip and repair the motor from his small engine division in front of vip’s and press when it failed during a plant opening grand ceremony.


Whoa, is there a video/photo from this? What year? Did a search but could not find it


That’s a good question. It’s mentioned in the book, “driving Honda” by Jeffrey Rothfeder


Can't recall that particular incident, but I do remember that when one of the engineers in the Honda pit at an F1 race (think it was Mexico City-the race they won in the late '60s) who the chief engineer was for their car, they pointed out Mr. Honda to the reporter.

And yeah, while Honda is considered an engineering-led organization, Soichiro Honda was the son of a blacksmith who may very well have not finished highschool. He was an obsessively skilled mechanic and designer, and had famously little regard for college-educated engineers who would make simple mistakes an experienced mechanic would avoid.

That race win in the '60s prompted Lee Iaccoca (then at Ford) to want to source Honda engines for the new Escort small car to be sold in Europe. Henry Ford II quashed that idea, saying "no car with my family's name on it will have a Jap engine!". Too bad.


...curiously, this has not had the effect of workers voting to unionize at any of Toyota's plants in the US, where they have that option.

Of course, as the managers of Suzuki put it when they were building out the CAMI site in Ontario...you don't adopt another company's production system exactly, any more than you would wear a suit tailored to someone else.


Doesn't stop programmers from arguing for a process because Google does it.


Especially when they are completely ignoring the fact that what works for Google usually works for Google because they have achieved a certain scale, not that Google achieved that scale by adopting these methods.


To be honest there are a lot of processes that don't work for google either but with their treasure chest of money and profits from an easily defended monopoly flowing in, it's really hard to argue that.

I can still remember all of the people who defended the brainteaser questions microsoft and google used to toss at interviewees. If I was right that they were bullshit why wasn't I worth a billion dollars?


I must be in the twilight zone -- could this thread be the watershed moment where sanity prevails in software?


Key Performance Indicators aka when metrics turn into targets for jumped up managers


People don't understand dress analogies any more, it's reduced to absurdity along the lines of 'I don't wear suits so problem avoided'


Anything followed to the extreme will start to break down. We base a lot of what we do on the philosophy of the TPS, but we don't necessarily try to map a car factory 1:1 to our software factory. There are certain abstractions that simply don't make sense.

5S can be a huge trap for businesses that don't realize what the intent is. This is another philosophical item, which doesn't necessarily involve putting labeled bins on everyone's desks and having precisely calibrated seating positions and standardized desk chair models. In our case, we apply the principles of 5S to how we manage our issues and labels in GitHub (which we consider to be a virtual factory floor) not to our physical workspaces.


Automotive analogies are dead we're not building widgets and we're not assembly line workers enough already


Standardization of code via frameworks and scripts also means we're not artisans either. We're somewhere in between.


As knowledge workers our role is to create knowledge and disseminate it we're researchers in many ways


A non shared space shouldn’t be set up by a third party. 5S is valuable for any shared spaces, including code bases


I would also say that anyone espousing The Phoenix Project should also read The Goal, on which it is based.


I really enjoyed the Phoenix project but I have to admit the practices they followed in the last 1/4 of the book were barely described and simply worked almost like magic.


I think they ran out of time or ideas by that point, and intended to follow-up with another book: The DevOps Handbook, which took them a long time to finally get around to writing.


I thought the Google SRE book was more actionable with a better set of principles to follow.


How much do they overlap conceptually?


I never found likers writing to be compelling. I’m sure the foundational ideas are great but I found other books about Kaizen were more interesting




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: