Hacker News new | past | comments | ask | show | jobs | submit login
Nobody Ever Gets Credit for Fixing Problems That Never Happened (2001) [pdf] (web.mit.edu)
201 points by xvirk on Jan 24, 2015 | hide | past | favorite | 50 comments



And the matching rule: badly designed projects that blow up end up getting a lot of attention and resources, their engineers end up looking like heroes.

I remember the worst module of a very large J2EE website getting all the attention, got more developers, got 10x more hardware than planned... And in the end the lead was promoted. While several other modules did their work flawlessly and went unnoticed.

This is what you get when business people make decisions on technology projects.


From 'Software Requirements & Specifications' [0], by Michael Jackson (not that Michael Jackson, and not the other one either), published in 1995, a parable [1]:

'What do you think?' he asked. He was asking me to tell him my impressions of his operation and his staff. 'Pretty good,' I said. 'You've got some good people there.' Program design courses are hard work; I was very tired; and staff evaluation consultancy is charged extra. Anyway, I knew he really wanted to tell me his own thoughts.

'What did you think of Fred?' he asked. 'We all think Fred is brilliant.' 'He's very clever,' I said. 'He's not very enthusiastic about methods, but he knows a lot about programming.' 'Yes,' said the DP Manager. He swivelled round in his chair to face a huge flowchart stuck to the wall: about five large sheets of line printer paper, maybe two hundred symbols, hundreds of connecting lines. 'Fred did that. It's the build-up of gross pay for our weekly payroll. No one else except Fred understands it.' His voice dropped to a reverent hush. 'Fred tells me that he's not sure he understands it himself.'

'Terrific,' I mumbled respectfully. I got the picture clearly. Fred as Frankenstein, Fred the brilliant creator of the uncontrollable monster flowchart. 'But what about Jane?' I said. 'I thought Jane was very good. She picked up the program design ideas very fast.'

'Yes,' said the DP Manager. 'Jane came to us with a great reputation. We thought she was going to be as brilliant as Fred. But she hasn't really proved herself yet. We've given her a few problems that we thought were going to be really tough, but when she finished it turned out they weren't really difficult at all. Most of them turned out pretty simple. She hasn't really proved herself yet --- if you see what I mean?'

I saw what he meant.

[0] http://www.amazon.co.uk/Requirements-Specifications-Software... [1] http://www.win.tue.nl/~wstomv/quotes/software-requirements-s...


I've noticed over the course of my career a fairly severe corollary, which is that making the best technical choices for my employer can often be bad for me in the long run.

As an example, for about the last 10-15 years, I've tended to choose Python over C++, for projects where it seemed it would produce a better solution (e.g., cheaper/faster to implement, easier to maintain and alter, performance sufficient to the task at hand). Because Python is actually quite adequate these days for large swaths of the task space, this means that my Python chops are quite good, but that my C++ is a little rusty.

In itself, this isn't exactly a problem, but in an interview scenario, it can look quite bad. "How much template metaprogramming have you done lately?" "Not much." "So, you're more of a junior programmer then?" "No, I just haven't hit a problem lately that needed that, and I prefer not to introduce unneeded complexity and maintenance nightmares into my employer's codebase." "How much work with GDB automation?" "Very little, since the code I write pretty much just works." "Ah, so weakness with the debugger as well." Etc.

Not really sure what to do about this. I don't really want to write tens of thousands of lines of unnecessary C++ just to impress future employers. But I have to acknowledge that that's really how the rewards are set up.


This dynamic is part of why I feel like programmers should really just stop resisting the pull to move into management. You might think about programming as the most amazing thing ever, to be agonized over with a fine-toothed comb and the 'best tool for the job' elevated to a nigh-religious consideration, but to your employer, it's one line item on a budget.


It might be 'easier' in a certain sense, and possibly a way to optimize for your paycheck, but 'resisting the pull to move into management' for me is about wanting to do high quality engineering, having a sense of professionalism and pride in my work, and feeling good about what I'm doing at the end of the day, even if my management doesn't understand it fully. If I'm a "line item on a budget", I want to be the best damn line item I can be. I am an engineer.


You can do high-quality engineering in management. You're just doing it with other people's code, as well as your own if you choose to continue programming.


What activities would you expect to be doing in such as scenario? For me, if I was doing it "with other peoples code" rather than "with other people", I would expect to be doing things like large scale architecture, developing tools such as domain specific languages, and investigating new technologies for possible adoption, setting the stage for the rest of the engineering team to write good code and solve hard problems. I would consider this engineering, and "working with other people's code." I would feel some secondary 'ownership' of the rest of the teams code in this scenario.

On the other hand, if I was working "with other people" as in a managers potions, the most technical things I would expect to do would be making hiring decisions, considering the skill sets and training needs of a team, and helping a team make good technical decisions, as well as other less technical manager duties. I see this as a separate management role, that doesn't really involve engineering, and working "with other people" rather than the code they produce. Importantly, I would NOT feel much ownership in their code in this scenario. I would feel ownership in their success coding, but not in the product itself.

You could, possibly, do some of each although I find the two work styles severely interfere with each other, so focusing on one at once is far more effective. If you are an amazing person you might get away with sneaking in some engineering time while being an passable manager as well. If I was in a managerial role, I would focus on that to allow myself to be more effective, and get my engineering satisfaction elsewhere.

Note that some large tech companies recognize this difference, and have explicitly create two distinct promotion tracks, one through management, and one through engineering.

I feel it is a tragedy that many companies feel that "engineering is the thing you do first, they you get promoted out of it into management to continue your career," and structure their org chart with engineers at the bottom.

I think this comes from people who enjoy and are good at management not recognizing that there are people who are not like them, and building a hierarchy that reflects their world view. If more companies realized that there are people who want to be good engineers throughout their career and could relate to them better, more amazing engineering would happen, and it would be well managed too.


Rebrand yourself as a python developer.

I know you might not want to hear that, but you'll probably find much more enjoyable employment anyway.


Essentially, that's where I'm at, as a practical matter. But it feels wrong to be typecast that way, since I regard my C/C++ capabilities as being quite sharp as well. And it seems like knowing when to write 100 lines of Python instead of 1000 lines of C++ ought to be valued, rather than seen as more of a technical faux pas. (Hey, I can dream, can't I? :-)


Work for people who are more concerned with the business value you provide and how fast, and reliably you provide it, and who don't care about the details of what technology you use.

This often means being hired by the CEO and not the CTO.


If you liked that, you should definitely read The Parable of the Two Programmers:

http://www.csd.uwo.ca/~magi/personal/humour/Computer_Audienc...


The main thing I read in this parable is the intense focus on comparing individuals and putting them at odds, instead of working to find the best way they can work together in a good system.

What's the core problem there? Why do people think in this way, instead of more systematically? What's the intention and what are the assumptions? Many questions raised.


I'll try to address a few points here:

-What's the core problem?

Evaluation. In this case, evaluating intrinsic difficulty of a certain task or group of tasks. There's no way to do so systematically, without inside knowledge about the task. Foe example, suppose you want to rank two mathematicians without a clue about the subject matter. One presents short proofs to many problems, while the other presents enormous proofs with great effort. There's no way to tell if the problems are really difficult or not and chose the reward without having knowledgeable peers evaluate the a) hardness, and b) usefulness intrinsic to the problem. In fact, it could be argued that simplicity is a good sign, but it's impossible to tell whether the problem is unimportant/trivial to begin with.

- Why do people think in this way, instead of more systematically?

Without insider knowledge, which management sometimes lacks, you cannot confidently rank as stated above. But there's one trivial thing you can do, which works, but not necessarily efficiently: if the solution works/is found, reward the solver, otherwise punish/look for others (and reward them more). Hard problems skillfully solved will go unrewarded and good performers will gravitate towards fixing crisis instead of doing good ground up engineering as they should. That's the aspect of inefficient, but it is possible with enough money to manage this way.


The answer to both is to look one level up, at the meta question. Both of your answers are still about evaluating individuals. Instead, look to the system both individuals are in.

In this case, the system encompasses the evaluation itself, and the answer is to cease evaluation altogether. You can clearly see the problems it causes.


How do you choose who to pay and how much?


In an individual manner commensurate with their contribution and market value, as simply and clearly as possible. It is not complicated, and making it complicated is how we end up in cultural negative feedback loops. The standard performance assessment does more harm than good, and we absolutely should not use it.

Some external resources:

https://www.psychologytoday.com/blog/wired-success/201211/wh...

http://www.theatlantic.com/business/archive/2014/01/the-case...

http://www.washingtonpost.com/blogs/on-leadership/wp/2014/01...

http://www.amazon.com/Abolishing-Performance-Appraisals-Back...

And finally, Deming's own brilliant opinions:

http://blog.deming.org/2013/02/the-idea-of-performance-ratin...

"The merit rating nourishes short-term performance, annihilates long-term planning, builds fear, demolishes teamwork, [and] nourishes rivalry and politics. It leaves people bitter, crushed, bruised, battered, desolate, despondent, dejected, feeling inferior, some even depressed, unfit for work for weeks after receipt of rating, unable to comprehend why they are inferior. It is unfair, as it ascribes to the people in a group differences that may be caused totally by the system that they work in.

The idea of a merit rating is alluring. The sound of the words captivates the imagination: pay for what you get; get what you pay for; motivate people to do their best, for their own good.

The effect is exactly the opposite of what the words promise. Everyone propels himself forward, or tries to, for his own good, on his own life preserver. The organization is the loser. The merit rating rewards people that conform to the system. It does not reward attempts to improve the system."

-- W. Edwards Deming

And even more in depth: http://blog.deming.org/2012/10/dr-deming-called-for-the-elim...

In short, the method by which you determine fair pay is very much irrelevant — so long as it does not involve the detrimental practice of performance appraisals. The main focus of both you as a manager and your employees should be on the system of work, not on the individual. Deal with the individual's fair compensation simply and clearly, and discard all by-products, as they are unproductive.


Is this book worth buying?


Great story. Reminds me of

"Fools ignore complexity. Pragmatists suffer it. Some can avoid it. Geniuses remove it." - Alan Perlis


Along similar lines:

"Make things as simple as possible, but not simpler." -Albert Einstein


I just saw this in the last project in my company: One group delayed the difficult decisions until the last moment and had to work nights and weekends to catch up. All other groups were on top of things and done way before the deadline.

Guess which group saw promotions and gets recognized by senior management at pretty much every meeting?

Seems the best strategy is to create problems and then fix them in the most visible way.


I agree with your comment except the last sentence. Engineers also value heroics, so it's unfair to lay the blame at the feet of 'business people'. Plenty of tech startups exhibit this pattern long before there are 'management types' around.


If you read the original article in full, it covers this type of situation quite well. This is true: both employees and managers believe the blame and the heroics go to individuals, and this is a fundamental attribution error.

The vicious cycle caused by this attribution error drives companies into holes as they continue to reward and punish the wrong people for the wrong reasons.

In reality, most problems in work are systemic, and the only solutions are systemic as well. You want to create an environment where both 'business types' and employees alike are on the same wavelength about this fact, and work to improve the system they're in continuously. This is how you achieve quality, period.


I completely agree with you and I normally describe it in terms of creating the right culture (aka systemic behaviours). I've yet to read the article in full but will make time for it.


But it's easier to make business believe it's just a hard problem instead of incompetent people in charge of the solution.


The same way the police describe uncaught criminals as smart and dangerous.


>This is what you get when business people make decisions on technology projects.

I would rephrase the conclusion: This is what you get when people make emotional decisions on something.

It doesn't matter if it is a technology project or not, it is just an emotional response. There are some professional categories (doctors, funeral or wedding businesses, churches, etc) that exploit this weakness. When people are emotional charged, they don't think about the unrealistic prices of some treatments or services, they are even happy to pay them. Without this emotional charge, they would bargain and complain about it.


What are some good strategies to avoid this problem (asking as an engineer looking upwards to management)?


There is only so much you can not know about what your people do. And try to hear the people in the trenches, not their PM with slideshows.


It's not just "business people". This was published the same year as the Agile Manifesto. We know which route developers chose.


> This is what you get when business people make decisions on technology projects.

I think this is a problem with people in general, not just business people. There is a huge amount of psychological literature on biases and one of the things I find most interesting about it is that even if you're aware of a bias your thinking is still affected by it.


That's a variant of "there's no such thing as bad exposure".

When the person with the fubared project gets his daily beating from the big shots, eventually he'll figure out how to look like a hero while blaming everyone else.


Nassim Taleb provides a good example in his book Fooled by Randomness. Imagine that a politician passed a $1 billion requiring all airline cockpits to be locked prior to 9/11. That politician would probably be criticized and then voted out of office.


This is what I think about government in general. People complain about it but we live in country (if in US) where most stuff works, most stuff is safe. We don't have ferry disasters, we don't have serious virus outbreaks, the power almost never goes out, and the water is drinkable. This is because people work everyday to keep this up.


To be fair, nobody really complains about the CDC or the local water treatment plant, and none of that stuff consumes an inordinate amount of tax dollars.

Most of the things people complain about are at least nominally legitimate. We spend too much money on healthcare (and thus medicare), and too much money on the military, and in consequence the taxes on the middle 80% are too high. The Department of Education micromanages local schools excessively. There is too much bureaucracy in general. These are legitimate complaints.


I don't disagree there is waste. I would say there are people who complain about non military spending though.


They might not complain about them, but they do take them for granted.


Sun Tzu talked about this 2500 years ago in the Art of War:

""" What the ancients called a clever fighter is one who not only wins, but excels in winning with ease.

Hence his victories bring him neither reputation for wisdom nor credit for courage. """


In my experience this goes way beyond productivity. I know you have clients and projects and feature requests and bugs to fix and meetings to attend, but how about that backup scheme for that new project, the one that was bumped to "post-golive"? Or the lack of free disk space monitoring? Or that everything for these three services run as root and use someone's primary AWS keys?

In software dev or ops, these are the things in my mind that sit in the gulf between "producing" and "improving". Actually, they sit on the other side of improvement, because given a free hour, I'd rather spend it learning something than fixing a logging service.


There is a video presentation on the topic from the author: https://www.youtube.com/watch?v=xCkyPc72XNQ#t=97


An extremely good primer on quality at the organizational level. Even if it's your "job" to make quality products, the visibly rewarded metrics are often counter to the desired result.

The assumed management strategy is to individualize results, punish poor behavior (often directly related to errors or defects) and reward perceived good behavior ("hard work") at an individual level. In reality, this causes a complex wash of quality-undermining behavior: politics, infighting, de-motivation, perceived unfairness, and worse. Despite the well-proven and scientifically understood reality, managers and even employees are entrenched in the surface level, when in reality the problems are deeper and higher level.

This is the wisdom of W. Edwards Deming and his contemporaries. The fundamental epiphany is that quality is the result of how companies are managed. That it is a result of four things: an understanding of knowledge itself (primarily, that our prior assumptions about work are incorrect) -- an understanding of psychology (behavioral science and real motivation) -- knowledge of statistics (for the ability to optimize process, and the knowledge that all behavior has a random element, even that of individuals) -- and finally, a deep understanding of systems and their effects.

This thinking brought Japan out of its post-WWII recession. It was the origin of the Japanese quality economy—of Toyota, Honda, Sony, Panasonic, and more Japanese companies with quality products at the core of their success. The Deming prize is still given to companies which embody the values of quality.

It worked in Japan because of a cultural readiness for a systemic worldview. It does not gain traction in the US for exactly the opposite reason: we are a culture of rugged individualists, who, despite scientifically proven truth, will not drop our core idea that individuals are the cause of both our successes and our failures. This is especially true for upper management.

To be successful, as companies and as a nation, we need a major cultural shift toward systematic quality. We need companies that understand that most of the deficit in the quality of products comes not from poor individuals themselves, but poor links between them, poor management of them, and the poor systems within which they work. Simply lack of attention to systems and process in general causes a standard of organizational chaos, where we truly do rely on the excellence of individuals to keep companies afloat. This is familiar to all of us, but surely there's a better way.

If every "whose fault is it" question would be translated to "what, at its core, caused this fault"—every company would benefit—and that's true even if some fault can be found with an individual. Display extreme restraint, as the better course of action is still systemic. And this isn't a strategy of simply "blaming management"—instead, every person in the company needs to work together to improve the system surrounding them, and when every person understands why that view is important and why they're a part of it, then the improvement will begin to multiply.

This is true in software as it is in manufacturing, and I think the industry is ready for a new body of work to translate Deming's ideas to the software field. Skip Six Sigma, skip TQM—too dogmatic and heavy. Focus on Deming's core ideas: the importance of systems, understanding of key underlying concepts, and the implementation of them with good leadership from the top. It's a new old way, it is a better way, and we need it now.

http://en.wikipedia.org/wiki/W._Edwards_Deming


It's famous: "The wheel that squeaks gets the grease."

Well, the title says something similar: Wheels that don't squeak don't get the grease.

At least in the US, a fairly strong norm is to be rock solidly practical which can mean operating close to the line of actual irresponsibility and following "Never do today what can delay until tomorrow." Is such thinking common? In the US, yes. Good? Not really.

Or, wait until there is a really bad passenger airplane disaster before implementing some new safety procedures.

Wait until there are actual sick people with a threat of an epidemic before implementing the long well understood public health measures.

Do much the same for desktop computer security, auto seat belts, actions against smoking, and dozens more.


The scientific term for work needs to get into the general culture. You can expend a whole lot of energy without doing any actual work. "Working harder" is actually "expend energy" while the actual work done is the same.


The pdf seems odd to me --- all "fi"'s are invisible, e.g., "fi"rm, signi"fi"cant, bene"fi"t...


You probably need a font that covers the relevant ligatures. Does "fi" display for you? That's the Unicode codepoint U+FB01.


Futurama said it well: "When you do things right, people won't be sure you've done anything at all." (2002)


Isn't that called a "paycheck"?


Well, you can increase your paycheque by getting promoted... If the 'heroes' are the ones that get promoted, then they are getting more then the unsung heroes.


I suppose that's not a terrible metric in theory, but by that token the people who make mistakes are, in the general case, still taking more credit since they are also are paid to fix the problems.


But one example of the dysfunctional situation present in that case. Many other examples of individualistic politics exist.


A "paycheck" is too individual to successfully result in quality in an organization larger than a couple dozen.

The correct method is to understand the system of work, and implement good process to continually improve it. Quality needs to be built in from the beginning.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: