
Why Is Your Team Falling Behind? Ask ‘The Penny Game’ - ingve
http://spin.atomicobject.com/2016/01/19/penny-game-software-dev/
======
csense
Part of the problem is that software is never really "done" \-- older software
needs a ton of maintenance and support, and new features are restricted by
existing code's technology choices, design decisions and architecture.

So pennies in later systems never fall onto the truck and get delivered to the
customer, rather they fall into the final state and randomly teleport back
into earlier stages as teams that implemented them start getting support work.

Unfortunately you're not going to be as productive when you're developing a
new product against a background of supporting multiple existing products /
customers, as you were when you were developing your startup's initial product
without much else going on.

~~~
humanrebar
> Part of the problem is that software is never really "done"

Absolutely. The cognitive dissonance is bad enough that we might be overdue
for a change in vocabulary, starting with the word "done". Even hypothetically
perfect code needs to get ported to a new architecture, OS, VM, and/or
compiler at some point.

I think we, as technical experts, do a terrible job explaining that every
piece of code _old and new_ has an ongoing cost. The discussion points should
be about which costs are higher and whether trade-offs are worth it. Are the
ongoing costs (is the team even aware of what they are?) lower than the cost
of change (how do we know it was worth it? can we change our mind if it
wasn't?).

While the article brings up many good points about bottlenecks, estimation,
and allocation of resources, I find that bad mental models of how software
development works costs far more, albeit in subtler ways, than "Heather can do
12 units of work but we only gave her 8 and she blocked a lot on task Alpha
instead of moving on to something else".

Though your mileage certainly varies.

~~~
alsetmusic
Continuing that line of thinking, one of my epiphany moments when I first
arrived at a dominant player was the acceptance that we would ship with bugs.

The way to stay at the front of the race is to prioritize your bugs and fix
the right ones in the time that you have because if you think you'll get them
all, you'll never ship. Get the ones that have the highest probability of
causing your customers pain and know that you aren't ignoring the ones that
are scheduled for your next maintenance release. Debate intelligently if you
think something needs to be reclassified as a higher priority and learn to
accept that there might be a couple that you don't win so you can get on with
covering as much of what remains as possible.

------
gclaramunt
No! No! No! No! WHY?? We spend years saying "software development is not
manufacturing", and now it the name of "agile" we allow all this to crept
back. Your team is not falling behind, your estimates were wrong!!

~~~
kyllo
That response is ineffective, because the managers will simply pressure the
engineers to give better estimates.

The penny game (where I work we call it the beer game) is a good learning
exercise, but in software development you really don't want to be having
different "stations" handing work off to each other in the first place,
because that leads to delays and quality issues. Each station will develop its
own little backlog and pretty soon a feature cannot make it from start to
finish without spending weeks waiting in the design queue, then the dev queue,
then the test queue...

Instead, you need to have a small, cross-functional team focusing on the same
piece of the product at the same time, collaborating and giving rapid feedback
on each other's work--that's what agile and XP and scrum are for. And you need
a pull system instead of a push system, to regulate the amount of work in
progress--that's what kanban boards are for.

~~~
mtviewdave
But doesn't specialization naturally push towards the creation of stations? If
your cross-functional team contains both a database engineer and a UX
designer, I don't see how either can help with the other's job.

~~~
kyllo
It really can help if they are actively communicating with each other--the
user interface is how the data gets into the system, so if the database
engineer understands the interface it can provide context for where the data
comes from, what it means and how it is used, and that can inform data
modeling decisions.

Understanding how the data model is laid out, how much structure is needed,
what fields are required/optional and what types they are etc. can also inform
UI/UX design decisions.

And this is an example of two roles that are at opposite ends of the
frontend/backend spectrum. For roles whose work products interface directly,
there's an even stronger need to collaborate.

Specialization is good in that it allows people to focus on using their best
skills, but it's bad if it fragments the product development team and inhibits
communication and decisionmaking.

~~~
mtviewdave
Certainly active communication and the prevention of siloing can help speed
development and reduce defects. But the Penny Game illustrates that backlogs
and queuing issues will occur even if each station is in perfect communication
with all the others. So you can't eliminate queuing issues with close
collaboration and rapid feedback. You can only eliminate it by eliminating
dependencies and specialization, which I don't think is possible in modern
software development.

~~~
kyllo
Well yes, on a basic level when you have specialization and division of labor,
you will always have dependencies.

But when you tighten the feedback loop and shrink the batch size of the work,
you make the dependencies smaller and more granular, so the queue gets smaller
and fluctuates less, and you get better flow of work through the process.

Here's a simple visual that illustrates why this is the case:

[https://lh4.googleusercontent.com/QGyl3JPoMH1wU0pNoGIsucQCOf...](https://lh4.googleusercontent.com/QGyl3JPoMH1wU0pNoGIsucQCOfLu0R9MKZ4kfj3ioNBA_5gdXLVM_YWQ7kRSlKO3dxzzHGyrhMi68l8MjmYjriUyYRQKbUkVuvw2KMeHWRlSjGQ8ey__OfpC8JDTeDEosA)

------
robodale
Having been both a Manufacturing Engineer and a Software Engineer, there are
similarities between the two scenarios: work-in-process within a manufacturing
facility and whatever features/functionality a given software development
department is creating.

I would create detailed step-by-step job instructions for each station in the
manufacturing line, along with expected times each unit should take at each
station. It was trivial to compare expected versus actual, then dive into the
details of what's going wrong if the two values are not matching up (new
vendor causing quality issues? machine troubles?).

With software, those "job instructions" are little more than the specification
hammered out by (hopefully) someone who knows the product, domain, technical
capabilities of the software development team (including design and QA). It's
rare. Very Rare....any single person can estimate exactly how long a sprint or
anything larger is going to take. One wrong technical hurdle, and every
estimate is out the window.

For the "production worker" on the software dev team, just following those
specs wont get them to the finished product. Their knowledge, how well the
team can support lack of expertise, shifting priorities...these all create
logjams.

~~~
EvanPlaice
The difference is, writing code IS the specification process. The code itself
is a functional design spec.

The persistent assumption that writing code can be handled like a production
process is flat-out wrong. The software equivalent to production is the
build/test/deploy phase which any intelligent developer will automate.

The concept of a 'production worker' programmer seems ridiculous. Somebody
more experienced researches the domain, comes up with solutions, specifies all
those solutions, and the 'production worker' programmer is left to to what
exactly? The engineering equivalent is handing a draftsman a hand written
drawing and telling them to implement it in CAD. It's good for practice and to
get new people up to speed with the process but it doesn't provide much value.

The reality is, unless the team is using an easily repeated process like
churning out Wordpress sites from a scaffold generator, there's no such thing
as 'job instructions' on a programming team. Each and every developer on the
team is assessing the domain and trying to hammer out a specification for the
production process. Any part of the process that's easily repeated should --
and ideally --will be automated away.

It's not an easy process to measure because specification requires extensive
research and discovery. Bad assumptions are made and corrected. Some
approaches require new and novel approaches. A significant amount of the work
comes with a great deal of risk.

Throw a few programmers into the mix who lack the battle hardened experience
and wisdom of previous successes/failures and they'll generally drag everybody
else down.

In short, software development is not a production line. It's hard to define
because it requires a lot of unknowns and even more unknown unknowns.

------
brudgers
Queuing theory is, like statisical thinking, is a topic where I feel like I
understand what will happen but I know that my intuitions are wrong.

Which is why everything always takes longer than I think it should.

~~~
jib
Spolsky quoted someone saying "Google uses Bayesian filtering the way
Microsoft uses the if statement."

I feel like there is room for an analogous quote about using the Poisson
distribution compared to capacity reasoning.

------
poof131
Perhaps the main problem is treating a team of professionals like an assembly
line. Execs and PM dream up the idea, throw it over the fence to design who
mocks it out, who throws it to the backend to build the api, who throws it to
the front end to piece together. What could go wrong with this process? Oh,
maybe you have a meeting so people get their input in, but the problem is you
don’t really know the important parts you are missing until you start
building. This isn’t the way to organize an elite team. Good people don’t want
to be a cog in a machine.

------
jordigh
Not to be confused with Penney's game, which is a different kind of
fascinating thing in itself: if two players each pick a sequence of heads and
tails and win if their sequence shows up first, the player who picks the
second sequence after seeing the first sequence has an advantage.

[https://en.wikipedia.org/wiki/Penney's_game](https://en.wikipedia.org/wiki/Penney's_game)

------
nradov
For a more detailed explanation of the importance of minimizing work in
progress (i.e. inventory) I highly recommend "The Principles of Product
Development Flow" by Donald G. Reinertsen. It contains a thorough economic
analysis based on queuing theory.

[http://www.leanproductflow.com/buy-the-
books/](http://www.leanproductflow.com/buy-the-books/)

~~~
Chris2048
Or [http://www.amazon.com/The-Principles-Product-Development-
Flo...](http://www.amazon.com/The-Principles-Product-Development-Flow-
ebook/dp/B007TKU0O0) without the affiliate link...

------
tunesmith
This is a fun game because it also is related to a lot of counterintuitive
things about constraints and premature optimization.

If you optimize something that isn't a constraint, the problem isn't just that
you are wasting effort - it's that you might also be unwitting _increasing_
the backlog of another part of the system. So it's not only "not helpful", it
could sometimes even be harmful.

That's why Goldratt et al emphasize constraint analysis so much - analyze the
entire system (usually using causal analysis; why certain parts of the system
explain other symptoms), identify the current core constraint, invest all
spare efforts to alleviate _that_ constraint only (and _not_ other
inefficiencies), and then re-analyze.

I don't believe this is necessarily counter to Agile! It's just that a lot of
people implement Agile badly. Which isn't a problem with Agile exactly, it's
just that there's a lot about complex systems that are counterintuitive. You
can't really be reductionist about it.

------
AndrewDucker
This is something that more managers need to understand - that even slight
variations in how long it takes for work to be produced can have a major
effect on throughput.

------
ikeboy
The first thing that comes to mind is to allow different stations to do each
others' jobs when needed. Sure, productivity will be lower when doing
something you aren't used to, but isn't it strictly better than being idle
because there's no work on your level? (Barring problems because you didn't do
the work well, but this is a simplistic model anyway.)

I wonder how the game would end up if stations could work on other levels for
a slight penalty to production for both stations, only kicking in when one is
idle over some threshold.

~~~
marcosdumay
Well, adding that to the model would make it not apply to software
development. You can't just add developers to a task.

~~~
ikeboy
Why not? Or rather, what hit do you expect productivity to take, and for what
level of idleness would that still make sense?

If the problem is that some developers are literally idle because their task
is done, they should do something else.

~~~
frandroid
The different stations don't represent developers, they represent discrete
tasks. So that could be designer > coder > qa, or dev stack 1 > dev stack 2 >
dev stack 3. A coder waiting for a design won't just go and design. A
developer with Stack 3 might be familiar with Stack 2 but might not know the
best practices from that stack, introducing bugs or funky code that will
create more work later on. If all your developers can do all the tasks, the
graphics in this story don't apply.

------
overgard
I think this pretty much cuts to the heart of what's usually wrong with the
whole "agile" thing -- by micromanaging programmers to such a degree, instead
of being able to use that downtime to do things that practically _everyone_
knows needs to get done, you have people sitting on their hands because
everything has to go through the "backlog."

IMO the solution is to stop treating programmers like cogs in a machine, and
actually give them some amount of autonomy to make decisions.

~~~
ebiester
On my agile team, we will say, "We see a problem here." We create an issue and
start working on it, and alert the product owner as to why it is a problem and
needs to take priority. We have that respect because we deliver working
software. We are not the "fastest" team, but we get the latitude to make
things better because we work with adults.

We refactor as we go, even if the business value could have been delivered
"faster" by compromising long term value for short term goals.

But we also let them do their job and trust them to understand the strategic
goals to the business better than we do.

~~~
overgard
Yeah, I recognize there's some places that do agile in a good way (hence the
qualifiers I put in); I've just been burned enough that I tend to be deeply
skeptical of it. What I've noticed is that some managers fall in love with
charts and measurements (because they can use those metrics to impress others
in the organization), and anything that can't be superficially measured ends
up either being devalued or outright banned.

I remember one place I worked, they brought in consultants to "help" us create
an agile process. After about four months or so, they declared the operation a
success because the "velocity" had increased by 4x. (Which would be incredible
if it were true). In reality, we were just measuring four times more things,
and development was going much slower, but those charts sure looked nice.

------
gr3yh47
...because you're all reading hackernews articles instead of working

~~~
purplerabbit
Feeling guilty? ;)

------
lukev
A really great book about this concept, applied specifically to software, is
_The Phoenix Project_ ([http://www.amazon.com/The-Phoenix-Project-Helping-
Business/d...](http://www.amazon.com/The-Phoenix-Project-Helping-
Business/dp/0988262592)). I recommend it highly.

------
BatFastard
How do you start the online version of the game? Doesn't seem to respond to
anything.

~~~
tga
Hover your mouse over the two hidden blue panels on the left side.

------
jdietrich
A similar example regarding quality is Dr Deming's famous Red Bead Experiment.
It is a fascinating insight into how dysfunctional management systems operate
and perpetuate.

[https://www.youtube.com/watch?v=ckBfbvOXDvU](https://www.youtube.com/watch?v=ckBfbvOXDvU)

------
adekok
I like the simplicity of the simulation. My take-away is that each station
needs to plan for average throughput, not for capacity.

The problem is that you don't know what average throughput is until you run
the system in production. So you might have either too much, or too little
capacity.

------
ssharp
It's been quite a few years since I read The Goal. I can't remember now if I
found the whole family narrative interesting and it kept me engaged in between
the operations education, or if I found it a distraction.

