
Yes, Estimate Software Projects - gregdoesit
https://blog.pragmaticengineer.com/yes-you-should-estimate/
======
jasonpeacock
This whole article rubs me the wrong way. It's holding up their failures in
software planning & management as best practices. And while they are normal
practices in the industry it doesn't mean they are good practices.

They start off by talking about how important estimates are, and how they
always meet their estimates across many types of projects/deadlines.

Then they admit that "delays are normal and expected" \- so you're not meeting
your estimates.

Then they admit that "they delivered something different" \- so you're not
building what you promised.

Then they admit that "they "spent too much time on non-deliverables" \- so
you're spending the last months pre-deadline working hard to catch up on
something that is already behind schedule and doesn't match the original spec.

But it's all OK because they kept the stakeholders updated with frequent
communication and conversations...basically the same result as if they had
_not_ estimated a date but instead focused on iterative delivery and constant
feedback with the customer. Sound familiar?

> Suddenly, the whole team became focused, distractions were all gone, and we
> moved at a faster pace than I've ever felt the team do so.

Why was the team not focused & distraction free from the start?

How could you achieve this efficiency without needing a deadline hanging over
your heads?

~~~
nilkn
As an engineering manager and director, I’ve found a lot of the value in
planning (including but not limited to estimation) is having the team,
stakeholders, and executives all engage in the exercise of thinking deeply
about various ways the next 6-12 months might go.

Some folks tend to think that if a plan is produced but not followed exactly,
then it wasn’t worth producing the plan. But actually having produced the plan
is sometimes what makes it easy to deviate when complications arise. If a new
decision is made that affects the plan, everyone can see exactly what the
compromises are and be on the same page.

~~~
bsder
> As an engineering manager and director, I’ve found a lot of the value in
> planning (including but not limited to estimation) is having the team,
> stakeholders, and executives all engage in the exercise of thinking deeply
> about various ways the next 6-12 months might go.

I always despise these kinds of exercises.

We _know_ how to roll up estimates correctly with statistics--even moreso if
we have historical data.

The problem is that management _always_ hates the _accurate_ number.

And that's where the dysfunction sets in.

While everybody else rolls things up "normally", I almost always roll up an
"accurate" timeline and try to inform people up front. It rarely wins me
friends, but it sometimes protects my team when things are going haywire
later.

~~~
knightofmars
A co-worker and I were asked to write up an estimate for a new project. We sat
down, walked through the components (best we could at the time) and came up
with an estimate (which was likely smaller than the actual given the
unknowns). We provided the estimate to leadership. They decided not to do the
project. Next, I hear from the head of engineering, "You can't go giving huge
estimates to leadership. It makes us look bad." It was one of those moments
that reinforced my belief that the "image game" and reality are in conflict
and people who enable the "image game" are one of the biggest problems in
software estimation.

~~~
s_gourichon
Yes. Let's rephrase the idea a little more formally.

Few hypotheses are necessary for a thought experiment that reproduces the
issue.

Given a project:

* some people/teams produce accurate estimates

* some people/teams produce overly optimistic estimates

If decision-maker is not aware, it can easily conflate _optimistic estimate_
with a _more efficient team_ (or just more enticing).

If the possibility of overly optimistic estimates are not taken seriously, the
more accurate team can seem just less efficient / lazy / trying to charge more
for the same work / etc. For example if the decision-maker is actually a
client comparing several suppliers.

Extra risk if decision-making is diluted among many people (harder to get
clear and consistent view).

This puts the teams in a competition similar to the prisoner's dilemma. Honest
estimates get punished, treacherous estimates are favored. No wonder so many
projects are over-budget, when they are not too over-budget to complete at
all.

The only actual hypothesis here is: "decision-maker conflating optimistic
estimates with better value". It seems very common. If decision-maker doesn't
fall into this trap, the problem disappears, right?

What can we do to shift the equilibrium? A manifesto?

------
journalctl
> Have you noticed how Apple ships most of their big bang projects at WWDC, a
> date they commit to far ahead?

Is this why so many Apple products have been increasingly low-quality? Instead
of committing to quality, they commit to arbitrary dates because some bean
counter decides quarterly profits needed to be higher.

I don’t disagree that we should estimate, but I think we should be realistic
about just how difficult estimating software projects is, especially now that
we’re “agile”, which really means “there aren’t any explicit requirements but
there _are_ requirements, we just don’t know what they are and also they
change every sprint so good luck”.

If we want to estimate software, we need to have more predictability. I can
estimate that it takes three days to fix a bug, but what happens when I pull
those covers back and see an unholy Eldritch abomination? I could redo my
estimate, but what if I run into an arbitrary quarterly profit-driven date?
“Well, just get it done.” Then we nail some more legs to the dog and the cycle
continues.

~~~
delusional
> there aren’t any explicit requirements but there are requirements, we just
> don’t know what they are and also they change every sprint so good luck

What a sad way to view your profession. The requirements aren't hidden from
you, and they don't just randomly change. You can't get them up front because
no one has any idea what we are looking for. Solving a small problem usually
reveals more of the problem.

I don't think you believe in long requirement documents either, but I
sometimes hear arguments very similar to yours.

~~~
ThalesX
What a sad way to hammer someone’s view of his profession based on a comment.

> You can’t get them up front because no one has any idea what we are looking
> for.

Is this requirement implementation or R&D?

While it’s understandable for R&D to be exist within the ‘no one has any idea
what we are looking for’, to fit all software development into this paradigm
is to have a pretty lax understanding of how most software operates...

> Solving a small problem usually reveals more of the problem

This is exactly what I think the parent post was arguing when shying away from
estimates.

------
klenwell
I hate deadlines simply because organizations I have worked in tend to assign
them indiscriminately without any reference to reality. Upper management wants
to set a deadline for a big project? Then they need to be prepared to commit
and compromise. While developers continuously deliver, product owners and
stakeholders need to continuously review, reevaluate, and adapt.

In Thinking Fast and Slow, Daniel Kahneman offers my favorite object lesson on
the topic:

[http://txti.es/kahneman-outside-view](http://txti.es/kahneman-outside-view)

He sums up:

 _This embarrassing episode remains one of the most instructive experiences of
my professional life. I eventually learned three lessons from it. The first
was immediately apparent: I had stumbled onto a distinction between two
profoundly different approaches to forecasting, which Amos and I later labeled
the inside view and the outside view. The second lesson was that our initial
forecasts of about two years for the completion of the project exhibited a
planning fallacy. Our estimates were closer to a best-case scenario than to a
realistic assessment. I was slower to accept the third lesson, which I call
irrational perseverance: the folly we displayed that day in failing to abandon
the project. Facing a choice, we gave up rationality rather than give up the
enterprise._

The suggestion that the enterprise should have been abandoned all together can
be misinterpreted as fatalistic, even nihilistic. I take it to mean that
you've got to do your research (the outside view), rigorously define your MVP,
and then work from there. If your deadline is fixed, be ready to scale back.

His distinction between the inside and outside view is the key. For a project
to have any hope of realistic timelines, it needs to be understood and the
outside view needs to be applied.

~~~
davidjnelson
> If your deadline is fixed, be ready to scale back.

True. Honestly, what else can you do if a deadline is approaching than to cut
scope? Even enforced death marches aren’t necessarily going to solve it...

~~~
bastijn
You have the four knobs to turn. Resources ($, FTE), scope, deadline, quality.
The dilemma typically is that the one calling the shots wants to fix three of
them and does not accept the fourth knob has to be turned. Another is that
close to a deadline the resources knob doesn't matter anymore unless the
deadline knob is turned. And when the deadline knob is turned it makes the
scope knob feeling very attractive suddenly to the same kind of people.

In such situations having people skills helps. You can say no in many ways but
you have to try and understand their problems as well. That big conference,
competitors having announced something recently, you name it.

------
Osiris
My current employer switched from SCRUM to KANBAN. We're broken up into person
squads. Epics are created by product by engineers are involved in kickoff and
story breakdown.

We use KANBAN cycle times to estimate delivery dates. For example, my squad
completes 85% of stories in 5 days. Given that, the number of stories in the
epic, and number of engineers, it's easy math to calculate an approximate
completion date.

It's important to note that we've convinced management that these are
approximate completion dates, not deadlines. The dates are used for planning
upcoming features and which squad will be free first to take it on. However,
there's still plenty of adaptability.

Of all the places I've worked, this model is one that I've found to be the
most productive.

~~~
mikewhy
I really, really like Kanban as well. Seems the hardest part is selling it to
upper management.

~~~
davidjnelson
Because they want hard deadlines. I don’t think this is something that can be
“fixed”. I think it’s something you discover in an interview and use as a make
or break join/pass decision if you truly care about it.

------
alecbenzer
I'm intuitively on team #noestimates but trying to have an open mind/not dig
in my heels.

I feel like no one does a convincing job at communicating why estimates matter
(and how much value you get out of how precise of an estimate).

Fundamentally, an estimate (or any piece of information) is really only of
value if you're going to do different things based on learning it. E.g., given
an estimate, you might:

* Prioritize one project over another based on ROI.

* Know when to communicate that something might be available to users (though that's still not _intrinsically_ valuable, you have to have some kind of marketing/sales-cycle based reason or something that it matters).

* Know when to begin supply-line stuff so that a factory is ready to build a thing as soon as possible (but no sooner, since then maybe you're holding up parts of the pipeline from doing other things? maybe?)

I feel like so often people try to underscore the importance of estimates but
don't say anything remotely close to the above. A lot of times it's just "this
is important/urgent". Ok, so? If it's urgent then that's even more reason to
start actual work instead of spending time planning, if you can't point to a
concrete thing you'll do differently having known the plan.

Even when people do talk about the actual reasons for wanting an estimate,
it's always totally in the abstract. The precision of an estimate is always on
a "delivery day" level, even if that level of precision is totally unnecessary
for what this particular estimate was used for.

I think the real reason for this amount and kind of emphasis on estimating
really has to do with:

1\. Deadlines make people work harder/faster.

2\. It gives people sticks with which to measure people/teams.

I think #2 is mostly straight-horseshit. #1 is maybe somewhat true, but I
think it's not nearly as good as other ways of getting people motivated to
work harder which have fewer negative externalities (e.g., more autonomy,
instilling more of a sense of ownership).

~~~
yibg
One reason for rough estimates is for making investment decisions. If we have
10 things to do but can’t do them all how do we choose which to do? The what
the thing does points to the return piece but we’d need the investment piece
too to evaluate which ones are better value.

~~~
alecbenzer
For sure, that's what I meant by the ROI bullet.

I just often feel like people want estimates that go beyond the level of
precision/accuracy you'd need to make a relative ROI call. And just in general
don't do a good job motivating _why_ they want the estimate.

------
generatorguy
I have to have all of the control system software ready for a power plant the
second construction is finished enough to begin turning things on and
commissioning them. This frequently overlaps with electrical, mechanical, and
civil works still in progress. Not starting as early as possible means not
finishing as early as possible which means holding up the transition from a
construction project with construction loans to an operating asset earning
revenue with much more favorable financing, which is significant in hundreds
of millions of dollars projects.

All of the software needs to be ready to start commissioning on an unknown
date. A lot of the programming can’t start until equipment details are
received, which makes the whole thing a game of identifying what information
you need and pointing out that if you don’t have it by x date your
deliverables will be late, or you’ll have to work overtime or bring more
people on it which will cost extra.

Having a good estimate for how long each piece of the programming will take
and what inputs are required to finish it is crucial to having everything
ready to go on D day or a bulletproof paper trail of how you’ve been fucked
over by other people who couldn’t deliver, because those liquidated damages
are no joke.

To me time estimates are a major component of engineering.

Edit: remove snark

------
dstroot
Cost, quality, and speed. Pick any two. You can’t have all three. There are
several modes of delivery. Some projects have fixed delivery dates because
it’s externally imposed. For example a regulatory requirement. Some are
internally imposed - an executive imperative. Projects with fixed dates need
dedicated resources and a highly functional scope management process. Others
are “innovation/exploratory”. For example “let’s build a bitchen website”.
What does that mean? What is the definition of done? This requires an agile,
“integrate until we are satisfied or run out of funding” project that may or
may not have dedicated resources.

No matter what, in a corporate environment you will need to be accountable to
cost, which is simply humans * time. So some type of estimate will be needed.

~~~
pas
Sure, estimation is needed. But the more important thing is that project
_management_ must involve the constant updating of those estimates.

After all estimates should be data based. As more data comes in they should
get more accurate not less.

If they don't that's a sign of basing them on hope or whatever ideology
someone found instead of hard truths.

------
awinter-py
This is such an important topic and missing skill in the tech workplace.

It's impossible to prioritize work without estimating the cost & value of
projects, and teams that don't prioritize are always overworked, playing
catchup and are unhappy.

What the author says about 'we shipped different functionality to what we
planned' is really important too -- beginning managers aren't willing to have
this conversation but scope de-creep is the only way to deliver quality work
on-time.

It's not enough to play planning poker, you need to have ICs in the room for
spec design and make time to identify & resolve unknowns before committing to
features / timelines.

~~~
shmundada
> make time to identify & resolve unknowns before committing to features /
> timelines.

I find it difficult to identify unknowns before I actually start working on
it. Personally it makes sense to just give it a go as estimating it correctly
may be as time consuming as completing it. Especially true for one off feature
or bugs in large codebase.

~~~
awinter-py
That's fine for smaller chunks of work, and you're right that resolving
uncertainty is not cheap.

But if you're doing resource planning for a large organization and you need to
decide between high-value, long-timeline projects, that decision is
irresponsible without some homework up front to validate the timelines.

Large projects involve high- and low-uncertainty work and it's useful to do
some of the former before spending resources on the latter.

Your estimate still won't be right, probably, but the chance of uncovering an
important problem is worth it.

~~~
pas
Agreed. Usually the problem is that technical folks are involved too late.
(They are the ones that actually think through the requirements after all.)

------
ken
I'm fine with either time- or complexity-based estimates _as long as they 're
treated as estimates_. I've not worked at any software company that did.

What are the consequences of missing an estimate (especially one I did not
create)? If it's "stay late until it's done" (with no OT because you're
legally "exempt"), that sucks. If it's "your career is penalized, the same as
if you were incompetent at the task itself", that sucks.

I'd rather go back to a low starting salary, and earn OT when my manager
decides to keep me working late. The money isn't as important to me as
aligning our goals.

------
neltnerb
I think it's good to write down your guess for your own future reference in
improving your own guesses, and at some point you'll be good enough to do this
and not get burned (as a contractor, I'm assuming).

I have been doing this a while and am not good enough to avoid getting burned
without building in a 3-4x buffer. Sometimes I end up finishing a project in
10% of the expected time, but at least it's not over 100%?

I don't think that's very healthy for anyone involved though, it's better if
for projects I can't estimate well we just do hourly so that I'm incentiviced
to spend lots of time making it great and they're incentivized to give me more
work in updating the spec without needing to renegotiate anything.

But yeah, I'd say yes estimate and do it in writing but keep it to yourself
until you're actually good at it. It takes practice, but honest logs of your
past projects will elucidate over time. Like how whenever my biologist
coworkers would plan an 8 hour project, 11 hours in I'm just like... how did
you not plan for cleanup and prep?

Some things are just not obvious right away, but can become clear in
retrospectives that inform future estimates.

------
torgian
Reading the comments ( and the article ) I think people are forgetting about
the customer causing problems.

For example, what do you do when you and your customer agree on certain
features by a certain timeframe and then... that customer turns around and
promises something to someone else, then asks you if you can do that _without_
_consulting_ _you_ _first?_

If you refuse, you cause trouble for your customer, and potentially yourself.
And then there's cultural considerations. And then your customer gets
distracted by this and that, and you lose control of you customer?

This kind of thing happens. Plenty of companies are square in the "customer is
always right" camp, which I think is a _bad_ idea. The customer is not always
right.

This is why I think planning and estimating is a good thing, but controlling
your customer is also important to keep in mind. You have to be willing to
compromise, yes, but you also need to say "no" sometimes, or at least say
"Well, let's finish this first, get the product out the door, and put that in
version 1.5".

However, I feel like a lot of people are too scared to do that.

~~~
learc83
It's not just customers. The same thing happens with internal stakeholders.
I've seen products where there is basically zero actual software design
because development is driven entirely by short term feature requests.

------
joe_blank
Being pretty young and just getting used to making estimates, this was a
wonderful read!

Having to guess how long a project will take is still very scary for me. Is
there any way to get over the fear of others disagreing? I always feel like
I'm about to embarrass myself by either estimating to high or to low...

~~~
adverbly
It sounds to me like what you really should be learning isn't how to estimate,
but rather how to change your mindset on disagreement. Disagreement is a good
thing, not a bad thing. Learning how to change your own mind is a very useful
skill that requires disagreement. When organizational disagreement stops, so
does organizational growth. Be friendly and kind to people who you disagree
with, and ask lots of questions to better understand their thinking.

------
UK-Al05
One thing he misses out is that story points, eventually get turned into
estimates using velocity. They are estimates, just a different approach based
on past history.

~~~
meh2frdf
They are used for forecasting

------
jamesfisher
This article misses the biggest reason estimates are important: to make
decisions between projects and approaches. Prioritization function is
impact/effort. If you have Approach A for which you estimate 10 person-weeks,
and Approach B for which you estimate 50 person-weeks, you should pick
Approach A.

Real time estimates, rather than abstract story points, are necessary to
calibrate your future estimates, by measuring how over-optimistic you were
with past projects.

------
dllthomas
I think estimates of effort required, estimates of timeline, and commitments
to timeline (deadlines) are three different things with only loose
relationships that are far too often conflated - to significant pain. I'm
trying to play with this a bit in my personal process (and talking to my team
about it) but don't really have any hard results yet.

~~~
milesvp
Very much this. I try to never give an estimate without giving 2 values. The
expected effort, and the expected calendar time. Sometimes I’ll also add
expected complexity since that adds a lot of risk to both estimates. There are
lots of things I’ve done that were quite easy, then I spend lots of time just
sheparding the feature into production. Especially if you’re working with
other teams, or vendors, and can expect 24hr delays for every email round
trip.

In fact each department or vendor involved in a project will easily double
calendar date, and often complexity as well. At a previous job we
automatically increased our t-shirt size for every vendor just to keep our
estimates honest. And there was only like one vendor who we would make
exceptions for since they were always amazing to work with.

------
segmondy
If you can't ESTIMATE your software project, then you have no idea what you're
doing.

An estimate is NOT A DATE, an estimate is a DURATION.

If you estimate 50hours and other things come up that eat into your time, you
continually adjust the DATE, the duration shouldn't change. If you depend on
an external sys and you estimate N hrs, but that system takes N+M hrs, then
your duration date moves forward M hrs.

If there were unknowns, risks, as you uncover them, you move the date and if
you communicated the unknowns and risks, then you can adjust the duration by
adding the new info.

Ideally what you wish to do is eliminate all the unknowns and risk by NOT
doing them. You're doing DEVELOPMENT not RESEARCH. Development is simply
pulling in all your prior knowledge on how to solve problems and tacking
similar or new problems.

Don't mix your R&D.

~~~
QualityReboot
I don't understand how solving a new problem that you haven't solved before is
somehow development instead of research. All software is research. If someone
had already solved it and built it, you could just copy their solution.

I've been doing dev work my whole life, and the only thing I can say about
estimates is to underpromise and overdeliver.

------
safgasCVS
I’ve worked at a company that took this too far and product managers used the
release cycle clock to game the system by launching features that were not
even half-baked but just outright didn’t work but since they hit their
official target of launching managed they to get promoted and the next person
in line is responsible for fixing the problem. Instead of using the release
cycle as a way to focus the product process it became a goal in itself. But
this author seems to work with really top class people so maybe that level of
stupidity in using the metric as the goal in itself is less of an issue in
those circles

------
timwaagh
I am glad it works for you but I am not comfortable or confident doing this,
unless i have a really firm grasp on the technology and that is the exception
rather than the norm, these days. Estimations tend to be driven by customer
expectations rather than technical limitations. If i estimate too high I look
bad. Even if that is based on previous experience. If i don't deliver in the
hoped-for-timeframe i might get fired. catch-22. Much better to not do it at
all, if the business allows it (if it doesn't then it's still important to
make keep hammering on estimation psychology ).

------
pinkfoot
> 100 plus people to port Skype in 16 months.

To all employers out there: if you put me on a project with a hundred others
to port a wee VOIP front-end to another platform, I too can get it done in a
year and a half.

~~~
temac
I don't know Skype but it can be easy to understate the scope of some
programs.

Like a web browser: why do we need that crap to be that large and complicated
again?

And that's a valid question (be it about voip software or web browser or
pretty much anything), but _even then_ it is still naive to pretend complexity
(whatever its origin and justification) does not exist and everything could
and should be completely rewritten in 3 weekends by 2 interns.

So: why don't you just go work at MS and see by yourself? Seems like they have
some kind of ideal conditions in your opinion.

~~~
pinkfoot
You missed the, admittedly subtle, point of my post.

------
romanows
The article seems contradictory. Is it really estimating if you drop features
you can't finish to ship by a deadline? Is a 16 month estimate worthwhile when
you screw around for 12 months before getting serious and then getting it
done? If the upshot is just to keep a conversation going with stakeholders and
always be working on the most important functionality, isn't this mostly
accomplished by 2 week sprints with a touch of externally set deadline
pressure (e.g., WWDC)?

------
barnaclejive
> Why the sudden productivity change? We had a big deadline coming up, and
> there was no option of not shipping. Suddenly, the whole team became
> focused, distractions were all gone, and we moved at a faster pace than I've
> ever felt the team do so.

Uh, what? Sounds like at the end of the day you ended up busting your ass
trying to meet some incorrect estimate. I was hoping to hear something about
how making better estimates made your life better and didn't happen.

------
username90
The advice to not give estimates is just to work around engineers bad soft
skills. So if you have engineers who can keep a healthy relationship with
stakeholders then estimates are not a problem, but if you have typical
engineers then someone else will have to do the estimates for the engineers
without consulting them. Ultimately someone has to do the estimates so if the
engineers can't handle it someone else will be forced to.

------
mbillie1
I thought I would hate this article, having years of delivering software with
and without estimates, and preferring the latter. But it makes a compelling
case!

~~~
generatorguy
Of course it is more fun to work with out any deadlines!

------
UK-Al05
The problem is is that most businesses refuse to negotiate on scope once
deadlines have been agreed.

Getting a manger who is willing to do that is great.

------
lasereyes136
The problem isn't estimates per se but how estimates are used:

1\. A commitment that can't change

2\. Manipulated to pressure people into work unreasonable hours

3\. If something will take "too long", estimates will conflate level of effort
with duration and be used, again, to manipulate people into doing unreasonable
things

4\. Used to cover up a failure to manage projects well

------
panchicore3
[https://youtu.be/sh7A8UChBTI](https://youtu.be/sh7A8UChBTI) funny, true and
fully related: stdout - stimates(software developers rap)

------
helltone
100+ teams for Skype on xbox?

~~~
B_Throwaway
Skype was 1 of the 100+ teams.

------
agentultra
Do it if you want but in all my years I've never seen a company sunk by a
missed deadline. At best it's a minor set back. At worst checks get delayed or
contracts terminated. Maybe that's enough to sink your tiny startup when
you're first starting out. For most companies that have made it past their
first year it's not the end of the world.

Empirical studies on the causes of errors, delays, etc in software projects of
different stripes and knowledge work in general suggest that _stress_ and
_sleep_ are the most important factors [0]. In video games deadlines are
paramount and yet, ironically, _crunch_ as the systemic form of over-work has
become known correlates strongly with review scores and sales: the more a team
is forced into meeting these arbitrary deadlines the lower the review scores
and overall sales of the game [1]. I don't think this is unique to the games
industry. I've seen enterprise development shops put in crunch time to meet
customer driven deadlines agreed to by the sales teams with similar results:
developers burn out and leave the company, tech debt runs rampant, and
customers get frustrated.

The article doesn't touch on the different kinds of projects one may be asked
to give estimates for.

A speculative project shouldn't be estimated. This is where the team doesn't
have the skills required or has never shipped any software with a similar
scope or feature set. There are so many unknowns between where you are and the
end goal that an estimate is only a dangerous guess. The only way you can make
an accurate estimate is to begin work and get close to the goal. Some where
between starting and finishing you will have enough information to make an
informed and reasonably confident estimate. Along with good release planning
this can be a relatively low-stress process. Estimates at the beginning of the
project create poor expectations and can lead to stress.

The enemy of a good plan is a perfect one.

Stress is so bad that it won't matter, much, that you do code review or have
automated CI and CD gates and checks. Your engineering team will simply not be
as effective as a team that isn't stressed or tired. Anything you can do to
keep your team from being stressed out or tired is going to translate into
better outcomes. If that means relaxing on deadlines and focus more on results
and progress than do that instead.

One thing I can agree with is that communication is key. And a good night's
sleep.

[0]
[https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2656292/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2656292/)

[1]
[https://gamasutra.com/blogs/PaulTozour/20150120/234443/The_G...](https://gamasutra.com/blogs/PaulTozour/20150120/234443/The_Game_Outcomes_Project_Part_4_Crunch_Makes_Games_Worse.php)

