
Don't measure developers, measure projects - jeanlaf
https://anaxi.com/blog/2018/11/25/do-not-measure-developers-measure-projects/
======
edejong
Don't automate what you can't first do manually. Your best developers might be
helping, supporting, designing and architecting your next solution.

One commit, one PR, might prove more valuable for your business than hundreds
of other tickets closed.

Remove the KPIs and talk to your team members in 1-on-1s regularly. Give
responsibility and trust to your team, and they'll be self-empowering.

Add KPIs and you'll take away their responsibility. Trust will be lost.
Probability for innovative solutions will go down, your best developers will
leave.

It is all about trust and responsibility. Hire for team spirit,
conscientiousness, communicative skills and diversity.

~~~
closeparen
The phase of a project with a flood of PRs and closed tickets is the easy
part. The real work is done before and after (planning and debugging), in
people’s heads.

------
quanticle
Of the metrics listed, the only one I will say _never_ to use is lines of
code. As a very wise manager told me, "Code is a liability. Functionality and
business value are assets." Measuring developer productivity by looking at how
many lines of code they've written is like measuring the health of a business
by how much money it's spending.

If you can delete code and retain the same level of functionality (for
example, because you found a library that will automate much of the task),
then that's a huge win. Every piece of code that the team writes is one more
thing that has to be unit tested, integration tested, documented, and
maintained in the face of future changes. If they can offload that work to a
library, or find a way to accomplish the same task by using existing code or
by reconfiguring the application, that's almost always preferable to writing
new code to accomplish a particular business need.

Of course, one can go too far in the other direction. It's easy to fetishize
using external libraries for everything (like we see so often in the JS
ecosystem). But that's where I would argue that the analogy of treating code
like money shines. Sometimes it _is_ worth spending money to have something
custom made for your business, rather than buying an off-the-shelf component.
Similarly, sometimes it is worth writing code to do something custom, rather
than integrating a third party library, or reconfiguring an existing
application. But that choice should be consciously made. Rewarding developers
for lines of code written moves that choice out of the realm of conscious
decision-making and into the realm of reflex. They get paid for more lines of
code, so they're always going to reach for the solution that results in more
lines of code, regardless of whether that code is necessary or appropriate.

~~~
eksemplar
I agree that lines of code is a terrible measurement of productivity.

I don’t agree that short is always better. Sometimes it’s better to write the
code so that it’s easier to understand than to use smart tricks to make it
short, because the next person looking at the code might not be on to your
tricks. You can argue both ways on this of course, but I prefer clean and
easily understandable because I’ve seen what happens when you don’t do that
and a junior programmer has to fix something 15 years later.

I also disagree that libraries are better. You seem wary of JS, and that’s
good, but no libraries are magical and none of them can be trusted to do
something for you, unless you look under the hood.

Especially not open source libraries. If you don’t know what is going on
inside the libraries you use, then you’re heading for a world of pain. Of
course some libraries are better than others, but using them blindly is wrong.
And if you have to read up on them, then they aren’t really quick fixes.

We had a developer who used a library to handle decimals. We have our own
library for this, that we’ve build because almost none of the open source
libraries for decimals are very good at handling memory allocation and floats
and computers aren’t 0.00000000000 they are 0.00000001436 etc. But our
developer didn’t know we had our own library for this, so he used a public
one. One that uses around 10 other libraries, one of which didn’t handle
memory allocation the way you’d want.

So suddenly our decimals were wrong, but the computer didn’t know they were
wrong, because technically the library worked it just wasn’t specific enough.
And it cost us a lot of money to clean things up afterwards.

This is the extreme example, but it’s never been more relevant than it is
today. Especially not in a world where less and less programmers seem to
actually know how computers or even compilers work.

~~~
jgtrosh
I assume that's “especially not-open-source libraries” and not “especially not
open-source libraries”

~~~
eksemplar
No, I meant to say that open source libraries are especially dangerous.

I get why you may object, but open source libraries have failed us a lot and
by contract the .net framework library hasn't.

There are a lot of great open source libraries, that are both well documented
and maintained, but there are a lot more that aren't.

~~~
jgtrosh
Then i don't understand why you associate not knowing what's in a library with
specifically open source ones. They tend by nature to be more open to
introspection by any member of the community.

~~~
eksemplar
[https://arstechnica.com/information-
technology/2018/11/hacke...](https://arstechnica.com/information-
technology/2018/11/hacker-backdoors-widely-used-open-source-software-to-steal-
bitcoin/)

------
austincheney
One thing I have noticed when working with juniors is that the wrong things
are considered for story point estimation and those story points are the
metric of success.

Juniors tend to be overwhelmingly concerned with basic literacy. First its a
"does it work" and then it graduates to "how to write the code". This often
comprises thinking about tools, frameworks, dependencies, and code style.
These aren't deep problems that moves a business forward. These are competency
problems that keeps a person employed. This is a distraction that consumes
developers with busy work instead of thinking and planning about how to make a
product better or how to release faster.

Worse still, too much emphasis on the particulars around _the how_ of it can
blow out a 30 minute task to a 2-5 day task with all the unnecessary
scaffolding and extra process. This is particularly frustrating when realizing
these concerns can be immediately eliminated when the proper automation is in
place. It is possible to really get good at spinning on mouse wheels and
cranking out those stories a little bit faster, but it still isn't moving the
business forward. Nothing forces out strong developers faster than cliches
like: "That is the framework way" or "You're reinventing a wheel".

Some better metrics to really measure real world success:

    
    
        1. does the change solve a business problem
        2. does it pass the project automation
        3. does it reduce total code instructions
        4. does it close out defects
        5. reduced execution time
        6. improvements to project automation (faster developers)
        7. improvements to documentation
     

Once the proper metrics are in place the concerns of the juniors are addressed
economically. For example, if reducing instruction count is important the code
gets leaner and _the how_ of it shifts from code style to leaner code. Code
style is important because developers need improved literacy from uniform code
expression, but once the goal of expression is leaner code the uniformity
shifts from expressive style to optimization concerns.

~~~
ascar
I'm not sure what you mean with "reducing total code instructions" and "lean
code", but I'm not sold on the idea of making code less readable which often
happens when you try to reduce code by being smart.

Reading code is mostly way harder than writing it, especially if you read code
from someone else. Optimizing for readability is the game changer for
maintainability.

* Use expressive function and variable names.

* Have clear consistent structure across your project(s).

* Use simple code where possible.

* Do not write one-liners that do multiple transformations. Each line should have one purpose.

* Document design decisions in code. Don't document "what" your code does, that should be readable by itself, but why you made the decision to do it like that.

* Documentation library style comments for complicated functions helps a ton to know what the function is supposed to do.

* Add tests that specify the expected behavior. This not only helps when changing code, but also when reading.

It's not readable code XOR documentation, it's both. Readable code is the
"what", documentation is the "why". Even if the docs get outdated, the why can
still be insightful.

Non of the above suggestions usually reduces code, but it will help everyone
else, including future-you in 6-months, to work with your code.

~~~
fouc
Why would you think less code means the code is less readable?

Does cramming 5 lines of code into 1 line make it less code? No.

Reading code is harder, so the less code there is to read, the better.

~~~
ascar
As I said in my comment, a lot of the stuff I see as positive for making code
more readable more often than not adds extra lines as comments or as code,
because you keep it simple. You are right that cramming 5 lines in one,
doesn't really reduce code, but I have worked with enough programmers that
thought it's a good thing.

Sure if you can reduce code without making it more complex, that's a good
thing.

------
djcapelis
Noooooooo. All of these metrics are a mistake, don't measure any of them.
Measure the things that actually matter about your product and whether or not
it's getting better or worse.

Measure outcomes. Only outcomes. Try and make sure your metrics capture the
real outcomes and not just what you see in a lab. This usually means you need
to use field data to get that information.

~~~
vlovich123
The post looks reasonable to me. The point he's making is that no metric is
useful for measuring developer efficacy/productivity & the underlying unstated
reason is that dev works tends to be random; problem A is nothing like problem
B & there's no way to objectively measure the quality of solution for problem
A against another developer's solution.

He's instead proposing metrics that you can use as a high level barometer of
project health. What outcome are you looking for? You can't mean the final
shipping SW because that's usually way too late. You want to figure out if
your velocity is going to let you hit your ship date _well_ in advance of the
actual ship date so that you have time to deploy process fixes/hire more
people so that you do hit the ship date.

~~~
djcapelis
> He's instead proposing metrics that you can use as a high level barometer of
> project health. What outcome are you looking for?

None of the proposed metrics capture project health. Test coverage is the
closest and that still isn’t great. (Unit tests are important but you can have
100% test coverage and a 0% working product. In fact, this is how every
project begins before the first line of code is written.)

No one gives a damn how many pull requests happen or how many days apart they
are.

1) Does the software work yet?

1a) How much of it works?

2) Does it do what we want?

2a) .... everytime? Less? How many 9s?

Report on the things that matter. Track the things that matter. Everything
else is actively harmful and distracts your teams from doing what matters.

Also, when you ship the software you haven’t finished, you’ve begun. It is
absolutely still useful to keep metrics on outcomes and track improving them
over time. It is absolutely not “too late” and any planning that assumes
software is done after you ship it is flawed. Features can become done,
software never is.

~~~
swish_bob
They're useful to measure in that _changes_ to them are indications that
something might be happening. Use them as canaries.

For example, your velocity has changed. Why has it changed? Are you bringing
new people up to speed? Have you suddenly hit some unexpected complexity? Your
mean time to resolve issues has gone down. Are you releasing code with simple
bugs you should have caught? Have your team started picking them up sooner?

Whenever something changes there's questions you can ask, the answers to which
might well help you do better. Using metrics as an early indicator of change
can help.

~~~
djcapelis
1) If you need metrics to answer _those_ questions you have a larger problem
than metrics can solve.

2) That's an argument for literally any possible metric. You could make an
equally passionate argument that looking at and keeping track of free/busy
times on calendars or soda consumption or how full the garbage cans are on the
floor as an indication and early warning sign that something changed. It
doesn't mean those are good things to track. Pay attention to the things that
matter. Align whole teams and organizations around those things. Don't let
them get distracted with metrics that don't matter.

3) Metrics are a double edged sword and influence behavior. If you're tracking
the wrong things, you will _get_ the wrong things.

~~~
ozim
People would like to have simple proxies for measurements instead of doing
actual work. I totally agree all those metrics are useless and only real
indicators are "is it working yet" and "are users finding it useful". Because
I can make loads of pull requests that are perfect but are not working for end
users, and have 80% test coverage for functionalities users never use.

Funny is that as project leader one should click through project as in using
it to see if it is working instead of wasting time on finding proxies. If
leader/manager does not know how to use system or never clicks through then he
is useless...

------
stcredzero
A friend of mine was looking at the metrics for a project he was managing and
saw that the bug rate of commits after 4pm was much higher. So then he
introduced a new rule against committing after 4pm, resulting in a big drop in
the bug rate. He chalked it up to less committing while fatigued and rushing
to get things done to go home.

~~~
not_kurt_godel
That's really interesting. Was there any drop in overall productivity/delivery
pace as a result? There are definitely times when I wind up cranking out
buggier-than-usual code later in the day/night to meet deadlines or get things
done that otherwise wouldn't (at least, not on schedule). Sometimes when I
timebox myself too much like that and it winds up dragging projects out days
or weeks longer than they would have been had I not just blocked out a few
hours to get the dang thing working and then cleaned it up afterwards; this is
particularly true for complex projects where it can take a large portion of
the day simply to build the mental context necessary to actually start
executing on what needs to be done. If I cut myself off at a certain arbitrary
time, the time spent building mental context earlier is totally wasted.

~~~
stcredzero
_If I cut myself off at a certain arbitrary time, the time spent building
mental context earlier is totally wasted._

If what you have to remember to make a certain change means you have to keep
27 things in your head at once to avoid making bugs, you might want to
consider re-organizing your code so this number is smaller.

------
itsdrewmiller
I think the title of this post is great, but the body doesn't quite deliver.
The best content I've found on this subject is this book:

[https://www.amazon.com/gp/product/0932633366](https://www.amazon.com/gp/product/0932633366)

It talks about the different between informative and motivational metrics -
the former being just for awareness while the latter being intended as
explicit targets for employees. While it's easy to bleed the former into the
latter, there's nothing inherently wrong with measuring game-able statistics
as long as the incentive to muck with them is minimized. Easier said than
done, but not impossible.

The article is pretty dismissive of story counting, which is a semi-popular
approach to estimation:

[https://martinfowler.com/bliki/StoryCounting.html](https://martinfowler.com/bliki/StoryCounting.html)

(Anecdotally, we had someone look at our teams' previous sprints and found
that # stories / sprint was more consistent than points per sprint over the
time frame they reviewed. YMMV.)

The focus on velocity-related metrics seems smart - Git Prime did some
analysis on this and found that frequency of commits was a useful measure:

[https://blog.gitprime.com/prolific-engineers-take-small-
bite...](https://blog.gitprime.com/prolific-engineers-take-small-bites-
patterns-in-developer-impact/)

I don't see how these are any less game-able than klocs though.

~~~
jeanlaf
Thanks for the book recommendation! Will definitely read it :).

Concerning Gitprime, I completely agree with you. And they do say that the
point is to game their metrics: [https://help.gitprime.com/general/cant-you-
just-game-the-met...](https://help.gitprime.com/general/cant-you-just-game-
the-metrics)

------
ausjke
Is Agile solving this problem now? or has Agile gotten out of fashion?

    
    
        * release often, release early, each release should function though with less features.
        * break large goal to small steps, one step at a time.
        * sprint every 2 weeks or 4 weeks.
        * 15-min stand-up every other day
        * etc,etc
    

I read some agile books in the past, and now I'm experiencing real Agile in sw
development for the first time, so far I found it is quite effective comparing
to my past projects, assuming you have some great scrum master and project-
owner that are really good at the balance of arts.

~~~
wpietri
If people actually do it, it helps a lot.

In my experience, if business stakeholders are seeing real value shipped to
customers every week (or 2 if you must) then they don't get very antsy.
Especially so if they are using a release-early, release-often approach to the
product, where they see a project as an opportunity to explore what customers
really want.

The problem starts to come in when the feedback loops are long. If they ask
for something in January and don't see results in June, then they've got 6
months to wig out. And one of the ways they'll try to manage their fear is by
getting up in developers' business by imposing all sorts of metrics.

~~~
ausjke
in our case it's a long term internal project, it's embedded system
development, we release bi-weekly internally before it reaches 1.0 for
customers.

embedded product is notoriously hard to manage and I found Agile process
actually helped quite a bit, via Scrum's 'story points' method.

------
abernard1
> If you don’t include story points or some equivalent, this might be the most
> misleading metric you could use.

I'm sorry, but if you think "story points" is something useful, your
organization is already doomed.

Story points is an inflationary currency. When your developers realize that
somebody important in the company cares about said currency, they will make
sure that their work has more of these "story points" in "sprint planning".

~~~
Jtsummers
Maybe it's just because I'm more confident that I won't get fired, but I've
not had any trouble telling the higher ups to stop asking about story points.

Story points are a way for your teams to improve themselves. If they don't
understand that, then teach them. And never report the points (directly) to
anyone above the project lead or perhaps one level above that. Any higher and
they are too disconnected from both the team and the work to understand what
they're seeing.

The only thing that higher ups should care about are trends up and down. Your
team, after say 6 months, should be pretty steady in their performance if
they're using story points correctly. Downward trends should be identified by
them and their project lead _early_ and addressed, and if it's an issue that
may impact other teams they should communicate that up and out. Upward trends
should be celebrated but also examined to see if some change could be
duplicated elsewhere (an improvement to your build configuration resulting in
faster feedback, adoption of a new development tool or technique, etc.).

~~~
abernard1
> Story points are a way for your teams to improve themselves.

These points have nothing to do with actual value produced to the business, or
the developer's worth / skill. It's also very easy for a developer to begin to
inflate the estimation of tickets they believe they will work on, creating a
false sense of confidence.

Alternatively, if people work at a "shop" where tickets are so fungible that
people pick up whatever the top ticket on the board is, it tells me that they
are working on low-value, repeatable things where expertise is unnecessary.

It's also not clear that increasing story points (even if they were somehow
related to work / hours perfectly) would be useful. I've seen teams drop down
to do minimal "work" for months and produce tremendous value because they were
reorienting the product to do things successfully. An organization that looked
at that labor input, sans objective outcomes in the product, would be doing
themselves a disservice.

Story points, tickets closed, lines of code: ultimately all of these are a
belief in the labor theory of value. But that's the thing. I don't want labor,
I want value, and the tragedy that is input-focused measurement is just due to
it's easier to do than measure product outcomes. It's always more convenient
to say "We did 24 story points this week" than "Our users hate our software,
and here's how we know it."

~~~
Jtsummers
> These points have nothing to do with actual value produced to the business,
> or the developer's worth / skill.

This is true.

Story points don't translate to value. They can't. A nail could be worth 5
cents, or it could be a million dollars. Value is determined by the customer
and their need.

Story points translate to cost. Customers determine the value. They examine
the cost. And they determine whether the value is worth the cost. Is that 5
cent nail worth 5 cents or is it the nail that'll save the project?

Story points are about developing an effective and consistent _estimation_
tool (how much will this cost or how much time will this take to produce). If
your team is inflating their story points, they're idiots.

Assign a number to a story or task. If the team has developed consistent and
reliable estimates (this takes time to happen) then this number translates
with some reasonable fidelity to cost and time to complete the task. If the
numbers are unreliable indicators of cost and time, then the team needs to
examine their estimation methods. Perhaps bring in other people with more
experience to help them.

The _team_ (themselves, no one else) has a history of completing so many story
points per cycle (sprint, week, month, whatever). So they know that if the
customer has asked for an estimated (their estimate) 50 units of work and they
can only complete 20 per month on average, then it'll take 2.5 months on
average to complete the work. This lets them, you know, plan. And discuss with
their customer the priority and objectives. If the customer is only paying for
one month of labor, the customer will have to reexamine their objectives and
priorities.

The customer never sees the story points. They see the cost and time estimates
the story points turn into.

The managers never see the story points. They see the cost of the projects and
the income from customers.

The only people who should see and care about the story points are the
developers and the project lead. Maybe a statistician if they can't do the
math themselves.

~~~
abernard1
> Story points translate to cost. Customers determine the value. They examine
> the cost. And they determine whether the value is worth the cost. Is that 5
> cent nail worth 5 cents or is it the nail that'll save the project?

There is absolutely no evidence that story points actually correspond to cost
in a product organization (as opposed to a contracting billable hours
situation), because the long-term scope of a project is unknown. When does a
project actually "end" with respect to its cost vs value production? Is this
accounted for via story points? Are all of those things known when starting
the first iterations? I would submit the answers to these are "we have no
clue."

Something that does however correspond to cost is well, cost. Team's have a
known salary burn rate, and this is completely independent of how many
imaginary points they make. Any organization whose value output is being
measured where two weeks is actually such a large timeframe that it needs to
be broken down into more granular estimates is 1) delusional about the
predictability of their actual value output and 2) pipe fitters working on
easy, well-known features.

There are many ways of estimating projects as a whole that are equally as
(un-)reliable for estimation and take way less time, and these can be coupled
together with financial rationalization from the business. If two week
increments can't have significant variance for the business to make a profit,
and this business is in the extraordinarily high margin software industry,
they're working on the wrong problems.

------
nixpulvis
Getting secure connection warnings.

~~~
daveslash
Maybe the author fixed it - I see no such warning (FireFox 63), but what did
catch my eye was a 3.6 MB jpeg that took 16.3 seconds to load (25 Mbps
connection) -- admittedly, haven't read the actual article yet.

~~~
jeanlaf
Oh let me check this! Sorry about that.

