I have worked in team with some kernel developers at Samsung, years ago, so let me put some perspective for those who do not understand the dynamics.
In some companies, the amount of patents or Linux kernel patches you get accepted is direct measure of your success.
As you know, whatever you measure becomes a target -- these guys feel very pressed to get ANY kernel commits accepted, no matter how small, irrelevant or nonsensical. Because in the very end almost nobody sees the amount of value that a company creates for Linux users. What everybody is looking at is that "Company X is nth on the list of biggest kernel contributors".
So "KPI grabbing" is meant to mean the practice of not caring for the value or quality of your contributions but rather for the count of the commits or LoC you can get accepted.
This is damaging to Linux kernel developer community because it just creates work for maintainers without creating much or any value at all.
These maintainers are absolutely critical resource for the project and their number and availability translates directly to how much stuff can actually be done. So no wonder people are irked by it.
A/B and telemetry-worship reminds me of something I posted on Fedi earlier. Pasting it below:
---
"Guys, I have an idea. Let's make a browser optimized for the kind of people who don't opt out of telemetry, and market it to privacy-conscious people." -- Top minds at Mozilla Corp
If you market to privacy-conscious people and you base all your decisions on data from telemetry, you're gonna get a very skewed perspective. You'll see disproportionate representation from fans (people who want to enable telemetry because they love Mozilla and want to help, and will rationalize any decision Mozilla makes) and very non-technical users who don't know what telemetry is and how opt-in/out works. Technical users who disable stuff like telemetry, analytics, etc. are going to look like a rounding error.
KPI = Key Performance Indicator = "thing we decided is important for us to measure and increase"
As such exactly what it is, will depend on company/industry/market/team/project/etc. It may be revenue or units sold or widgets made or turnover or customer satisfaction or days without accident etc. In principle, it communicates to teams what is important to business and helps everybody focus and sync.... with usual real-world caveats and implementation risks.
I imagine here, for a large-company linux team, a "KPI" was "how many commits you made" - they measure it, they track it, and reward/recognize/punish team members based on it. So people start working toward increasing the KPI in any feasible way - if you only want me to increase the KPI of commits done, OK, fine, I'll do some commits.
>thing we decided is important for us to measure and increase
This is a great explanation, and I just wanted to nitpick that for some KPI's (e.g. percentage of defective components, or latency), you would aim to "decrease" the KPI, and in some rare instances, you'd want to keep the KPI within a range (e.g. resource utilization, or inventory size, when you want a predefined amount of buffer).
KPI = Key Performance Indicator. It's basically a list of things that are considered during your evaluation. The more items off that check list you cross (if they're one-time items, like "ensure an uptime of X"), or the more of each you do (if they're things like "upstream patches" or "patents granted"), the better your score is.
It's a shit system that does exactly the opposite of what it's supposed to do IMHO, but that kind of stuff doesn't belong in an ELI5 explanation :-D.
KPIs are not a shit system, they are actually important to running any nontrivial organization.
The issue isn't whether KPIs are good or bad but what you do with them.
KPIs don't need to be tied to actual people, usually they are tied to teams, systems, projects, processes, etc.
For example, a number of users resigning from our services might be a KPI.
KPIs are an important tool to understand what is going on and whether we are going in the desired direction or not and to communicate this to your teams.
KPIs are a tool for telling your organization what you think is important, to set the incentives in the right direction.
KPIs will not help you if you have no effing idea how to set incentives -- they are just a tool in case you know what you are doing.
Because they are just one step from being a target for your teams, you need to be very careful to communicate what you mean exactly when you set KPIs.
Now, if you make number of users resigning from your services a KPI, you immediately need to set some additional guarantees that this is not overused -- for example by making it difficult for your clients to resign. This might be another KPI (user satisfaction with regards to how easy it was for them to resign).
Overuse law. If you overuse a law too much it stops being meaningful.
Organizations have hard time improving without measuring their performance and communicating incentives.
There isn't magical solution you can scribe on a paper and tell everybody -- this is exactly what you need to do to achieve success.
The best what you can do is compromise, it is unavoidable.
So we know setting targets can make wrong incentives. It is also not a reason to not be setting targets. It is a reason to make sure you damn set those incentives right and take close look that you are getting what you have intended.
The point they're trying to make is, I believe, that metrics shouldn't be targets. In a sense I think that this is obvious to anyone with a rudimentary understanding of applied statistics.
> The point they're trying to make is, I believe, that metrics shouldn't be targets.
Metrics are either (1) targets or (2) things that you are trying to analyze in relation to the metrics that are targets or (3) a waste of the time you spent gathering them.
The reason metrics are often bad when they become targets is the metrics arr usually not actual direct measures of goals, but things assumed to be convenient proxies, but when you push hard on optimizing them, they stop being good proxies because as well as being easier to measure than the real objective, once you pick any low-hanging productive fruit they are inevitably easier to improve in ways that don’t improve the real objective as much as the proxy (or at all, or which negatively impact it.)
What you want to measure and what you actually can measure are usually very distinct things, that's where "operationalization" comes in.
You have a target that you can't measure directly, hence you measure a proxy. That only works for as long as people don't game it and aim for the proxy instead of the actual target. Also, in addition, at least in research every statistician worth their salt knows that their proxy is not the actual thing and only their best effort at approximating it, hence in good studies, limitations of the methodology are extensively discussed.
Importantly, the moment you realise that your proxy is being gamed, you need to switch proxy.
I think this is only one example of this annoying trend of pretending to be "data-driven" by coopting numbers and fancy statistics without also adopting everything we know about uncertainty and how we can quantify it. "Data" in many businesses often implies a level of clarity ("look, the graph always goes up") that often does not actually exist and I suspect that this can be hard to understand for certain types of managers.
As someone who is periodically involved with this sort of thing, the challenge is that far and away the easiest things to objectively measure are almost always output metrics: commits, blog posts, external presentations at conferences, etc.
Output metrics chosen correctly also tend to represent things that the team/person has a reasonable degree of control over. For example, a developer KPI probably shouldn't be number of new customers because that's something they have vanishingly little control over, especially at an individual level.
The problem is that those things that a number of different teams are contributing to are probably the thing that the company cares about.
IMHO to not be gamed, management needs to operate on a conceptual level which is similarly sophisticated, or surpasses in sophistication, the conceptual level of the employees (or suppliers).
If there's a single KPI and all the reward is tied to that without any balance, sure that will be gamed.
But if there is a well-defined, appropriately complex reward function aligned with the utility of whatever the organization delivers to the outer world, I would consider that a positive.
This is trite. Just about any way of understanding something is going to create perverse incentives. Keeping the metrics secret has other problems too.
I don't think there is a a good solution. Maybe with radically shortened work hours and less pay disparity, the strives can strive off the job instead.
Um, what? The degree to which a metric is directly tied to incentives absolutely impacts how much that metric will be gamed. It is not trite at all to caution that metrics that are critical to understanding your business should remain as decoupled as possible from incentives. This allows you to maintain the quality of the signals your metrics provide.
This isn't just theory. This is precisely why there are laws that prohibit rewarding people based on how they voted.
So tell me this: how is the degree to which a given metric is tied to incentives meant to be varied?
> This is precisely why there are laws that prohibit rewarding people based on how they voted.
Voting is not an evaluation method of the voter: the voter is the one doing the evaluating. Because the powers should not discriminate between voters as you say, the vote cannot be used to evaluate the voters on and individual basis at all.
Maybe the answer is that: no individual/team assessments of any sort. That makes no incentives. But that also doesn't help with fine-grained strategy.
> So tell me this: how is the degree to which a given metric is tied to incentives meant to be varied?
...by not tieing incentives to it? I feel like I must not understand your question because this seems blindingly obvious...
> Voting is not an evaluation method of the voter: the voter is the one doing the evaluating. Because the powers should not discriminate between voters as you say, the vote cannot be used to evaluate the voters on and individual basis at all.
I don't even get where you got this strawman from.
> no individual/team assessments of any sort. That makes no incentives. But that also doesn't help with fine-grained strategy.
No, just dont use KPIs that are critical to understanding your business in assessments tied to incentices because that will distort your KPIs and make you less capable of understanding your business.
Actually I'm being silly, there is a good solution: Workplace Democracy. You can fool the higher ups come promotion time, but you can't fool all the people all the time.
When there are just a few decision makers, a standard evaluation procedure is usually necessary for fairness / reproducibility.
When there as many evaluators as evaluatees, no standard evaluation procedure is needed on the theory that all the different ways and biases will cancel out. That doesn't work over time, as clearly America's electorate gets interested in different things, but I am less worried about that for co-ops.
The hope for co-ops vs state democracy is two fold.
Firstly, because people do work many hours at their job, but don't necessarily spend any time running the state, I hope they are more informed and engaged.
Secondly, the "both options are bad, I hate this" problem should be addressed by A, proportional representation (just like should be done with state democracy states), but also B of switching jobs. The barrier of switching jobs is much lower than immigrating, which ought to more than any thing else reign in defeatist apathy. And Co-ops can still fire people, remember.
> KPIs don't need to be tied to actual people, usually they are tied to teams, systems, projects, processes, etc.
Right, but my post, the post I was responding to, the post the post I was responding to was responding to, and the article in the link, was specifically about a situation where KPIs are tied to actual people. I'm not sure what you're arguing for or against.
I have seen many examples of KPI’s not working as intended. Do you have any real world examples of KPI’s that actually worked in practice (and not just in the make-believe heads of managers)?
KPIs are actually very good when used properly, but they are (like many things) easily misused. For example if you provide a service then your obvious choice for the team's primary KPI would be service uptime.
Use SMART objectives (Specific, Measurable, Achievable, Relevant, Time-bounded) and set a realistic service uptime (e.g. how many nines you want in a given month/quarter/whatever) and KPIs directly enable good decisionmaking at all levels of the org.
Manager up in your grill about a choice you made? Point at the KPIs and say "this will help make it easier to meet our objectives for KPI 1.1" or whatever.
I didn't know open source programmers were subjected to those kinds of performance reviews. It sounds to me like the pandas have come and visited the penguins and there's a whole lot of shock and horror as the penguins get eaten because you'd think the pandas only want bamboo.
Engineer's performance is measured, in part, by number of kernel commits. So, the number of kernel commits is a key performance indicator for the developer.
Oh god no. It means their managers evaluate them on that metric. In this case, the company has set targets on how many Linux kernel patches employees should land.
Thanks for your perspective I didn't actually realize that dynamic was at play (at an organizational level, anyway, I'm well aware individuals game # of contributions).
These organizations pushing trivial/garbage patches and wasting maintainers' time absolutely have to be called out by the community as unacceptable, so I'm glad to see the post. This is a massive waste of time, time which open source contributors/maintainers often are already short on.
Maybe a panel of maintainers could publish a quarterly review with attribution/thanks to the most important contributors. No metrics to game, only pure human expert opinions.
IMO part of the problem is that Greg KH publishes this annual 'state of the kernel' that is just a list of metrics to game. Which feels like it was started to shame non-RH companies to contribute more often. So, kinda maybe beware what you wish for?
Maintainers aren’t interested in taking on extra chores. However, important contributors already do receive attribution, in the format of news articles, here:
This is why ousting Linus puts the long-term health of the kernel at risk. It’s very difficult and expensive, at a personal and professional level, to tell colleagues to “fuck off” if they are submitting low quality garbage for reasons related to their salary. Linux was Linus’s baby, he had nearly absolute control, and he didn’t care too much about politeness - this was a magic recipe for him to be able to stave off garbage patch requests from motivated corporate peons.
Now, if the people approving patches are just normal employees who can get fired if a kernel sponsor complains enough, it’s going to be much harder for them to successfully deflect all the bullshit without ever caving in.
Good theory, but it's factually wrong. "Maintainers" in this context doesn't refer to Linus personally, it refers to individual maintainers of parts of the kernel. You can check the kernel MAINTAINERS file, and note that such a file has been there for years.
There has been a longstanding problem of people submitting low-quality patches to maintainers - not to Linus personally - and that problem predated Linus stepping back for one release.
Linus was not ousted and continues to have nearly absolute control; he has, for decades, chosen to exercise that control by delegating ownership of parts of the code to other people. Those people receive and review patches and incorporate them into their repos, which Linus pulls in bulk. (This is the origin of the term "pull request," which predates GitHub's use of it.) He quickly reviews those patches to make sure there's nothing weird, but he generally trusts that his "lieutenants" are making good decisions.
That means that the time being wasted here is the time of individual maintainers, who are reviewing the patch, making a full judgment on it, and pulling it into their tree. Linus sees all such cleanups once per cycle and looks at them fairly quickly.
Almost all of those maintainers have been "normal employees" of various companies for many, many years. Again, take a look at the MAINTAINERS file.
It is naive to claim you can tell which part of Linus is and which isn't contributing to the success.
For one, I think it might be possible Linus's no-nonsense attitude is for the better of the organization as people who can't work with it are leaving causing Linux developers, on average, to be more no-nonsense and also able to coexist better with other no-nonsense people.
But, it is my conjecture only. We would never know how Linux would fare if Linus was another guy.
It is probably easier and more defensible to look at a thing that is, and say "why is it this way" than to see something that isn't and ask, "why isn't it the way I want it to be?"
These are logically not the same structure of argument. I'm not making any character judgements about people involved in this discussion, or on the LKML either for that matter.
I have trouble understanding why it bothers people so much that Linus is rude sometimes. You don't have to interact with him if you don't want. You can even contribute to the kernel without interacting with him. Whatever he's been doing has been working for going on 3 decades now, I don't see a burning need to change it because it bothers some outsiders.
There's this weird sense of entitlement. People want to elbow their way into this long-running and successful project and start telling everybody how they ought to behave. They think they have the right to contribute on their own terms without bothering to understand the project culture, and think it's everybody else's responsibility to make things easy for them and behave in a way they approve of.
If you really think the LKML is too toxic and hostile, to the detriment of the project, then feel free to fork it and start up a parallel project without the problems you see. Surely your friendly corporate-style culture will attract more contributors and soon you'll have the more successful kernel, right?
Personally, I like having somebody in charge of the core of the OS who's so concerned with correctness and hygiene that he gets upset when they're violated.
The main problem for me is the old "fish rots from the head down" effect -- when the rude and entitled behavior comes directly from the top, it's no surprise when everyone else starts acting like that and gets at each other's throats. It should be obvious by now where this weird sense of entitlement and refusal to understand other people's culture is coming from.
>feel free to fork it and start up a parallel project without the problems you see
Most Linux distros are basically already doing this. They all have their own patchsets. It's well beyond correctness and hygiene at this point, if you actually look at the changes that are being disputed, it already falls a lot more in the "cultural differences" category.
So the LKML has been rotting from the head down for...30 years? If the result of that is Linux, maybe the rot isn't as bad as you fear.
The cultural differences that cause distro forks are debates over free-vs-libre, what belongs in the kernel vs userspace, and standardization issues, not (afaik) "we won't submit this to the upstream kernel because we don't approve of the way they talk to each other there". And I think it's very important for the core kernel devs to hold the line on those issues.
I think the kernel developers got on pretty well for all those years and managed to put a pretty good product together, despite the big rotten head. And Linux distro maintain patchsets, but they're against mainline, that's a very salient difference here. They still start from whatever upstream publishes, they're not so upset by swearwords that they took a snapshot at Linux 2.4 and only add their own code on top of that.
The point there is, while some negative attitudes may not be preventing other projects from using the code, they do prevent the project from growing, and it leads to fragmentation, which I believe they are seeking to minimize. So something has to give.
Well...I like Linus just fine as kernel maintainer, but I remember a dozen times (over 2 decades, to be fair) when he was, uhh, very direct. It's often not as aggressive as people make out, but he can definitely get a bit mean when people keep making the same mistakes.
And the list goes on and on. That said, his decisions seem to have worked out amazingly well. At some point I can imagine getting tired at having to repeatedly defend your views.
>his decisions seem to have worked out amazingly well.
As someone who regularly deals with bugs and inconsistencies in Linux, I wouldn't say so. For me personally, I just want to be able to fix the issues, and would rather not spend the time bickering with someone who has some kind of personal vendetta about something that I don't know about. I try hard not to dump my baggage on other open source maintainers, I hope others can do the same.
While I don’t necessarily agree with the GP, there’s a huge difference between humble incompetence and arrogant incompetence. Everyone starts out incompetent, but that’s fine as long as they don’t repeatedly shove it on other people. Incompetence, without the self-awareness or humility to avoid wasting other people’s time? Maybe getting a verbal slap on the wrist is exactly what’s needed…
> he didn’t care too much about politeness - this was a magic recipe for him to be able to stave off garbage patch requests from motivated corporate peons.
Linus disagrees with you, so strongly that he invested a great amount of his time, reputation, and political capital in changing his behavior and that of his organization.
Their younger versions might not have agreed with it, but some people do mellow out in their old age. In this case, to me it's clear that he is doing a large turnaround, which validates Wyager's statement that he did not care too much before.
I'm not sure what that means, or what basis there is for any of it ? The basis for my comment was Torvalds' own statements and actions, and he's a pretty good source for Torvalds and for effective management and communication in the Linux community. Do you know him personally? Are you a maintainer?
It means that Linus used to be fairly brusk when he was younger (basis: e.g. flipping people off and telling people to "fuck off"), and recently he has apologized for his behavior (basis: the link you shared). To me this is him mellowing out in his old age. His younger self might not have agreed with this mellowing out, seeing it as unnecessary (basis: his responses previous to the apology to other people who called him out on his behavior). The fact that he apologized means that even he realized that he did not care too much before, but now he does, and therefore he wanted to make up for what he has done.
If anyone gets ousted from the process, they probably deserve it, for wasting more maintainer time. Nobody is immune from that, not you, not me, not the maintainers, not the BDFL. Expecting the BDFL to be able to have full control and to review every patch is also nonsense that causes the exact same type of issues: https://lwn.net/Articles/393694/
The issue is that creating objective measure of contribution value is just unfeasible if at all possible. Not to mention amount of work required which is exactly the issue.
But mostly, maintainers want to just focus on their work and not be bothered by corporations and their developers trying to game the system just to prop up their position.
> The issue is that creating objective measure of contribution value is just unfeasible if at all possible.
Could you give an example of a non trivial patch which value would be unfeasible to assess objectively? I am not familiar enough with the maintenance process to understand how hard it is to measure objectively the quality of a patch.
It is not about any single patch, you can't assess value of any patch objectively.
For starters there isn't even an objectively established definition of value. You can't objectively measure something that only has subjective definitions.
In a free market you could say that the price could be proxy for the value (ie how much somebody can pay for it should at least theoretically correspond to its value for that person). But how you do that for an open source project?
It might be feasible to assess a single contribution, but not feasible to assess all of them. To make this KPI work, maintainers would have to publish an objective quality rating for each accepted commit. That's an enormous amount of work, and impossible to get right anyway since the value of a change might not be immediately obvious, or might be negated by a future change, etc.
The advice to companies using this KPI would be: let your own dev managers evaluate the quality of the commits of their engineers, and rate them on quality as well as quantity, then submit. Engineers will quickly stop creating busywork when their managers get wise to it.
Yes, so either accept the status quo or create positive change.
Given that it’s annoying enough that we have articles and comments on it, perhaps investing maintainers time into developing metrics that work for the maintainers isn’t a terrible idea.
Doesn't work. "When a measure becomes a target, it ceases to be a good measure."
You can totally build something that is a decent metric, but as soon as you create incentives to "game it" (optimize for the metric not the actual goal you're trying to measure), it will be gamed, and creating metrics that are resistant to that is nearly impossible in most cases.
Edit: Nvm, I misunderstood the initial problem and my solution doesn't address it.
Why not just keep commits as a metric, then estimate the quality of those commits by sampling. If the organization appears to be gaming the stats then flag it as such.
I tend to look on commit count as a negative when reviewing open source projects. TeX is probably the highest quality open source project and it's has had maybe 15 commits since I was born. Bash has had about 15 commits in the last five years. Then there's everyone else who's playing to win the Github game with 500 stars and 5,000 commits. That level of activity might impress normal people but it doesn't impress me.
> I tend to look on commit count as a negative when reviewing open source projects.
I do agree that tracking commit counts isn't very useful but tracking commit frequency, in my opinion, is quite useful. I'm obviously biased but I use commit frequency to tell me how fast a project is moving and what is the investment.
by breaking down how frequently contributors commit on a daily basis, you can get a very good sense of investment and speed. In the case of the vscode project, Microsoft is investing a lot of resources into vscode and it is evolving at an extremely fast pace.
Software metrics in general isn't the issue, it's the lack of context around them that is the issue.
Disclaimer: I'm the creator of the tool that I'm linking to
Because that would be a huge time sink for the maintainers as well. They’re the only ones qualified to evaluate quality and they have better things to do with the time it would take. This problem is created by these companies and it’s their problem to solve.
I like this idea, but the gamification / rules / meta around it sounds like it would be even more work. And like all metrics it would quickly become a target and... Goodhart's law.
This is one of those ideas that I think should totally be a thing, and yet I think wouldn't work and thus shouldn't be a thing.
Then you get them into trying to get their own people into maintainer positions to let the golden stars flow. Maybe also be more hesitant to give out this starts to the "wrong" people from other companies. Not like there already is enough politic bullshit in open source.
In some companies, the amount of patents or Linux kernel patches you get accepted is direct measure of your success.
As you know, whatever you measure becomes a target -- these guys feel very pressed to get ANY kernel commits accepted, no matter how small, irrelevant or nonsensical. Because in the very end almost nobody sees the amount of value that a company creates for Linux users. What everybody is looking at is that "Company X is nth on the list of biggest kernel contributors".
So "KPI grabbing" is meant to mean the practice of not caring for the value or quality of your contributions but rather for the count of the commits or LoC you can get accepted.
This is damaging to Linux kernel developer community because it just creates work for maintainers without creating much or any value at all.
These maintainers are absolutely critical resource for the project and their number and availability translates directly to how much stuff can actually be done. So no wonder people are irked by it.