
Vanity Metrics - kawera
http://firstround.com/review/im-sorry-but-those-are-vanity-metrics/
======
cpitman
Another good technique is to pair metrics. Find a second metric for each
metric you care about that you expect to see negatively impacted if you
overfit for the first metric. For example, you might pair user growth rates
with per user acquisition cost.

------
sanswork
The bitcoin world has a real problem with vanity metrics to call out a few
specific ones:

It's full of companies like Coinbase and blockchain bragging about number of
accounts/wallets but I've yet to see any of them report MAU.

Companies like Bitpay talking about massive use growth using % figures and no
absolutes and constantly changing some part of what they report on so it
cannot be compared.

SCI/Rebit regularly talks about how they can't give actual figures to their
volume because they don't want to attract too much attention yet spend half
their time posting to forums with % increases and bragging about how fast they
are growing and how huge they are.

Cointip spent years bragging about growth and how much volume they were doing
before suddenly shutting up shop.

The only companies in the space that show any real information are the
exchanges and Openbazaar and in the case of the later I'm sure they wouldn't
if they could hide it.

~~~
davidgerard
The important point, though, is that each of these is actually _good news_ for
Bitcoin.

------
mtkd
I call them input metrics and output metrics - you can't usually directly
improve output metrics like 'daily active users' \- but they are necessary
validation of operational performance (just don't look at them every day)

input metrics are things that you can impact directly (like site speed -
which, if improved, will likely indirectly improve the outputs)

from Jeff Bezos 2009 shareholder statement:

 _360 of the 452 goals will have a direct impact on customer experience._

 _The word revenue is used eight times and free cash flow is used only four
times._

 _In the 452 goals, the terms net income, gross profit or margin, and
operating profit are not used once._

[https://www.sec.gov/Archives/edgar/data/1018724/000119312510...](https://www.sec.gov/Archives/edgar/data/1018724/000119312510082914/dex991.htm)

~~~
slededit
I could easily increase daily active users of even the worst app by paying
people in India to use it. This wouldn't exactly help the business - but if
that's all you are optimizing for then its not hard to get. In the same vein I
could increase people's time in the app by simply making it harder to use.
They'd spend more time trying to get what they wanted.

There are no benchmarks that cannot be gamed.

~~~
pedalpete
I'm not sure exactly how this relates to the above comment.

But you are making the point that the OP made. You need to pair your metric
with an opposing one.

Nobody honest person is going to say "we'll increase time on site by
decreasing load time". But on my current project, I've had to take into
account that the improved load speed and UX has kept time on site the same.
This is actually an improvement as uses are able to interact with the app
quicker and therefore their relative active time on site is higher.

~~~
true_religion
Ironically, I have seen cases where increased load time lead to higher
conversions by getting people to read the page copy rater than skimming it.

------
colourincorrect
Protip: the quality of any metric used to measure performance will ultimately
degrade over time, as the metric itself becomes the target for optimization
instead of actual performance. any metric can be gamified

~~~
FabHK
Also known as _Goodhart 's Law:_

"When a measure becomes a target, it ceases to be a good measure."

[https://en.wikipedia.org/wiki/Goodhart%27s_law](https://en.wikipedia.org/wiki/Goodhart%27s_law)

------
tlogan
Like this blog post.

1) Vanity metric: great. First page on HN.

2) Clarity metric (getting right type of readers and subscribers): not so
much. Since I do not think target audience like "sign up" pop ups in the
middle of reading the blog.

~~~
mst
It was kind enough to give me TWO pop ups - one because I tabbed away to
another window for a bit, and then after I closed that a second one when I got
3/4 of the way through the article.

One is at least understandable. The second resulted in me closing the tab
because fuck you.

------
fullshark
I think it's more that "vanity" metrics are those that get reported by
companies to the press, and by higher ups to their employees, because in both
cases they want to give the appearance that everything is going great.

In my experience the key decision makers see the reality of the situation and
care about measuring what's actually going on.

------
defined
tl;dr measure the right things, not just those that make you look good.

