
Andreessen and Mixpanel Call for an End to “Bullshit Metrics” - billclerico
http://allthingsd.com/20121217/andreessen-and-mixpanel-call-for-an-end-to-bullshit-metrics/
======
physcab
The truth is that there is no one single metric that rules them all. You have
to look at a suite of metrics to fully understand how users are using your
product. It might be nice for YouTube to measure numbers of views a video has,
but what if the majority of those views come from the same person? What if
that person doesn't come back to YouTube? It might also be tempting to just
measure the retention of users. But that doesn't give you the whole picture
either. Is a user who comes in on Day 1 and drops $99 but never comes back
again worse than a user who uses your app everyday but never pays?

For real businesses (read: those with actual business plans) the closest
single metric of importance is Customer Lifetime Value. And the equation is
very simple. Make your cost of customer acquisition less than your LTV and you
will be making money.

~~~
ritchiea
First of all, they are not saying there is one true metric that rules them
all. They said each site will have one key metric, which makes a lot of sense.
Each site will most want to measure how users engage with their domain. The
example mixpanel offers is that Instagram wants to know how many users
uploaded a photo. This matters way more than pageviews, etc. It tells you
something about user engagement.

The bigger issue I see in your comment is that sites at different stages in
their lifecycle will want to measure different things. Customer lifetime value
is the most important number for a mature business but early stage startups
are nowhere near getting a realistic number for customer lifetime value. If I
am the founder of a startup I want detailed knowledge of how users are
interacting with the product I created to address a problem domain. I can get
that info immediately after I start acquiring users, well before I have an
idea of whether I have a viable business and what my customer lifetime value
is going to be.

~~~
simonholroyd
It seems to me, if Instagram could really only chose one metric to track it
would be an app visit, not a photo upload. Instagram must have users who are
primarily content creators and another larger group (probably a superset) that
are content consumers. Not focusing on the actions of the content consumers
would seem really bad for business since what early-stage Instagram's
investors care about is eyeballs not photographs and what late-stage
Instagram's advertisers care about is eyeballs not photographs.

Really Instagram probably has a few OKMs (visits, uploads, hearts, comments,
follows). All of those are important and should be tracked. In their
documentation and educational videos, I've not seen Mixpanel focus on visits
as a key metric. I'm not sure why.

------
BrianEatWorld
I think theres a difference between advertising or celebrating "bullshit"
metrics and actually using them.

For our most recent press releases, we reported many of these "bullshit"
metrics because thats what gets acclaim and attracts attention to help the
company gain the necessary notoriety to thrive. Internally, we still use real,
actionable metrics to drive our development. We just don't report them to
outsiders because its either sensitive information and likely uninteresting
outside of the context modeling or users experience.

~~~
001sky
_Internally, we still use real, actionable metrics to drive our development.
We just don't report them to outsiders_

\-- Pretty much how the world works.

------
mattmaroon
A little ironic coming from the firm that invested in Groupon, inventor of the
"Adjusted consolidated segment operating income".

------
cwilson
The first most important metric: When it goes up and to the right, it means
your company doesn't die.

Death can be alluded in many different ways, depending on the nature of your
business, but it usually means making money, hitting a milestone that allows
you to raise money, acquiring users, and so forth.

Once your company evolves, and has beaten death for the foreseeable future, a
new set of metrics come into play and are now the most important. These may be
similar to the death-evading set above, or they may be something new like
increasing revenue, retaining users, increasing engagement, optimizing CLV,
and so forth.

The point? If a metric means your company is not dying, or your company is
growing, it matters. If it doesn't fit into those buckets? Probably doesn't
matter.

------
throwaway-29382
Good thing they don't use any BS metrics on their about page.

<https://mixpanel.com/about/>

~~~
suhail
We don't. Data points (or actions as we say on the about page) correlate
directly with the amount of revenue we make. 85-90% of all data points are
paid for by our customers. It's how we make money:
<http://mixpanel.com/pricing>

It actually does a better job correlating with our growth than purely the
number of customers we have. We've also been public about that.

~~~
physcab
It makes sense that data points correlate to your growth, but what's
interesting is that data points do not necessarily mean higher value for your
customers. If there was a metric that only required one data point to measure
and that metric could be the basis for a successful company, people would
probably use less data points and gain more value. But most people don't know
what to measure, so they default to measure everything. Thats good for you,
but doesn't necessarily mean its good for them.

~~~
suhail
That's a valid point. The thing is, our growth in data largely correlates with
customer satisfaction. If a customer finds no value but sends us a lot of
data, they stop because they are paying for our product. That causes a
decrease in data points.

Admittedly, we do use a different metric to determine how "valuable" our
product is in a more honest way. We're talking about non-bullshit metrics that
highly correlate to growth. Not all valuable products necessarily grow.

~~~
mbesto
What qualifications (or quantifications) do you use to ensure that your value
is more honest than someone else's? At the end of the day, value is determined
by someone who is not yourself.

This article for more me is pretty straight forward. It's a well crafted PR
play (whether made by commercial means or not is irrelevant), that draws
readers to putting more importance on what metrics are used to determine
valuation. Something that they can draw from using a service such as Mixpanel.
The reality is that you, nor I, can definitely provide one metric that clearly
correlates to financial valuation. Every case is different and every case
warrants negotiation.

Mixpanel appears to be a great service btw. I hope my tone doesn't dictate
otherwise.

------
Avitas
This will not happen. Individuals and corporations use these because they make
them look better.

Should companies selling, say, food products be mandated to show pictures of
the actual delivered product on advertisements, menus and packaging? That's
not going to happen without additional legislation.

Should resumes of individuals highlight workplace weaknesses, medical
conditions, personality faults, past failures, major conflicts, etc.?

The web is no different. Unless individuals and corporations are persuaded to
do otherwise, they will continue to use metrics as they please.

------
bryanh
I really love Mixpanel, it is the "best in breed" when it comes to event
analytics. I don't think many people would argue with that, it's awesome. In
fact, we use it at Zapier for pretty much everything (basic funnels, A/B
testing, retention, engagement, etc...) and we haven't even reached our full
potential with it yet. But I still have some pretty fundamental beefs with the
state of analytics software in general.

About the article, the problem stems from the fact that proper analytics is
hard and is (arguably) getting harder with more advanced packages.Shouldn't it
be going in the opposite direction?

It is a lot easier to track discrete downloads or pageviews than some other,
more insightful metric, so people will naturally gravitate to the cheaper
metrics. Until this is reversed, bullshit metrics will reign.

Anyways, my beefs:

 _First_ : how do you decide what data to send into the package?

The more data you send, the better (sure), but at a certain point you are just
duplicating your internal datastore, so that is too much, right? But not
enough and you'll miss a chance to understand a phenomena that you didn't
predict seeing (isn't that the point?). After you decide, then you write a
crapton of code to send it all (what about backfilling data when you want to
track something new?).

 _Second_ : once you are collecting the data, how do you know what metrics to
actively track?

This is definitely existential, but it's back to the core problem: doing
analytics properly is _hard_. Why couldn't the software let me define some
properties about the _type_ of app I am running and suggest some strategies
(you have a subscription SaaS app? Try tracking paid plan retention, signup
funnels, etc...). Maybe it could go even further with _reverse_ funnels, as
in: what events are the most important and work backwards. I could see some
automation and discovery possibilities there.

 _Third_ : do I really have to dig around trying to find something useful?

All the data is there, the software should _tell_ me what is useful or
interesting. It's definitely a hard problem, but I would throw money at
software that could send me this email: " _Looks like users who experienced
event "ABC" also performed your highest priority event "Signup" at a 13%
higher rate. This observation is 99% confident._ " Of course, you'd need to
investigate a littler deeper to see if that isn't just a fluke or something
stupidly obvious (like: people who view a page signup at a higher rate than
those who don't), but at least I might learn something.

I know this is certainly a pipe dream as of today, but I vow to shower someone
with money if they can do this.

In my opinion, the next generation of analytics software won't just have more
bells and whistles, it will fundamentally shorten the time to some sort of
real "AHA!" insight.

~~~
suhail
We deal with this every day at Mixpanel. Education is a tough problem to
solve. Here's how we think about it with respect to your questions:

"First: how do you decide what data to send into the package?"

My recommendation is you think about this gradually. Don't instrument
everything. Start by picking 5 metrics you really, really care about. One of
them should be your One Key Metric (OKM) - a metric that you would be the
company on if you could only pick one thing to measure. I think the pressure
of picking 5 helps you decide what to measure.

"Second: once you are collecting the data, how do you know what metrics to
actively track?"

When you start gradually, there isn't much to pay attention to. Stay focused
only on a set of metrics. Add more metrics when you want to understand those
specific metrics more deeply. Add more data to split those metrics into groups
to dig even deeper. Keep driving your OKM up as much as possible--sometimes
you have to do it with something that correlates to it.

"Third: do I really have to dig around trying to find something useful?"

Today, the answer is yes. Businesses are fundamentally very different.
Analytics is a lot like science experiments in high school: You start with a
hypothesis, you test to see if you were right, and if you were then you came
up with a conclusion or answer to your experiment. It's hard work to build a
good business. I think the first step is for companies like us to help you
understand your data. The next step that I think is more important than simply
pulling "insights" out automatically is to help you take action on it so you
grow.

~~~
bryanh
Thanks for the response.

I just signed up fresh to Mixpanel to play around and you guys have solid
technical on-boarding around just installing and collecting _any_ data, but
not a lot of details on _what_ to collect. I'm sure that getting someone to
collect _just anything_ is a better first step than some sort of high level
tutorial, which is why you do it that way.

I like the OKM idea a lot. Perhaps that could be a pretty solid center stone
for on-boarding a new user. I just remember being overwhelmed by the blank
slate every time I signed up for any analytics package.

~~~
suhail
We generally reach out on the support side proactively to help. Anyone on our
team is happy to think it through for you.

If you want to learn more in the abstract we recently came out with an
education series: <http://mixpanel.com/education>

------
kevinyun
People will always be measuring everything by the numbers. Look at your number
of Facebook friends, your LinkedIn connections, your GPA in college, your
annual income. It's always a rational argument to say that bigger is better.
In a world where metrics are quantified like this, this is the natural way of
thinking. It's going to be difficult to seep through the bullshit metrics,
especially when you're such a small startup with other numbers that are too
embarrassing to display to the world -- but, really, when you think about it
this is the only way to find and fix the problems that you have.

------
evolve2k
I would have liked the article to pony up what the companies own 'non BS'
metrics are. Seemed a little flat to 'call BS' on everyone else then not
deliver the goods itself.

------
majani
Actually, I find that a lot of the metrics mentioned like uniques and
pageviews are the ones that affect revenue for ad-driven startups. That's as
real as it gets with metrics.

When it comes to ad-driven startups, you could actually say that the "real"
metrics suggested, like active users and engagement, are the actual bullshit
metrics. For many ad companies like Google(search) or YouTube, high engagement
by the users adds nothing to the bottom line.

------
fidanov
So they call to replace one bullshit metrics with other bullshit metrics.
Nice.

Not once in this article there were the words "Profit" or "Revenue" and yet
they talk about business and companies.

Here is a surprise for you. Real companies are interested in real profits from
real revenue and you count revenue and profit only after the money is actually
in your bank account.

Otherwise one day you get the kind of terms of service just like Instragram
these days.

~~~
marcuskaz
Yeh, if there is one true metric it would be profit

------
JacobJans
Analytics only matter if they help you make better decisions. Anything else,
is, indeed, just vanity.

------
mrdavidjcole
This article inspired me to create a twitter handle to highlight the best(?)
examples of this: <http://twitter.com/bullshitmetrics>

Anything recent come to mind?

------
robryan
An example in the article, number of pictures uploaded on Instagram, it is not
clear that this is a good metric, at least until they work out a way for each
additional image to bring in more revenue.

------
karolisd
This seems to be focused on web app start ups. What the most important metrics
to track for eCommerce?

~~~
suhail
It depends on what you're willing to be public about. I think your OKM is
people buying something and looking at all the metrics around that.

Great metrics for e-commerce are churn, lifetime value, repetition of
purchases, average revenue per user. On the marketing side, I think it's
useful to understand what kinds of marketing produce certain kinds of repeat
revenue.

Publicly, I think understanding how many customers purchased something in the
past 30 days is useful. Potentially, the number of customers that purchased
something above $X.

~~~
joshwa
OKM for eCommerce is revenue growth TY/LY by week, closely followed by average
order value, average items per order, and conversion rate

------
endlessvoid94
Gaming companies have been doing this already for years.

------
joe-mccann
The need for new metrics! Written about extensively here months ago:

<http://subprint.com/blog/the-need-for-new-metrics>

------
mmaunder
Everything except bottom line profitability is a bullshit metric.

~~~
wildwood
Until you can't make payroll because the deals that you booked as profit two
months ago haven't paid you any money yet.

Then cash flow suddenly becomes essential.

