

MixPanel (YC S09); my mini review - jacquesm

After spotting the mixpanel logo at the bottom of HN I decided to give it
a spin on a reasonably serious site to see what it can do. I've got some
issues with GA that I can't seem to resolve so I figured maybe MixPanel
can replace them.<p>First impression is that it looks really neat and simple, but on my browser
(ff 3.01) the form fields all somehow manage to hang outside the containing
blocks.<p>The sign up process is very quick, but there is no email verification, which is
a bit of a pain because it could mean that people use a fake address leading
to emails dumped on unsuspecting bystanders.<p>After signing up you get a secret token and a skeleton piece of javascript
that you can flesh out to track variables that are important. This is mostly
a manual operation, you insert the relevant bits in strategic spots on your
site, 'instrumenting' it so MixPanel can keep track of what is happening.<p>The upside of this approach is that you, not they decide exactly what gets
tracked and what it is called.<p>Right after signing up you are offered to get some 'datapoints' if you get
someone else to sign up, that seems a bit premature, I haven't a clue what
datapoints are.<p>Other than the initial signup form error the design of the site is very clean
and functional, the documentation is easy to read and stuff can be found
more or less where you expect it.<p>The 'secret' token that gets generated for your site is not that secret
however, since it is plainly visible in every page that has the javascript
embedded in it. Maybe it should be named 'site specific' token ?<p>Meanwhile the tag is in place on my site now, and registering visitors
according to the live stats, time to read up on the data points.<p>Datapoints are mixpanels way of making a business out of this, essentially
they charge you 'per event' on your website. The whole thing revolves around
the 'funnel' concept that anybody in internet marketing is familiar with.<p>Garbage in at the top, and 'espresso' out at the bottom, mixpanel allows you
to track exactly what the loss percentages are at every stage of the process.<p>Datapoints cost money, and I think they are <i>way</i> overpriced.<p>The current pricing scheme is:<p><pre><code>    0...10,000 points:          Free
    10,000...100,000:           $25
    100,000...500,000:          $100
    500,000...2,000,000:        $200
    2,000,000...10,000,000:     $700
 </code></pre>
Let me explain why I think they are overpriced: at 100,000 points for
$25 (the first 'plan' they have)  for any small ad driven website that
has an ECPM &#60; $0.25 they are costing you as much (or more!) for your tracker
as what you were making on your visitors.<p>And on a for-pay site that makes $20 / month per user, with a COA of $5 and
a 1:2000 conversion rate on a 1:50 new traffic vs old traffic ratio (all of
those numbers are pretty 'normal') you would again be making a net loss.<p>One of the sites I run that I could have tried mixpanel on gets 3,000,000
pageviews monthly, the ECPM is around $0.11, so it makes $300 /month.
It's a no-brainer because it costs less than $5 to host on a flatrate 100Mbit
leaseweb box. Using MixPanel with that site is out of the question though.<p>So, you'd have to be buying a very large volume of points in one go to make
it worth your while, but that means a fairly big up-front cost. $700 in
one go is essentially several months hosting cost for many sites. The
price difference between 'volume buys' and smaller buys here is
$c0.007 / hit on the largest plan, versus $c0.025 per hit on the smaller
ones. That's a 350% or so difference. For a virtual product that is
not reflecting the difference in cost to the provider, which essentially
is only the extra cost incurred by the transactions to process the
payments.<p>MixPanel currently uses Amazon as their payment processor, which
is a pretty good indication you don't need to worry about where your
credit card details will end up.<p>The real time statistics are neat, but they will also be very expensive for
MixPanel to maintain long term, and they don't seem to add much actual
value, it's more of a gimmick. A functional real-time stats tracker that is
cheap to operate is very hard to design, which is one of the reasons all the
free tracking tools are using batched updates.<p>Google Analytics also allows funnel analysis using the 'goal conversions', only
they do it for free and a whole lot more besides.<p>Meanwhile, I've set up a secondary instrumented set of pages, but for some
reason the data never showed up in the reports. I double checked the code
and the javascript error console, no idea what is going wrong there. The
'feedback forum' seems to be mostly enhancement oriented, not set up
to track problems. It is also externally hosted, which means that you leave
the site, it would be better to have a simple ticketing system (a la trac)
inside the MixPanel environment, where there is a lot more context available.
Incidentally, uservoice, the site that hosts the forum is listed as a customer
of mixpanel, but they also use analytics and quantcast.<p>MixPanel will probably work well if you have well defined landing pages and
are a straightforward e-commerce, but at the current pricing levels it is too
expensive for high volume low conversion websites, especially for continuous
deployment. And that's a pity because it is especially those sites that would
benefit most from a tool like this.<p>You could use it for targeted a/b testing though, but setting up a test
seems to be quite labour intensive and error prone.<p>MixPanel provides you with csv exports of the reports so you can import the
data in to your favourite spreadsheet.<p>On the whole I think that the concept of an analytics site that concentrates
on the funnel aspect of operating a web service has promise, but I would
like it to be significantly cheaper and easier to set up. As it is it really can't replace GA for all the other features it offers.<p>edit: in response to an email, COA = cost of acquisition
======
jack7890
As a longtime GA user, I checked out MixPanel extensively to see if it was
worth switching. I can't find anything that MixPanel does that can't also be
done with GA. You can track any event in GA by calling
pageTracker._trackPageview('page/here') with JS when the event takes place.
It's just as easy as setting up a tracking event in MixPanel. Plus, the
segmentation and analytics are far more rich in GA. And you can also tie it in
with Google Website Optimizer, which makes multivariate testing easier.

GA tries to appeal to non-coders, so they don't widely publicize their ability
to track custom events. But the functionality is there, and it works great.
When you consider the fact that GA has more comprehensive analytics and is
free (jacquesm you're right on target...MixPanel's is bewilderingly expensive)
the choice is a no-brainer.

I love startups that are trying to innovate in web analytics. The MixPanel
folks have clearly put a lot of time and thought into their app. But they have
a long way to go before they can compete with GA.

~~~
qeorge
I've been frustrated with GA's new event tracking actually, and was going to
try Mixpanel as an alternative. Here's my qualms with GA's events, maybe
you've been able to work around them?

1) It doesn't tell me which page the event happened on

2) You've only got 3/4 pieces of data you can send (category, action, and an
optional key/value pair). Even if you shove data into fields where it doesn't
belong (i.e. category), its hard to send all the info I want.

As Mixpanel will allow me to send as many fields as I need, it looks like the
right solution. However, as jaquesm noted, its just too expensive. I'm
frustrated enough that I've considered rolling my own event tracking, but I'd
much rather find an existing solution that works and won't break the bank.

------
xal
I'd go back to the drawing board with the pricing plan. Usage based metering
is really good for bottom of the barrel pricing ( electricity at home, amazon
aws for servers ). For everything else people want predictability and
therefore monthly subscriptions.

------
trefn
Hey jacques, thanks for the review and the honest feedback. I'd be happy to
help you debug the second set of pages you mentioned - tim@mixpanel.com.

~~~
jacquesm
Hey Tim,

I was burning through my 'datapoints' so fast that I have already removed the
tags (I think about 70% had gone by the time I finished my review), I'll re-
implement it on a site that gets a bit less traffic later next week and spend
more time on it.

You guys spent a lot of time on that and it shows.

edit: on another note, how are you set up for handling _really_ large volume ?

~~~
trefn
'really large volume' is obviously subjective but we've had individual clients
send 100M data points in a month.

~~~
suhail
jacquesm, we watch our infrastructure all day long and while we may not have
enough hardware if a company like Zynga showed up tomorrow we're prepared to
provision more nodes to handle scaling and work through it just like any other
company would. The point though is: we can scale. Additionally, we'll be fine
as we have many sites that do more traffic than that already on our system. If
you have any concerns feel free to email support@mixpanel.com. The email will
go directly to both Tim and I.

