

Ask HN: Develop your own dashboard for metrics - yes or no? - joshdotsmith

I'm wondering whether its worthwhile to put time into developing your own dashboard for metrics. This would be for the kind of internal metrics you likely couldn't acquire from external APIs/services, things like hourly updates of total # of users, # of users with friends, # of users with a profile photo, etc. You could then also measure things like % increase over X amount of time, or % difference of growth rate between X and Y time periods.<p>For me, this would involve writing some SQL queries, running an hourly cron job, and designing/developing something that makes use of some jQuery charting plugin. I'd also probably like a way to export the data in CSV format. Total time involved is maybe a couple days at most.<p>Is this worth all the effort involved? Do you have experience with doing something like this?
======
btilly
Is there value in doing this? Absolutely. Figure out the core metrics for your
business, measure them, pay attention to those metrics, and figure out how to
improve them. Going one step further, I recommend creating an A/B test
platform for yourself where you can easily measure how A/B tests impact the
metrics you care about. There is a lot of value to be derived here, and in the
end it is going to be an open-ended commitment.

Is it worthwhile for you to do this right now? Only you can answer that. These
things are certainly not part of a minimum viable product. But if you find
yourself looking at certain figures over an over again, then by all means
build a dashboard.

Pro tip. If you're still small, you can probably query your whole database
interactively without any noticeable impact on your site. Throw together a
prototype based on doing that that before you start to get fancy with things
like regular cron jobs. It is faster to put together, and it is easier to make
tweaks. Only after you know you've got the right stuff should you migrate the
queries to cron jobs and refactor your dashboard to hit cached data.

~~~
joshdotsmith
Yeah, I'm almost done with what I'd call our MVP (and that's after spending a
long time working on a very full-featured site that just wasn't as compelling
as the MVP itself), so these metrics are going to help me figure out where to
head next.

I'm not sure yet how I'd go about building an A/B platform, but its on the
docket of things I want to do.

Also, the reason I'd be running a cron job is to have hourly data for all of
these metrics (to look for those changes over time).

~~~
btilly
See <http://elem.com/~btilly/effective-ab-testing/> for my advice on how to
put together an A/B platform. Sections 2-4 are the most relevant.

The most important detail is the warning in slide 34 that you should not use
the person_id to set the A/B test version. The reason is that if you make that
mistake, then you can only run one test at a time, and can't easily drop
versions of tests later. On the other hand if you make assignment random, then
you are free to run tests in parallel, which is _very_ useful for trying out
lots of ideas. (Particularly when you have less traffic.)

~~~
joshdotsmith
Thanks for that!

I actually just signed up with Optimizely today and really love the way they
run A/B tests so far. I need to think a little deeper on how to integrate
these tests with the rest of my metrics, but it's a really good (and easy)
start. While I haven't reached a level of real statistical confidence yet with
my first test, I've already been able to increase engagement with a single
change on my landing page by 35%. Wow.

clickable: <http://www.optimizely.com/>

------
pmjoyce
At the risk of coming across spammy, I'm the founder of a web service called
Geckoboard [1] that might be useful here.

It integrates with 3rd party services such as Google Analytics, Freshbooks,
Pingdom, Mixpanel, Zendesk, Uservoice etc. and we're pushing out new
connections regularly.

Where it might be useful for you is that we also have an API that allows you
to use your own scripts to get data that's in your database onto the status
board. You create a script that outputs XML or JSON in a certain format, add a
widget to your Geckoboard that's configured to poll your script at regular
intervals (support for webhooks is coming) and the data appears on your
dashboard that's accessible anywhere you can get on the web.

We're in private beta but will be offering paid plans very soon. I'll
reiterate my open offer to HNers: if you would like to check it out the beta
then just use the email address in my profile and tell me your HN handle.

[1] <http://www.geckoboard.com>

~~~
joshdotsmith
Heh, I actually already use Geckoboard, and I have created my own custom
widgets for it. It's an awesome service, to be sure; I plan to continue to use
it.

If you revamp your charts at all, let me know, as I'm really trying to get
some more intense data into my dashboard.

~~~
pmjoyce
Great to hear you're using it, thanks!

We're planning a major revamp of all the widgets. In the meantime you can use
a custom text widgets (which accept images and inline styles but no JS) to
render whatever you want. Drop me a line if you would like any help getting
something up.

------
wrath
Is it worth the effort involved?

Yes absolutely it is. I always tell the developers that I work with that the
most important thing we can do is figure out a way to "peek" inside our
systems. Keeping statistics o "background" metrics and displaying them on a
simple webpage using plots can be very useful in discovering problems that
would be otherwise impossible to notice.

There are many things which are easy to QA because you can visually see them.
On the other hand there are many things which are hard to ensure top notch
quality because there's nothing to "see". For example, if your system gets
sluggish how many users were viewing your site, what was the CPU usage, memory
usage, etc. Having cron jobs or other statistical capturing systems which you
can use to store and plot data can help you understand when a problem happens.
Then you can look for similar patterns and debug the issue.

I work with a system which is rich in algorithm and we keep lots statistics on
every major algorithm. That way, if we modify an algorithm or add data to our
system we can see if there's any abnormal behavior. We've learned what
acceptable is and how to identify problems by looking at the plots we generate
with the captured data.

I wouldn't do anything overly complicated. We use the google chart api, easy
to use and very fast. <http://code.google.com/apis/chart/>

------
naithemilkman
Yes without a shadow of a doubt. These are the most important metrics to look
at in fact. It will be helpful as well to identify a (one) core super metric
unique to your business as an overal measure.

E.g. Blogger - Number of blog posts per day Flickr - Number of photos uploaded
per day

After that once you start breaking down into AARRR, you can start defining
more mini metrics to watch and optimise at each stage of the process.

Be careful with data overload. Less is more. Is finding out the number of
users with a personalised profile picture meaningful to your business RIGHT
NOW? If not, shove it and move on to the next metric.

Less is more!!

------
destraynor
Not pimping , but my post on this exact topic was recently on Hacker News ,
<http://contrast.ie>

~~~
aymeric
Awesome post: <http://www.contrast.ie/blog/evolution-of-a-dashboard-design>

------
damoncali
Yes. It's worth it.

Don't jump into charting plugins. Just dump the data you need on some admin
screens and start watching it. If you need a chart, add it later when it hurts
to not have it. Just like any feature, add it only when there is a need for it
- your dashboard is an MVP for you.

You can create enough to get started in a couple hours, not a couple days.

------
proexploit
I've been building my own in Sinatra. Haven't got everything I want yet but if
you're interested in the source, it could save you some time on the design
side of things. It's all CSS3/HTML + Basic HTTP Auth. (Screenshot with
different color "themes" displayed: <http://i.imgur.com/8h9jK.jpg>)

------
fjabre
Yes it's worth it. We made our own and it's the best thing we ever did. It
really illuminates your business and allows you to make better decisions. We
used Fusion charts for the charting bit.

------
ig1
Yep. I'm even building my own dashboard that sits on top of the google
analytics API to give me quick access to more sophisticated analysis than GA
allows.

------
macca321
Just create some excel spreadsheets and import the data into them from SQL.
Then you can munge it all you like using pivottables.

