
Ask HN: Replacement backend for RRD - crad
I am currently re-engineering an stats tracking, analysis, and alerting application which is based upon rrdtool.  There are currently roughly 20k data points being monitored every minute and pushed to rrdcached. I'm seeing a substantial performance impact as the quantity of data points grow, even though I am running it on a fairly big box, as the initial implementation of our app targeted roughly 1000 data points.<p>Ultimately I'd like to not stress about scaling the application to 100k data points.  I've thought of using Redis, PostgreSQL and Hadoop/HBase, all which are tackling different domain problems. One of my concerns is if I remove the lossy trending of rrd, the size of the data footprint is substantial at 20k data points (20k * 1440 * data lifetime * per-row-overhead).<p>Do you have any suggestions on a data backend, preferably something lossy like rrd?
======
pedoh
At a previous job we were using Ganglia (<http://ganglia.sourceforge.net/>) to
collect both system metrics and custom metrics from our servers. At my current
job, we still use Ganglia for redundant storage of our metrics, but then we
feed the data into Graphite (<http://graphite.wikidot.com/>) for graphing.
Graphite uses Whisper for it's storage mechanism. It's very similar to RRD.
The author wrote a page about why he decided to write his own instead of
leveraging RRD (<http://graphite.wikidot.com/whisper>). We've got over 60k
metrics in our system; happy to talk shop if you want more info.

------
delano
I've worked a lot with Redis over the past 6 months (my own projects +
analytics for yellowpages.ca) and not so much with rrd. Would you be
interested in an email exchange?

