

Ask HN: Custom logging for web applications - almost

I have an application (written in Django) that needs to log lots of data (visitor stats from a Flash app, a url is pinged whenever users do certain things). I've rolled by own logging system along with a library for extracting statistics from the logged data based on the map/reduce pattern. However, I'm having performance problems with my initial quick and dirty implementation (in the stats generation, not in the logging bit yet). This seems like a problem that must have been solved plenty of times before. What do people use for this sort of stuff?
======
rarrrrrr
A few months ago we actually created a system that does exactly this, plugging
into the Ganglia data visualization system many sites use. We have bindings
for Perl and Python. Your app code just emits non-blocking UDP packets to a
local collector daemon and goes on with its business. GPLv3 code here:
<https://spideroak.com/code> (StatGrabber)

"If you can't measure it, you can't manage it."

It's really nice to see graphs of the business or application specific stuff
(signups, connected users, revenue, anything that's event driven) in parallel
with the regular system graphs of things like bandwidth, memory, and so on.
When you look at the weekly or monthly graphs usage trends and bottlenecks
become forehead-smacking obvious.

Ganglia demo: <http://monitor.millennium.berkeley.edu/>

------
bayareaguy
I'd suggest you send all your custom stuff to syslog so that you can easily
consolidate events however you like and then work with your admin to deploy
whatever variant of syslog best fits your requirements. Personally I think
nsyslog is good. Some others are listed here:

[http://www.loganalysis.org/sections/syslog/syslog-
replacemen...](http://www.loganalysis.org/sections/syslog/syslog-replacements)

