
Amon - Python-powered server monitoring, logging, error reporting with JSON API - jemeshsu
http://thechangelog.com/post/17116309951/amon-python-powered-server-monitoring-logging-and-error
======
antoncohen
At first I was really excited about this, I've spent the last couple days
thinking about how I would architect a similar setup. Integration of
monitoring, trending, and logging. Good graphing without rrdtool. Looks good.
Digging deeper I see issues.

The free version is single server (monitors itself) and doesn't have alerting.
Monitoring without alerting has its uses, but that's not what I'm looking for.
Also, single server isn't what I want. There appears to be a planned payed
version that supports multiple servers and alerting (check the page source for
a link).

In a previous HN discussion of Amon
(<http://news.ycombinator.com/item?id=3313991>) it was mentioned that the
install script had errors. Looking at the install script was the first thing I
did. It's kind of insane. It's the kind of thing it might be OK to do on your
own servers, but asking others to run it is really dangerous.

The installer starts off by running "type dpkg &> /dev/null". If that exits
cleanly is downloads a Debian version of the installer and executes it. If
type doesn't exit cleanly it does the same check from rpm and runs the rpm
version of the installer. dpkg is available in the standard Fedora repository.
rpm is available in the standard Debian repositories. /etc/debian_version and
/etc/redhat-release might be a better check.

The installer goes on to download binaries on MongoDB and install them in
/usr/local/mongodb. It puts the mongo client in /usr/bin. Does "apt-get
install mongodb" not work? It consistently does "tar -zxvf filename.tar.gz >
/dev/null". tar -v is verbose, why direct verbose output to /dev/null? The
init scripts _are_ the client and server, written in python. It works but
that's odd, I want my executables in bin or sbin.

There seems to be a general lack of systems understanding, which is not what I
want from a system monitoring tool.

If you are looking for server to accept and store application logs, check out
logstash (<http://logstash.net/>). It will accept pretty much any non-binary
input on any protocol, convert it to JSON, and output it to pretty much any
storage system using any protocol. E.g., input AMPQ -> output Redis, or input
syslog -> output MongoDB. In includes graphing and searching of your logs if
you use the default output storage.

~~~
martin_rusev
Thank you for the feedback.

For the Mongo installer - I really, really, really wanted to install it with
one line of code, instead of writing 100 lines bash spaghetti, but that's not
possible - it works on some distributions ( it works fine on Ubuntu 11.04+,
but I always had problems with Ubuntu 10.10 and CentOS 5.5 ) and it doesn't
work on others.

The problem with the logging are not application logs in general. I started
working on that module, not because I didn't like the way I store my log
entries, but because - I wanted flexibility on a program level. Every
programming language out there has a diverse set of datastructures, but the
logging modules are always limited to strings and levels. So you can't really
"personalize" and "humanize" your log entries and also - you have one
additional step - to convert the log entries to strings. And that depending on
the language is between 1 and 4-5 lines of code. And these lines really add
up.

I fixed the OS detection script, thanks for that

~~~
antoncohen
I really appreciate the hard work you have done, like I said, it is something
I want, and I haven't put in the effort to do it. I know how annoying it is to
log JSON, I made a tornado app the logged JSON through the logging module, and
it was really annoying that the logging Formatter pickled objects into
strings.

I'm not sure what the issue is with MongoDB. But if you aren't aware there is
the EPEL (Extra Packages for Enterprise Linux) repository for RHEL and CentOS.
It's a semi-official and safe repository run by the Fedora Project to add
addition packages to EL. MongoDB is in there for EL5 (pretty old) and EL6.
Also, easy_install is available for Red Hat and Debian in the 'python-
setuptools' package.

I know it's common for people to install things form source on their own
systems, but I feel that if you are doing something on someone else's system
you need to extra careful. And that means ask permission before installing,
use package management, and if you can't use package management suggest ways
to resolve the dependencies (easy_install, pip, gem) without actually doing
it. See the Homebrew install script as a good example
(<https://gist.github.com/323731>). Another thing is following convention. The
way you are using /usr/local is how /opt is supposed to be used. If you use
/usr/local the files should be in /usr/local/{bin,etc,lib}. If want to create
a package directory and have things like 'bin/' in it, use something like
/opt/amon/{bin,etc,sbin,var}.
<http://refspecs.linuxfoundation.org/FHS_2.3/fhs-2.3.html>

Jordan Sissel, who created logstash, also created fpm (Effing Package
Management) (<https://github.com/jordansissel/fpm>). It makes it easy to
package things. It might help avoid bash script hell.

Create an rpm from the files in /tmp/install:

    
    
      fpm -s dir -t rpm --name amon --version 0.2.0 --depends mongodb --maintainer "Martin Rusev" -C /tmp/install etc/init.d/amon etc/init.d/amond opt/amon
    

Create a dep of tornado from pypi:

    
    
      fpm -s python -t dep tornado

~~~
martin_rusev
Thanks for pointing out fpm. It looks really neat. Also with the Homebrew
example, you made me realize that I don't need the bash installer at all (
It's extremely hard to maintain and extend, but that's the default convention
- one line Unix installers have to be in bash script). Python and Ruby are
perfect scripting languages, so the next installer will be in one of those
languages with more options for the user.

------
sciurus
Discussion from two months ago- <http://news.ycombinator.com/item?id=3313991>

------
IgorPartola
I wonder why they didn't use SNMP instead. It seems to be the standard for
this type of thing.

~~~
Dobbs
I don't use SNMP neither do any other operations people I know. Its interface
is complicated, hard to navigate and a pain to manage.

It's still used for network monitoring since that is what routers and switches
talk but most people use something else for everything else.

I've see Ganglia, NRPE, Munin, custom servers and my personal favorite: bash
scripts with graphite.

None of these quite match what Amon is trying to do but they all can get you
an approximation of it. I'm personally going to have to look more at Amon. It
might end up installed shortly.

------
someone13
Pretty cool! Does anyone here have any "real-world" experience with this?

