
A heartbeat monitor for software as a service - qixxiq
http://blog.snapbill.com/heartbeat-monitor-for-software-as-a-service/
======
mmcconnell1618
Nice article on how heartbeats are used. I would like to know more about the
technical implementation. Do they send heartbeats via HTTP Post? Email? Some
other mechanism?

What monitors the heartbeat server to make sure it is up and running as
expected?

~~~
qixxiq
The technical implementation is rather simple, and actually not too important.
It doesn't need to be super-reliable since if it breaks the logic will report
it to us.

Currently we simply write each heartbeat to our redis database, along with the
UTC time it occurred. Some heartbeats are global (string key) and others we
have one per server (hash key, server name => last heartbeat). There is an API
endpoint set up that some of our systems use to trigger heartbeats.

We then have a public page at a secret url that renders out the full status
check. Pingdom is set up to check this page for "warning-status: OK" and if
its not there send us an email. It also checks for "final-status: OK" and if
that is not there, it triggers PagerDuty which will wake us up if needed.

I would prefer if PagerDuty could check it directly, then they would be our
only point of failure.

\-- Edit -- I added a screenshot to the post which should make it a bit
clearer.

------
beeffective
Nice article, I would be curious how it is implemented and how you setup the
rules for alerting.

I'm working with Microsoft's Semantic Logging Application Block to standardize
error logging and auditing across all my .NET apps. It utilizes the built-in
Event Tracing for Windows (ETW) to capture events written by any application,
and writes them to a destination of your choice (db, file, etc). .NET 4.5
includes a new EventSource class that I inherit from, and simply call
WriteEvent() and the message goes off to ETW. Reference:
[http://blogs.msdn.com/b/agile/archive/2013/02/07/embracing-s...](http://blogs.msdn.com/b/agile/archive/2013/02/07/embracing-
semantic-logging.aspx)

I would like to add heartbeats to my design, in addition to just error
logging, tracing and http auditing.

------
daniel-levin
Awesome to see a South African company on the front page of HN. Keep it up,
guys! Just two questions though - how have you secured the heartbeat? An
attacker could take down your servers and forge a heartbeat. How do you
prevent that?

------
darkarmani
Can't this be piggy-backed on syslog collection? If you have centralized
syslog and a facility for heartbeats, you could post-process these heartbeats
when log parsing your "heartbeats" log while only using up resources on your
parsing machine. Syslog-ng provides tcp logging.

------
lazylizard
something like this <http://www.pushmon.com/> ??

