
Monitoring Go Applications with Monit - yawboakye
https://boakye.yiadom.org/go/monit/
======
jen20
I'm not sure why it is necessary to add Monit into the mix given that the
distribution is using systemd, and a `Restart` (and maybe also `RestartSec`)
directive in the `[Service]` section of the unit is all that is required?

~~~
JdeBP
Possibly not _all_. There's OOMScoreAdjust as well. (-:

~~~
yawboakye
I found this disingenuous.

------
tamasnet
Monit is a gem that doesn't get enough attention. The developers also have a
web app you can easily install to aggregate a bunch of monit instances with a
nice web interface, alerting, and historical statistics. Absolutely the best
value in systems monitoring for small-to-medium environments.
[https://mmonit.com](https://mmonit.com)

~~~
h1d
Agreed.

Have used monit instances on every server I manage for nearly 10 years and
have been using m/monit since about a year ago (finally provided them with
some money after all these years) and with it, I can check every servers'
health from a single web app, I can make alert message to Mattermost (or
anywhere else as it can run custom script) and its performance metric graph
capability is good enough I dropped munin for it.

The great thing about it is that it's absolutely lightweight and its config
files are so easy to understand and has broad checking mechanism. I can also
check SSL certificate ages and in case certbot fails to renew them, I get
warned well before the expiration date.

------
wyaeld
If you are running Go you should check out Mike Perham's inspeqtor, which is
Monit-like but also Go and therefore potentially simpler for dependencies.

[https://github.com/mperham/inspeqtor](https://github.com/mperham/inspeqtor)

------
robxu9
I see that you created an init script, but then you called systemctl. Any
reason not to use systemd units?

~~~
JdeBP
Then one would not have the fun of writing

    
    
        #! /bin/sh
        ...
        source /lib/lsb/init-functions
    

and have it not work on systems where /bin/sh is (say) the Debian Almquist
shell. (-:

------
imhoguy
Maybe you could restart the app before peak of memory usage and OOM? If the
app performance depends on OS cache hits then this e.g. process line will
prevent its long slow death:

    
    
        if total memory usage > 80% for 2 cycles then restart

~~~
yjftsjthsd-h
I think that would only help when the problem is that your application is
using too much memory in the first place, in which case the real solution is
to fix your app so you're not leaking memory. But it looks like what's
happening here is that something else on the system uses enough memory to trip
the OOM killer, which case a preemptive restart won't really help.

~~~
imhoguy
Absolutely. I think the app may have interesting challenges, like whatever
some real-time stream processing could be introduced to avoid loading entire
stuff into memory or even to skip disk IO. But I assume the OP did some
effort/gain calculation which made him choose the existing path.

~~~
yawboakye
OP here. You're right. The processed outputs and the recorded tracks are
expected to be available <= 2s after the stream ends. That forces our hand to
not be able to use memory conservatively. For example, one way to avoid
massive memory usage will be to record concurrently (this has small memory
footprint since we flush to disk after the write buffer hits 512KB so for
about 50 concurrent recordings we need ~25MB), but process and upload
sequentially. The recorded audio tracks are big, and have to be loaded to
memory since the other tools we use for processing can't work with file on
disk yet. The <= 2s requirement is impossible to meet when things are done
sequentially.

~~~
imhoguy
Check memory-mapped files, may possibly work for you
[https://godoc.org/golang.org/x/exp/mmap](https://godoc.org/golang.org/x/exp/mmap)

