I'm not sure why it is necessary to add Monit into the mix given that the distribution is using systemd, and a `Restart` (and maybe also `RestartSec`) directive in the `[Service]` section of the unit is all that is required?
The biggest argument in favor of systemd that I heard was "it's installed everywhere." To me that's not a good enough argument. I mean, after I've said this people respond with another argument non-argument: it's the newest and looks like the future. I sincerely think that there's significant benefits to choosing and using systemd but good arguments for them haven't been made to me yet. Secondly, I've been using Monit for a while now, and like some of the comments below, it does exactly what I need. I use it to monitor other things apart from processes too and it meets my expectations over there too. I thought I should leverage my expertise when it came to Monit-oring a process.
Monit is a gem that doesn't get enough attention. The developers also have a web app you can easily install to aggregate a bunch of monit instances with a nice web interface, alerting, and historical statistics. Absolutely the best value in systems monitoring for small-to-medium environments. https://mmonit.com
Have used monit instances on every server I manage for nearly 10 years and have been using m/monit since about a year ago (finally provided them with some money after all these years) and with it, I can check every servers' health from a single web app, I can make alert message to Mattermost (or anywhere else as it can run custom script) and its performance metric graph capability is good enough I dropped munin for it.
The great thing about it is that it's absolutely lightweight and its config files are so easy to understand and has broad checking mechanism. I can also check SSL certificate ages and in case certbot fails to renew them, I get warned well before the expiration date.
If you are running Go you should check out Mike Perham's inspeqtor, which is Monit-like but also Go and therefore potentially simpler for dependencies.
Maybe you could restart the app before peak of memory usage and OOM? If the app performance depends on OS cache hits then this e.g. process line will prevent its long slow death:
if total memory usage > 80% for 2 cycles then restart
Good point. But that would be me taking over the work of the OOM killer. It's possible for the app to hit 80% memory usage without causing OOM, when there's no other expensive process starting or demanding more memory. That's why I defer to the OOM killer. Also if the app hits 80% memory usage it's probably at its peak activity. Restart and it will quickly climb to >= 80% again. It will be restart games at peak activity, which won't be good at all.
I think that would only help when the problem is that your application is using too much memory in the first place, in which case the real solution is to fix your app so you're not leaking memory. But it looks like what's happening here is that something else on the system uses enough memory to trip the OOM killer, which case a preemptive restart won't really help.
Absolutely. I think the app may have interesting challenges, like whatever some real-time stream processing could be introduced to avoid loading entire stuff into memory or even to skip disk IO.
But I assume the OP did some effort/gain calculation which made him choose the existing path.
OP here. You're right. The processed outputs and the recorded tracks are expected to be available <= 2s after the stream ends. That forces our hand to not be able to use memory conservatively. For example, one way to avoid massive memory usage will be to record concurrently (this has small memory footprint since we flush to disk after the write buffer hits 512KB so for about 50 concurrent recordings we need ~25MB), but process and upload sequentially. The recorded audio tracks are big, and have to be loaded to memory since the other tools we use for processing can't work with file on disk yet. The <= 2s requirement is impossible to meet when things are done sequentially.
It seems like this would be easiest to do with a cgroup memory limit and supervisor process. In systemd or kubernetes, this should be pretty easy, IIRC.
