
Ask HN: Kibana, Grafana, Chatbots, SMS – How do you keep an eye on your product? - BloodKnight9923
I&#x27;ve been playing with ELK (Elasticsearch, Logstash, Kibana) stacks for the past couple months and that has given me great insights into what my product is doing in a clustered state in real time. That coupled with a few monitoring scripts and I&#x27;ve finally gotten past feeling helpless. I have a few people saying I should check out Grafana, and I&#x27;ve seen several paid services in this areas too (logentries for example).<p>I also use chatbots to notify me when things are going wrong, or say when a customer signs up.<p>How do you keep track of your products, keep monitoring costs low, and know what&#x27;s going on?
======
jmkni
I've gotten really into Slack recently.

I just log everything ye old fashioned way (text files) and then post anything
important/noteworthy to a couple of Slack channels.

It's really easy to build custom integrations and Slack is running on all of
our devices so as long as we are all subscribed to the relevant channels I can
target _@channel_ in my integration and we know straight away if something bad
is happening.

I've use the ELK stack on previous projects, it works really well and isn't
that difficult to use (there are some great guides on Digital Ocean) and when
I get the time I'm going to set it all up for my current project too.

The only thing is you have to host it somewhere, and it's more justifying the
budget for that.

~~~
BloodKnight9923
I use slack for a lot of notifications too. We strapped a python chatbot on as
an integration and gave the bot access to our product which made asking the
bot what's happening really easy.

Chatops are catching on, and I'm intrigued. Thanks for the response!

~~~
ghilston
Hey, feel free to checkout my library on GitHub. Its goal is to simplify
interacting with Slack API (which is amazing IMO) and let users focus on
simply writing their bot.

Its still very much in development, but figured this thread was relevant
enough for me to post a link.

[https://github.com/GregHilston/Simple-Slack-
Bot](https://github.com/GregHilston/Simple-Slack-Bot)

~~~
BloodKnight9923
Hah. I went to college with you. Small world. Will check out your library,
I've been doing a lot of work with chatbots over the last year, focused on
Slack. I've only really been working with Java solutions though, so a python
library is a nice change of pace.

Thanks for the link!

~~~
ghilston
Oh hey man! Please do, feel free to use it in its current state, but the dev
branch will be merged to Master shortly and break current compatibility.

Its a sweet simple library to make slack bots real simple!

------
man5quid
Elastalert + Kibana + Slack is great if you use logstash/streamstash to
aggregate logs internally. I also combine new relic and slack for incident
management.

~~~
BloodKnight9923
Never heard of Elastalert, I'll have to check that out. By the name I'm
guessing it's triggers off an elasticsearch server, which would cover
infrastructure I already have - so that's nice.

Thanks for the suggestion!

------
wampwamp
For log aggregation I use Elasticsearch + Logstash + Kibana. For statistics I
use Influxdb + Telegraf + Grafana. For service alerts I use Consul + Consul
Alerts. Notifications get sent to Pager Duty and/or Slack from one of Kibana,
Grafana, or Consul.

The Grafana graphs are especially fun to watch and Kibana has a nice
logentries-esque plugin for searching and tailing logs.

It's quite the system to setup but once it's running everything else (i.e. the
rest of your app) is much more manageable.

~~~
BloodKnight9923
I've been meaning to use grafana for more than homelab server monitoring.

I've always found it so satisfying to sit back after setting up these types of
systems and to watch everything churn out metrics / logs. I'll check out the
kibana plugin, I'm using that in production at the moment.

Thanks for the reply!

------
atmosx
If you're a lone developer (or a small team of developers) isn't a lot better
to use something like CloudWatch instead of messing with ELK?

~~~
wampwamp
For small apps I would use Monit to watch the server processes combined with
CloudWatch to watch the servers themselves.

I would investigate the TICK stack (influxdb) before ELK. TICK is great for
metrics that are not stored in log files. tail + awk should be enough for
managing log files if you only have a few servers.

~~~
BloodKnight9923
I'm only working with logs across 8 servers with a lot of cross talk and ELK
covered the logging side brilliantly, but falls over on metrics from what I
have seen so far. I'll check out the TICK stack, that sounds like it may be a
welcome replacement.

Thanks for the suggestion!

------
exodos
Zabbix(zabbix-slack-alertscript) & StatusCake + Slack

