
Show HN: A program to email log files using Go - muzzammildotxyz
https://github.com/muhammadmuzzammil1998/Log-mailer
======
dewey
As this seems to be a beginner project to learn here's some thoughts on the
code:

1) You should really not build your json by manually building the string.
That's very error prone and hard to extend.

    
    
      fmt.Fprintf(f,
      "{\n\t\"from\": {\n\t\t\"name\": \"%s\",
    
    

Use
[https://golang.org/pkg/encoding/json/](https://golang.org/pkg/encoding/json/)
([https://gobyexample.com/json](https://gobyexample.com/json)) instead.
Working with json in Go is really one of the nicer features of Go (It's Go,
not "GoLang")

2) By setting / typoing a config option you could accidentally delete files
from your computer, even your home directory

    
    
      err := os.Remove(jsonfile)

~~~
eptcyka
It's GoLang, because that's easier to search for.

~~~
dewey
I was just nitpicking, it's not really an important point.

People usually prefer "Go" or "Golang", I only mentioned it because I noticed
it in the project description.

------
badrabbit
A few concerns I have...

1) The logs are not sent encrypted, this exposes them for every smtp server
and mitm party between unencrypted hops

2) you have to save a plain text version of your log mailer's smtp password to
disk

3) Monitoring by email sucks and future compromise of any recipients' inbox
exposes all historical logs

I have written something similar in the past as well and seen email monitoring
in real world scenarios. You're not doing it "wrong" per se ,but I think
modern protocols allow better solutions.

For instance,you can POST the logs to a server(ec2 instance or DO droplet)
over TLS, have it generate a link and email that link. You can then control
link expiry and encrypt the logs so that they are decrypted in-browser via
Webcrypto. You can also do some sort of push monitoring(in addition to sending
the link via email) by the server which lets you avoid the whole "email
messages can be stuck on an smtp hop for 24-48 hours" and having to monitor
for NDR's before resending the logs.

~~~
muzzammildotxyz
Yes, these are valid concerns. 1) I don't know how to encrypt logs and send
them. If you can help please open a PR on GitHub or walk me through it :)

2) How else should I store it? Should I encrypt the json file with a master
key?

3) This is correct but I just wanted a quick way to do it :) Thanks for the
concerns.

------
heegemcgee
Overall, i would say "using email for system tasks considered harmful". I have
worked several devops / sysadmin jobs where my inbox took weeks to tame with
filters because of rampant abuse of automated system emails.

Every alert should be actionable. And email doesn't have good reliability or
timeliness - it can take hours for me to get a push notification on my phone
that there is email, and in the evenings, i really shouldn't be looking at
email at all. So we should be using a proper alert system via SMS (pagerduty
is pretty great for this, but i also like twilio, and amazon SNS is just fine
too).

More germaine to the topic at hand: I'd recommend a) setting up log monitors
with Nagios, or Zabbix, or your favorite tool. You want to regex match on
certain strings in the log file, like "Deadlock" or "out of memory". Pass that
alert on to your monitoring system and get a proper, actionable alert.

And b), aggregating the logs. As far as convenient access, i'd recommend
Graylog (or ELK or Splunk) if you have more than a handful of nodes. This
makes it easy to search through logs or review them without signing into all
those nodes. You can also push them over to Amazon Cloudwatch Logs for
archival and rudimentary search.

~~~
AnIdiotOnTheNet
> Every alert should be actionable.

How do you know the difference between "everything is working properly" and
"the logging and/or monitoring has stopped working"?

~~~
heegemcgee
Who will watch the watchmen, right? :D It's a real concern. Personally, i have
a monitoring agent running, and then i have the config management agent
(puppet) validate that the monitoring agent is running.

And what Dewey said is absolutely right - you can monitor the code / service
itself through health checks. In the case of a reports service, perhaps your
monitor asks the API for a very small report. Or you could implement a special
endpoint / controller that calls on the core code. I recently implemented a
monitor that emulates a typical user session, logging in, performing popular
tasks, and logging out. If any step in that process has an error, i get an
alert with the step listed, and i instantly have some idea of where things are
jammed. In this manner, i don't need to have pre-defined log monitors for
specific errors; i can catch novel error types by virtue of exercising the
code and watching for the expected responses - 200 in the header, ability to
perform tasks that are only available on login, checking for certain strings
in the response, etc.

------
cejast
I thought this was what logrotate was for?

[https://linux.die.net/man/8/logrotate](https://linux.die.net/man/8/logrotate)

~~~
submeta
My thought also. - I was wondering why this post made it to the first page of
HN. Obviously many fellow HN users find it valuable / do not know that there
are solutions to this.

~~~
dvfjsdhgfv
I'm baffled by this, too. Maybe it's interesting for people who are interested
in learning Golang, but I think it's clear it's not the best piece of code to
learn from.

~~~
muzzammildotxyz
Gee, thanks for the boost in confidence. :|

~~~
dvfjsdhgfv
I think you're self-confident enough.

------
dsr_
It's very nice that you can specify the period between sending logs in
nanoseconds, microseconds or milliseconds, but also utterly useless. Even if
there's a time library being used that has these options, it's appropriate to
only advertise the units that make sense.

Sadly, it does not include "days", although one can always specify the value
in milliseconds. (The fifth hyperfactorial, as recently mentioned.)

~~~
muzzammildotxyz
Well, you caught me. I copied and pasted that line from Go Source comments :D

But hey, you can use 1h4ms6ns as intervals :D

Maybe I should parse days... Thanks for the Idea.

------
noponpop
Have you considered syslog, which represents a standard for shipping logs?

~~~
weavie
Syslog (at least on Solaris) seems to have a max length of 1024 characters.
Highly annoying and seemingly arbitrary limitation.

~~~
muzzammildotxyz
Yup

------
liotier
Any reason for not using Logwatch ?

[https://wiki.archlinux.org/index.php/Logwatch](https://wiki.archlinux.org/index.php/Logwatch)

~~~
actionscripted
Looks like the author is taking a stab at working with Go and wanted to build
something to fill a need.

I would absolutely recommend Logwatch or, if you're fancy, Filebeat and
Logstash.

~~~
muzzammildotxyz
actionscripted, you are a wizard! How did you guess that? :D

------
hestefisk
You can get FreeBSD to email any log periodically using cron. No go program
needed. This seems like an absolute overkill but it was probably fun writing
the code.

~~~
muzzammildotxyz
Yes, it was a great experience. Just made it to see if I could :)

