
Guide to Logging - makaimc
https://www.loggly.com/ultimate-guide/
======
jedberg
I have a long rant about logging, but the TL;DR is: aggregated logging in
general is useless don't bother, everything you need to know should be in your
monitoring system. Basically, if something is broken and it's a problem, it
will happen again, so you just start logging when something breaks. How do you
know something breaks? Log your application metrics. Log your customer
experience in the form of metrics and monitoring.

But don't aggregate every code exception and warning. If they aren't affecting
your customer experience, who cares? And if they are affecting it, then it
should be expressed as an application metric in your monitoring system.

Edit: To make my point a little clearer, I believe that central logging and
aggregating should be available, just not turned on all the time.

~~~
jacques_chester
Logging is for _events with identity_ , metrics are for events _without
identity_. They are not the same thing.

Sometimes I care about the bulk behaviour of my application. I hit 2757 RPS
and suddenly find a catastrophic gap in the performance landscape of my
application -- that's useful. I don't care much about the particulars of the
requests. What matters is the _statistical behaviour_.

But sometimes the application crashes when a _particular input_ is given.
These are outliers, they don't show up clearly in _statistical behaviour_. In
that case I care about the details.

Metrics are warnings that smokers die younger, on average, than non-smokers.

Logs are a biopsy showing which kind of lung cancer you have.

No sensible production system or PaaS treats them as identical, even if you
can derive metrics from logs.

~~~
jedberg
> But sometimes the application crashes when a particular input is given.
> These are outliers, they don't show up clearly in statistical behaviour. In
> that case I care about the details.

But my question is, why do you care about the outliers if the customer
experience is not harmed by them? And if it is harmed, then it should should
up in your 99th percentile metrics, even as an outlier, at which point you can
turn on aggregated logging to look for that one thing. And if that one thing
doesn't happen again, then did it really matter?

~~~
mostlyjason
Imagine customer support where you really need a record of the individual
transactions for that particular customer. A monitoring solution won't give
you that level of detail.

~~~
jedberg
Individual transactions aren't found in application logs, they are found in
the data store. Also, again, it's important to stress that central logging
should be available, just not on all the time. If it's a specific sequence of
events that triggers a problem, turn on logging and then ask them to do the
sequence again. If they don't need to do the sequence again, then it wasn't
really of consequence. And if it happens for multiple users, then customer
service should be able to turn on let the developers know to try and find the
cause which they can do by adding more metrics or perhaps some temporary
central logging.

~~~
brazzledazzle
A better compromise would be always having logging on but only retaining very
recent logs. You need the infrastructure to support logging and correlation
anyway and if you have a short enough retention period you don't have to worry
about dealing with growing infrastructure just to support it.

~~~
mostlyjason
Yeah I think most people store them for a limited time, such as a week or a
month. Most issues that pop up will only be relevant that long. You can have
permanent archives for individual data since storage is cheap, or even summary
metrics. Better to start with ground truth because you often don't know what
data you'll need in hindsight.

------
mostlyjason
Hey guys I'm one of the technical editors for the guide. We created it because
logging is really valuable for troubleshooting and monitoring systems.
However, it's a complex topic and so many resources today are spread around on
different forums. We wanted to have a place to consolidate all that useful
information into something much easier to read. If it helps a dev or ops
person solve a problem faster or easier than we've done our job!

There is still a lot of content to be written for languages like python, ruby,
and more. We'd love to have more contributors, or even just suggestions for
the site. I can also get in touch with any of the authors if you have
questions for them.

~~~
kordless
> many resources today are spread around on different forums

Centralizing the all the knowledge that's been written about a given topic
isn't scalable in terms of contributions. It's far better to focus on enabling
_contributions_ than it is to focus on building a solution in which to put the
contributions. There's a reason products like StackExchange are more
successful at providing features for encouraging contributions than topic
specific sites.

> There is still a lot of content to be written for languages like python,
> ruby, and more.. We'd love to have more contributors,

There is already a ton of contributions to logging solutions on the Internet.
Why not build a solution categorizing those, instead of trying to build new
content in a centralized (and I might add commercially curated) solution?

Also, where's the source to this "open source" project? I'm not seeing any
links to code.

~~~
mostlyjason
I think readability is a concern. Often when I'm learning a new technology I'd
rather sit down with a good book on the topic rather than reading a bunch of
forum posts. I can be pretty confident that it will cover the most important
concepts, building up from basics to more advanced ones, and showing all the
connections in between. A lot of times reading regular posts, you have to
build all these connections yourself. We do link to really good resources and
documentation where they are available though so if you have any to suggest
please let me know!

I think the open source part of the title refers to the fact that we cover all
different technologies from a variety of perspectives, not just a single
vendor solution. Also, we allow contributions from anyone in the community. I
couldn't find an easy way for WordPress to share or collaborate on source
code, but if you think of a way, let me know. In the meantime, we're giving
access to people who want to contribute.

~~~
kordless
I'd suggest building the content off a Github repo, rendering from markdown
files. This solves the problem of contribution enablement. Rackspace has a
docs/developer project that I was involved with for a very short time which
has its source code available for use. Start there and get your content moved
off Wordpress. Nothing against Wordpress, but it's not built for this. Also,
code snippets.

Rackspace site on Github:
[https://github.com/rackerlabs/developer.rackspace.com](https://github.com/rackerlabs/developer.rackspace.com)

I agree with the readability part. It's important to curate from the stance of
educating the multitudes quickly. Having disparate content can be an issue in
starting the education, but variety is the spice of life - and use cases as
time goes on. I'd recommend content + reference as a solution here. Cookbook
style, if you will.

Keep in mind I've spent a fair amount of clock cycles thinking about this than
most people. I've rejected things like logging standards as well. That doesn't
mean I'm right, but it is, at the least, a data point for you.

------
mugsie
[https://www.loggly.com/ultimate-
guide/?s=parse+WSGI+logs&pos...](https://www.loggly.com/ultimate-
guide/?s=parse+WSGI+logs&post_type=post)

0 results

[https://www.loggly.com/ultimate-
guide/?s=parse+python+logs&p...](https://www.loggly.com/ultimate-
guide/?s=parse+python+logs&post_type=post)

Gives you a page on NodeJS

Ultimate ... not so much.

~~~
jt2190
I've downvoted you because the world doesn't need yet another post about how
some open source project that has the audacity to call themselves "the
ultimate resource" for their initial release doesn't live up to some-random-
person-on-the-internet's idea of "ultimate".

I'm guessing from the specifics of your example queries that you already have
some knowledge about WSGI and Python that you could contribute to the guide.
Why not do that instead?

~~~
mugsie
And I commented, as it calls itself ultimate, and it has articles about Java,
Node, Apache, Linux (including SystemD) and Windows.

If it was titled "We want to build the Ultimate Logging Resource" I would be
OK with it.

I would have no problem contributing, and may well do so over the next while,
but titles like this get my goat

Edit: Also, while I know this will be downvoted to hell - Open Source? It is
by a commercial company, with -all- most of the examples being their product.

~~~
mostlyjason
I think that's a fair point. It's really something we want and believe will
become ultimate over time as more people contribute to it. It's got a good
start but needs a lot more work. I think it's useful to show the vision of
what we want to become, because logging is a complex topic and so many
resources today are spread around on different forums. That's why we wanted to
have a place to consolidate all that useful information into something much
easier to read. I think it's probably worth a adjustment to the tagline and
mission statement on the front page to make this more obvious. Thanks for the
suggestion!

------
strickjb9
Your site's readability would be greatly improved if you darkened the text
color. #444 or darker is good enough for a white background.

------
packetized
'Become a Contributor' seems to mean 'please give us your email address to be
added to our mailing list.'

This seems moderately suspect.

------
DKnoll
Loggly is crazy expensive for what they have on offer (at least given my use
case which is aggregating syslog and Windows event logs). You're much better
setting up Logstash or Graylog2 on-premises or in public cloud.

------
avn2109
Was I the only one who was really hoping this was going to be about best
practices for cutting down trees?

------
mirimir
I find nothing on tor or lighttpd :(

------
acheron
My four-year-old watches this show, whence I've learned all about logging:
[http://www.amazon.com/Mighty-Machines-In-The-
Forest/dp/B0000...](http://www.amazon.com/Mighty-Machines-In-The-
Forest/dp/B00008YZ0E)

~~~
0xdeadbeefbabe
The subtext that I've just started to notice, as I watch with my four year
old, is that Canada is awesome.

~~~
DKnoll
I'm going to take this even further off-topic but if you enjoy cheesy vintage
Canadian children's television Hammy Hamster AKA Tales of the Riverbank is
required viewing.

