
Grepping logs is still terrible - madhouse
http://asylum.madhouse-project.org/blog/2015/05/07/grepping-logs-is-still-terrible/
======
ghshephard
It's beyond me how he doesn't understand that text logs are a universal
format, easily accessible, that can be instantly turned into whatever binary
format you desire with a highly efficient insertion process (Splunk is just
one of those that does a great job).

Here is the thing he doesn't seem to understand - all of us who are sysadmins
absolutely understand the value of placing complex and large log files into
database so that we can query them efficiently. We also understand why having
multi-terabyte text log files is not useful.

But what we find totally unacceptable is log files being shoved into binary
repositories as the primary storage location. Because you know what _everyone_
has their own idea of what that primary storage location should be, and they
are mostly incompatible with each other.

The nice thing about text - for the last 40 years it's been universally
readable, and will be for the next 40 years. Many of these binary repositories
will be unreadable within a short period, and will be immediately unreadable
to those people who don't know the magic tool to open them.

~~~
scrollaway
> text logs are a universal format

Uh, I don't know what world you live in but I'd like the address because mine
sucks in comparison.

Text logs are definitely not a "universal format". Easily accessible, sure.
Human readable most of the time? Okay. Universal? Ten times nope.

Give you an example: uwsgi logs don't even have timestamps, and contain
whatever crap the program's stdout outputs, so you often end up with three
different types of your "universal format" in there. I'm not giving this
example because it's contrived, but because I was dealing with it the very
moment I read your comment.

~~~
ghshephard
But at least you have a fighting chance. What if that exact same data was
dumped into a binary file, that you did not know how to decode?

Originally, you had a problem - the data wasn't formatted in a manner that you
could parse cleanly.

Now, you have a new problem - not only is the data not formatted properly,
it's now in some opaque binary file.

Saying that there are poorly formatted text files isn't a hit against text
files, it's a hit against poor formatting. The _exact same problem exists_ if
the file is in binary form, and not formatted properly.

~~~
rubenv
Sure, an opaque binary file is pointless.

But that's not what most logging systems that log to binary files offer. They
give you specs (example:
[http://www.freedesktop.org/wiki/Software/systemd/journal-
fil...](http://www.freedesktop.org/wiki/Software/systemd/journal-files/)) and
tools.

Binary doesn't have to mean closed/opaque.

~~~
kuschku
And the specs will be gone in 40 years. While ASCII will stick around.

~~~
jkyle

        And the specs will be gone in 40 years. While ASCII will stick around.
    

Why would they be gone? You realize ASCII is a 'spec' too?

If a binary format has an open specification, it's as future proof as ASCII.
ASCII's durability is due to a clear and open specification that's easily
implemented. Not some magic sauce that makes it instantly human readable.

That text you see? It's not what's actually in the file. That's just 1's and
0's like every other format. There's literally no difference between ASCII and
any other "binary" format.

~~~
TazeTSchnitzel
Text encodings have come and gone before, too. We don't use the Baudot code on
modern computers still, and EBCDIC is confined to IBM mainframes.

------
thaumaturgy
Cool, so which standard binary log storage format should we all switch to?

Should I submit patches to jawstats so that it'll support google-log-format
1.0 beta, or the newer Amazon Cloud Storage 5 format? Or both? Or just go with
the older Microsoft Log Storage Format? Or wait until Gruber releases Fireball
Format? Has he decided yet whether to store dates as little-endian Unix 64 bit
int timestamps, or is he still thinking about going with the Visual FoxPro
date format, y'know, where the first 4 bytes are a 32-bit little-endian
integer representation of the Julian date (so Oct. 15, 1582 = 2299161) and the
last 4 bytes are the little-endian integer time of day represented as
milliseconds since midnight? (True story, I had to figure that one out once.
Without documentation.)

Should I write a new plugin for Sublime Text to handle the binary log formats?
Or write something that will read the binary storage format and spit out text?
Or is that too inefficient? Or should I give up on reading logs in a text form
at all and write a GUI for it (maybe in Visual Basic)?

Do you know when I should expect suexec to start writing the same binary log
format as Apache, or should I give up waiting on that and just write a daemon
to read the suexec binary logs and translate them to the Apache binary logs?

Should I take the time to write a natural language parsing search engine for
my custom binary log format? Do you think that's worth the time investment? I
would really like to be able to search for common misspellings when users ask
about a missing email, you know, like "/[^\s]+@domain.com/" does now.

I look forward to your guidance. I've been eagerly awaiting the day that I can
have an urgent situation on my hands and I can dig through server logs with
all of the ease and convenience of the Windows system logs.

~~~
geographomics
The system should provide a standard API for writing and reading logs. The
precise format of the underlying log files is thus rather unimportant at this
level of abstraction. Other than the logging subsystem and recovery tools,
there's no need for any software to be accessing such log files directly
(outside of the API functions). This is how Windows has done it for years.

~~~
felixgallo
You missed the joke at the end where he correctly pointed out that Windows'
logging is a total joke, and that discovering information from Windows logs is
essentially impossible unless the tool writer specifically predicted your use
case.

And that's the nub of it: text logs are for when you may have many varied,
complex reader use-cases, and you don't understand all those cases well enough
yet to lock them down forever, and you have a thousand excellent tools at your
disposal that you would like to be able to continue to use.

Recent log spelunking for me included 'cat _log.? | grep fail | sed
's/^._worker_id$//g' | awk '{ print $5, $4 }' | sort -n -r | sed 30q'.

There's no analogue in any binary logging system I've ever found.

~~~
dragonwriter
It seems to me that a simple transitional tool for a binary logging system
would be for the implementer of the binary logging system to also include a
tool that consumed a binary log file on stdin and produced a stream on stdout
in one (or more, selecting which by command line arguments) common text log
formats.

That lets you develop an ecosystem of supporting tools that take advantage of
any strengths of the binary format, while still allowing the freedom of using
the (initially, at least, probably far more capable) set of tools available
for the text formats.

~~~
felixgallo
what is the point of such a 'transition' if there never arrives any point at
which there is net added value to a binary format?

~~~
dragonwriter
If there is some (not initially necessarily _net_ for _all_ users -- benefit
being, after all, something that varies from user to users, but significant
for some subset of users) benefit, the point is to mitigate the cost of moving
out of a native text format, and increase the number of users for whom there
is initially a net benefit, which also increases the initial use of the binary
format and the effort likely devoted to building auxiliary tools which
leverage it to some advantage, increasing the speed at which the net benefit
of the format for a wider range of users is increased.

This may or may not ever make it a net benefit for _every_ user, but that's
okay. There's a whole lot of space between "this technology is the best choice
for everyone" and "this technology is the best choice for no one".

------
pjc50
_Binary logs may be fine for you, but don 't force it on us!_

This is really the important point here. For small systems, grep works fine.
The number of people administering small systems is much greater than the
number of people administering large systems. The systemd controversy has
caused people to fear that change they don't want will be imposed on them and
their objections insultingly dismissed: a consequence of incredibly bad
_social_ "change management" by its proponents.

They are therefore deploying pre-emptive rhetorical covering fire against the
day when greppable logs will be removed from the popular Linux distributions.
Plain text is the lingua franca; binary formats bind you to their tools with a
particular set of design choices, bugs and disadvantages. My adhoc log
grepping workflow has a different set of bugs and disadvantages, but they're
_mine_.

~~~
mdekkers
"The number of people administering small systems is much greater than the
number of people administering large systems"

Do you have any evidence for this statement? Because it sounds all kinds of
wrong.

~~~
4ydx
[http://www.internetlivestats.com/total-number-of-
websites/](http://www.internetlivestats.com/total-number-of-websites/)

If I read it correctly there are about 250 million active sites (roughly). It
seems unlikely that they are all massive corporate sites.

As an aside, the idea that systemd is a good thing is hilarious to me at the
least because it is so brash about making an important change to a huge chunk
of the system. Yes the bugs will eventually get ironed out, but in the
meantime? Count me out! I have work to do and am not interested in being a
free tester for Redhat on my live systems.

~~~
nl
I'm pretty sure that counts (eg) each wordpress.com subdomain as a separate
website. [1] counts like that and gives a roughly comparable number.

That gives a lot of economy scales.

[1] [http://news.netcraft.com/archives/category/web-server-
survey...](http://news.netcraft.com/archives/category/web-server-survey/)

~~~
4ydx
The link I included states that those are unique hostnames. Perhaps they are
including subdomains on the same ip address, but you might note that rather
than quoting the 1 billion sites, I reduced that by their estimated 25% being
actually active. Additionally they state that there are on average 3 users per
site in 2014. Maybe that doesn't mean anything, but as a rough estimate that
all implies far more small sites than large ones.

------
rlpb
Take this philosophy to an extreme and you end up with a dedicated data format
and tooling/APIs to access the data for every subsystem, not just logging.
Essentially, this is Windows.

The downside to this is that now you don't have a set of global tools which
can easily operate across these separate datasets without writing code against
an API. I hear PowerShell tackles this; I don't know how well. The general
principle though harms velocity at just getting something simple done, to the
benefit of being able to do extremely complex things more easily. See Event
Viewer for a good example of this.

Logs don't exist in isolation. I want to use generally global tooling to
access and manipulate everything. I don't want to have to write (non-shell)
code, recall a logging-specific API or to have to take the extra step of
converting my logs back to the text domain in order to manipulate data from
them against text files I have exported from elsewhere for a one-off job. An
example might be if I have a bunch of mbox files and need to process them
against log files that have message IDs in them. I could have an API to read
the emails, and an API to read the logs, or I could just use textutils because
I know an exact, validating regexp is not necessary and log format injection
would have no consequence in this particular task.

I do see the benefits of having logs be better structured data, but I also see
downsides of taking plain text logs away. Claiming that there are _no_
downsides, and therefore no trade-off to be made, is futile. It's like playing
whack-a-mole, because nobody is capable of covering every single use case.

~~~
pjmlp
> Essentially, this is Windows.

Actually any non-UNIX OS clone out there, including mainframe and embedded
OSes.

------
mugsie
Honestly - I agree about the ELK stack side - piping all your logs into ES /
Logstash is a great idea. (Or Splunk / Greylog / Logentries)

If you run any sort of distributed system, this is vital. And while that
counts as binary logs, I would argue that on the local boxes it should stay
text.

I would agree, if you are running any sort of complex queries on your data -
go to logstash, and do it there - it much nicer than regexes.

If on the other hand, you just want to see how a development environment is
getting on, or to troubleshoot a known bad component tail'ing to | grep (or
just tail'ing depending on the verbosity of your logs) is fine.

I don't have to remember some weird incantation to see the local logs, worry
about corruption etc.

One problem I will point out with the setup described is syslon-ng can be
blocking. If the user is disconnected from the central logstash, and their
local one dies, as soon as the FIFO queue in syslog-ng fills, good luck
writing to /dev/log , which means things like 'sudo' and 'login' have ....
issues.

Instead, if you have text files being written out, and something like beaver
collecting them and sending them to logstash, you have the best of both
worlds.

------
Spooky23
Windows has had binary logging forever. Is windows administration some
wonderland of awesome capability for getting intelligence out of logs? Hell
no.

For administering Unix like systems, the ability to use a variety of tools to
process streams of text is an advantage and valuable capability.

That said, your needs do change when you're talking about managing 10 vs
10,000 vs 100,000 hosts. I think what you're really seeing here is a movement
to "industrialize" the operations of these systems and push capabilities from
paid management tools into the OS.

~~~
dfox
I think that largest problem with Event Log is overreliance on structure.
Often you have one particular log record that you know is the problem, but no
idea what it means because you have some generic event code and bunch of
meaningless structured data.

Freeform text logs usually contain more detail as to what exactly happened.

~~~
geographomics
That's not a limitation of the Event Log system itself, as you can easily
write freeform text in the message rather than building it up with localised
strings and structured data, e.g. [https://msdn.microsoft.com/en-
us/library/6w20x90k(v=vs.110)....](https://msdn.microsoft.com/en-
us/library/6w20x90k\(v=vs.110\).aspx)

------
indymike
Grepping logs is terrible. Reverse engineering a binary format so you can
diagnose why you are down/crashing/losing data is far worse. Logs should be
handled as text until they reach their long term storage... then whatever
helps analyze and query is fine...

------
phn
Yeah,in the presence of adequate tooling you don't need to grep logs. But how
much more effort is required to use those tool-friendly loggings? Where is
your god when the tool fails?

For me the main reason to access plaintext logs is they seldom fail, and they
are simple. They are a bore to analyse, they CAN be analysed.

Anyway, this discussion only makes sense if the task at hand involves _heavy_
log analysis, don't complicate what is simple when it isn't needed.

As for the razor analogy, you're right, however I wouldn't change my beard to
be "razor compatible only". In the software world I'd say it is still not
uncommon to find yourself "stranded in a desert island".

------
laumars
Oh jeez. Yes there are better and more performant tools for parsing optimised
binary databases; nobody disputes that. And yes, tools like Splunk are more
user friendly than grep; nobody disputes that either. But to advocate a binary
only system for logs is short sighted because logs are the goto when
everything else fails and thus need to be readable when every other tool dies.
There's quite a few scenarios that could cause this too:

    
    
      * log file corruption - text parsing would still work,
    
      * tooling gets deleted - there's a million ways you
        can still render plain text even when you've lost
        half your POSIX/GNU userland,
    
      * network connection problems, breaking push to a
        centralised database - local text copies would still
        be readable.
      

In his previous blog post he commented that there's no point running both a
local text version and a binary version, but since the entirety of his rant is
really about tooling rather than log file format, I'm yet to see a convincing
argument against running the two paradigms in parallel.

~~~
nailer
If 'tooling gets deleted' is a problem you probably have much bigger concerns
than log files.

~~~
gambiter
'tooling gets deleted' could easily happen after changing logging systems...
while it would be shortsighted to uninstall your old logging system entirely
(if you have logs laying around in that format) it's not unheard of.

The more likely situation would be that the logs are stored on a shared
storage server, and the machine you are using to look at the logs doesn't have
the logging system installed.

~~~
nailer
> The more likely situation would be that the logs are stored on a shared
> storage server, and the machine you are using to look at the logs doesn't
> have the logging system installed.

So expose the shared storage to a system running any current mainstream Linux
distribution. I understand what you're saying, but this still doesn't seems
like a huge concern.

~~~
gambiter
... We were talking about logging systems with proprietary tools for
manipulating logs. Ergo, 'any current mainstream Linux distribution' wouldn't
have them installed by default.

------
arpa
This is a discussion for a sake of discussion. The way I see it is that author
has a niche situation on his hands and therefore should use a product designed
for that particular niche, instead of complaining how everyone's wrong and
trying to shove his perspective down peoples' throats.

------
4ydx
Sounds like somebody in the systemd camp. I really dislike added complexity
when it is totally unnecessary. If people want to transform their logs into a
different storage format, that is up to them. Text files, however, are a
fantastically simple way of storing... (drumroll please) text. Surprising /s

------
robinhouston
> For example: find all logs between 2013-12-24 and 2015-04-11, valid dates
> only.

That’s a straw man. If you’re grepping logs, you don’t need a regular
expression that matches only valid dates because you can assume that the
timestamps on the log records are valid dates. But I suppose

    
    
        2013-12-(2[4-9]|3.)|2014-..-..|2015-0([123]-..|4-(0.|1[01]))
    

doesn’t look so bad.

The whole thing is similarly exaggerated.

~~~
ghshephard
Not to mention 99.9% of the searches one does of a log file isn't really that
complex. Heck, I'm willing to wager that 90% + of my searches over the last 20
years have been in log files from a particular _day_.

That's the thing about having simple text log files - the cognitive load
required to pull data out of them, often into a format that can then be
manipulated by another tool (awk, being one of the more well known), is so low
that you can perform them without a context switch.

If you have a problem, you can reach into the log files, pull out the data you
need, possibly massage/sum/count particular records with awk, all without
missing a beat.

This is particularly important for sysadmins who may be managing dozens of
different applications and subsystems. Text files pull _all_ of them together.

But, and here is the most important thing that people need to realize - for
scenarios in which complex searching is required, _by all means_ move it into
a binary format - that just makes sense if you really need to do so.

The argument isn't _all text_ instead of binary, it is _at least text_ and
_then_ use binary where it makes sense.

------
erikb
After reading the article I wonder if there are lots of tools that do all the
binary advantages in indexes but leave the logs as text files, why that is not
fine. To get the binary advantage the log does not have to be binary.

The example with the timestamps is also strange. No matter how you store the
timestamps, parsing a humanly reasonable query like "give me 10 hours starting
from last Friday 2am" to an actual filter is a complex problem. The problem is
complex no matter how you store your timestamp. You can choose to do the
complexity before and create complex index structures. You can choose to have
complex algorithms to parse simple timestamps in binary or text form, you can
build complex regexes. But something needs to be complex, because the problem
space is. Just being binary doesn't help you.

And that's really the point here, isn't it? Just being binary in itself is not
an advantage. It doesn't even mean by itself that it will save disk space. But
text in itself is an advantage, always, because text can be read by humans
without help (and in some instances without any training or IT education),
binary not.

Yesterday I was thinking there might be something about binary logs. Now I'm
convinced there isn't. The only disadvantage seems to be that you also lose
disk space if you store it in clear text. But disk space isn't an issue in
most situations (and in many situations where it is an issue you might have
resources and tools at hand to handle that as well) It is added complexity for
no real advantage. Thanks for clearing that up.

~~~
geographomics
Another advantage of using structured data rather than free-form text is that
you can more precisely encode the essence of the event, with fields for
timestamp, event source, type of event, its severity, any important
parameters, and so on. This permits logging to be independent of the language
of the system operator. Rather than grepping for what is almost always English
text, one can query a language-independent set of fields, and then, if a
suitable translation has been done, see the event in one's native language.

When applied widely throughout a system, this leads to the
internationalisation of log messages. Thus lessening the anglocentric bias in
systems software. Windows has done this for years, at least with its own
system logging (other applications can still put free-form text into the event
logs if they wish.)

~~~
erikb
About your first point: Independence. You are less independent of English and
more dependent on the binary format and the tools who can handle it. It's a
trade-off. And it might be just an opinion, but for me it's not a good trade-
off. Learning English was a one-time endeavour for me. But binary formats and
tools have to be learned separately.

About what you put in the log message: You can also put different fields in a
line of text. Not getting the advantage or trade-off here.

About the internationalisations: As non-English developers we force all our
systems who have logging internationalisation to English system language so we
have a common ground for the messages. Understanding the English message is
nearly no burden. Log Messages are Event triggers, either in code or in a
developer's/admin's mind. If I get a log message in my native language I don't
know which event that triggers, which makes it actually harder.

Really. I don't know any non-English person who considers log
internationalisation a good thing. Fighting anglocentricism is a very
anglocentric topic. Outside of UK/US that's a non topic. We (non-English
people) are happy that there is a language we can use to talk to each other
and we don't really care how it came to be that widely known.

And even if you don't speak English, I don't see the advantage of parsing \x03
instead of "Error:.*". Both are strings that have a meaning which is rather
independent of its encoding.

~~~
geographomics
This is also just anecdotal, but I used to work with a Chinese sysadmin (I was
in the UK, he was in China) who found it much more preferable to work on the
localised Windows servers we had installed over there, as all the UI and
messages were in his native tongue. I'm sure it was easier for him to gain
expertise in the tools he needed for his work, than to become proficient
enough in English to understand every obscure log or error message that the
system might throw at him.

------
indymike
Grepping logs is terrible. Reverse engineering a binary format so you can
diagnose why you are is worse. Logs should be handled as text until they reach
their long term storage... then whatever helps analyze and query is fine...

------
4ydx
My main problem with this is that ascii is not something that will ever change
over time. The data format is wonderfully static. Forever. Introduce a binary
format? You get versioning. It is a major downside.

------
Frondo
What you lose when you move away from text logs is not any real benefit; what
you lose is the illusion of control you have with text logs.

Text logs can be corrupted, text logs can be made unusable, you need a ton of
domain-specific knowledge to even begin to make sense of text logs, etc.

But there's always a sense that, if you had the time, you could still
personally extract meaning from them. With binary logs, you couldn't
personally sit there and read them out line by line.

The issue is psychology, not pragmatism, and that's why text logs have been so
sticky for so long.

~~~
4ydx
A substring of text may or may not be a date and based on the excellent tools
available in linux you can decide how to extract that "data point". If binary
logging is little more than a stream of text, then that is fine, but I
seriously doubt that is the push happening. Personally I prefer having a raw
stream of data that I have to work with as best as I can rather than having to
use some flag defined by somebody else to range across dates. That is the
fundamental difference it seems: do you want a collection of tools that can be
applied in a variety of ways or do you want the "one way" (with potential
versioning... have fun!).

Again if the binary log is simply better compressed data, well we have ways of
compressing text already as an afterthought. This really, fundamentally, seems
to be a conflict in how people want to administer their systems and, for the
most part, this seems to be about creating a "tool" that people then have to
pay money for to better understand.

------
jack9
> Does database store the data in text files? No? That's my point.

This guy is a first class idiot who knows enough to reformulate a decided
issue into yet another troll article. "a database (which then goes and stores
the data in a binary format)". How about a text file IS a database. It's
encoded 1s and 0s in a universal format instead of the binary DB format which
can be corrupted with the slightest modification or hardware failure.

------
KaiserPro
I think there are a number of issues that are getting mushed into one.

* Journal is just terrible.

* some text logs are perfectly fine.

* when you are in rescue mode, you want text logs

* some people use text logs as a way to compile metrics

I think the most annoying thing for me about journald is that it forces you to
do something their way. However its optional, and in centos7 its turned off,
or its beaten into such a way that I haven't noticed its there.... (if that is
the case, I've not really bothered to look, I poked about to see if logs still
live in /var/log/ they did, and that was the end of it. Yes, I know that if
this is the case, I've just undermined my case. Shhhhh.)

/var/log/messages for kernel oopes, auth for login, and all the traditional
systemy type things are good for text logs. Mainly because 99.9% of the time
you get less than 10 lines a minute.

being able to sed, grep, tee and pipe text files are brilliant on a slow
connection with limited time/mental capacity. ie. a rescue situation. I'm sure
there will be a multitude of stable tools that'll popup to deal with a
standardised binary log format, in about ten years.

The last point is the big kicker here. This is where, quite correctly its time
to question the use of grep. Regex is terrible. Its a force/problem amplfier.
If you get it correct, well done. Wrong? you might not even know.

Unless you don't have a choice, you need to make sure that your app kicks out
metrics directly. Or as close to directly as possible. Failing that you need
to use something like elastic search. However because you're getting the
metrics as an afterthought, you have to do much more work to make sure that
they are correct. (although forcing metrics into an app is often non trivial)

If you're starting from scratch, writing custom software, and think that log
diving is a great way to collect metrics, you've failed.

if you are using off the shelf parts, its worth Spending the time and
interrogating the API to gather stats directly. you never know, collectd might
have already done the hard work for you.

The basic argument he puts forth is this: text logs are a terrible way to
interchange and store metrics. And yes, he is correct.

~~~
ownagefool
Not that I think it matters all that much but journald will be running on your
centos box, but it'll be configured to spew text into /var/log.

Just type journalctl and you should see the data there.

~~~
KaiserPro
I suspected as much, so long as I don't have to use its tools I couldn't care
less. Unless it eats resources.

~~~
kasabali
> Unless it eats resources

I have no idea on its effect in practice, but theoretically it should have
negative effect because it soaks in all the logs, and _then_ it forwards them
to the logging daemon, even when journal storage is turned off.

------
sika_grr
Of course you need to log some data in textual format for emergencies, but if
you had a tool that indexes events on timestamps, servers, monitorees,
severity and event type, while severely reducing the storage required, you
would be able to log much more data, and find problems faster. Arguing binary
vs text logs is like arguing serial port vs USB on some industrial systems.

------
arenaninja
Great to see some effort in this area. I've been using New Relic and it's
pretty great for errors because we've setup Slack/email notifications.
However, there's nothing for general log (e.g.: access log) parsing. I'm
installing an ELK stack on my machine right now and hope that it's enough

------
amelius
Doesn't this just mean that we should have a more "intelligent" version of
grep? For example, this "supergrep" could periodically index the files it is
used on, so searching becomes faster.

------
erikb
*edit: I'm wrong, this was not the link posted yesterday. [https://news.ycombinator.com/item?id=9496850](https://news.ycombinator.com/item?id=9496850)

------
hartator
Isn't everything will be solved by a some kind of grep that's date/timespan
aware?

------
lurkinggrue
But then how will I watch the log files go by in real time?

------
deathanatos
It seems to me that most of the worry about a binary log file being "opaque"
could be solved with a single utility:

    
    
        log-cat <binary-log-file>
    

… that just outputs it in text. Then you can attack the problem with whatever
text-based tools you want.

But to me, having a utility that I could do things like, get a range of log
lines — in sorted order —, or, grep on just the message, would be amazing.
These are all things that proponents of grep I'm sure will say "you can!" do
with grep… but you can't.

The dates example was a good one. I'd much rather:

    
    
        log-cat <bin-log> --from 2014-12-14 --to 2015-01-27
    

Also, my log files are not "sorted". They are, but they're sorted _per-
process_, and I might have multiple instances of some daemon running (perhaps
on this VM, perhaps across many VMs), and it's really useful to see their logs
merged together[2]. For this, you need to understand the notion of where a
record starts and ends, because you need to re-order whole records. (And log
records' messages are _going_ to contain newlines. I'm not logging a backtrace
on one line.) grep doesn't sort. |sort doesn't know enough about a text log to
adequately sort, but

    
    
        $ log-cat logs/*.log --from 2014-12-14 --to 2015-01-27
        <sorted output!>
    

Binary files offer the opportunity for structured data. It's really annoying
to try to find all 5xx's in a log, and your grep matches the process ID, the
line number, the time of day…

I've seen some well-meaning attempts at trying to do JSON logs, s.t. each line
is a JSON object[1]. (I've also seen it attempted were all that is available
is a rudimentary format string, and the first " breaks everything.)

Lastly, log files sometimes go into metrics (I don't really think this is a
good idea, personally, but we need better libraries here too…). Is your log
format even parseable? I've yet to run across one that had an unambiguous
grammar: a newline in the middle of a log message, with the right text on the
second line, can easily get picked up as a date, and suddenly, it's a new
record. Every log file "parser" I've seen was a heuristic matcher, and I've
seem most all of them make mistakes. With the simple "log-cat" above, you can
instantly turn a binary log into a text one. The reverse — if possible — is
likely to be a "best-effort" transformation.

[1]: the log writer is forbidden to output a newline inside the object. This
doesn't diminish what you can output in JSON, and allows newline to be the
record separator.

[2]: I get requests from mobile developers tell me that the server isn't
acting correctly all the time. In order to debug the situation, I first need
to _find_ their request in the log. I don't know what process on what VM
handled their request, but I often have a _very_ narrow time-range that it
occurred in.

~~~
scrollaway
What you're describing is journalctl.

~~~
deathanatos
I've not yet had the opportunity to try systemd. :-) Someday.

------
imaginenore
Logstash, Kibana, Splunk

------
geographomics
Windows systems have had better log querying tools than grep for years now,
with a well structured log file format to match. It's good to see Linux
distributions finally catching up in this regard.

Not that the log files on Linux are all entirely text-based anyway. The wtmpx
and btmpx files are of a binary format, with specialised tools for querying. I
don't see anyone complaining about these and insisting that they be converted
to a text-only format.

