
A distributed and coordination-free log management system - avitzurel
https://github.com/oklog/oklog
======
sagichmal
Here's the blog post describing the system

[https://peter.bourgon.org/ok-log/](https://peter.bourgon.org/ok-log/)

And the original HN post

[https://news.ycombinator.com/item?id=13418125](https://news.ycombinator.com/item?id=13418125)

------
_dark_matter_
He describes Heka (built by Mozilla) as abandoned. This is true, but we have a
re-write in C that is significantly faster and in active development [0].

[0] [https://github.com/mozilla-
services/hindsight](https://github.com/mozilla-services/hindsight)

~~~
sagichmal
Isn't it irresponsible to be writing infrastructure software like this in C
nowadays?

~~~
aleksei
On HN: yes. In the real world: no.

~~~
floatboth
Regardless, it's weird coming from literally the company behind Rust…

------
packetized
While super-interesting (and something that I wish I had time to really
evaluate against Elasticsearch/Solr), I feel like it's missing one of the most
useful things about Loggly/ES/Solr - the ability to quickly visualize log
trends. And despite the statement that it's a "distributed and coördination-
free log management system for big ol' clusters", I'm confused why you would
want a "big ol' cluster" without visualization or field searches more complex
(and less computationally expensive) than regexes. Am I missing something?

e: To be fair, I'm not attempting at all to defend Elasticsearch/Solr, just
slightly confused about the actual use cases in prod.

~~~
sagichmal
What does "the ability to quickly visualize log trends" mean concretely?

~~~
packetized
In my mind, some sort of graphical user interface that would allow you to
query data and have the log system return results that are easily parsed. At
the very least, have some sort of API that allows you to do this. The /query
endpoint exists, but doesn't seem to return anything that's really
fundamentally useful from a large-scale perspective.

~~~
benley
I agree that the nice graphical tools available for Elasticsearch are super
useful, but it seems to me that those are not a feature of the logs storage or
query interface, but rather one of an external app (e.g. kibana) that queries
the system.

It would actually be somewhat concerning to me if a system like Oklog had all
those graphing and visualization features built-in; that would be quite some
scope creep for a distributed storage engine :-P

~~~
NikolaeVarius
I disagree. Elasticsearch itself is doing most of the heavy lifting. Kibana is
developed in sync with Elasticsearch so that it always has the ability to use
most of the features of Elasticsearch.

The nice visualization methods are a direct result of the aggregation
abilities of Elasticsearch

~~~
grey-area
Ways to query the data are important and on this newly launched pkg probably
need some work (haven't looked at that side of it), but visualisation does not
belong in a logging tool, it belongs in external tools querying that tool.

------
thisisananth
The thought process that went into the design is very enlightening. Thanks for
writing that up.

------
kordless
This looks pretty slick. There's a well done explanation of the architecture
process in the blog.

------
solidsnack9000
There is actual a fair amount of coordination in this logging system -- with
the combination of redirects and gossip to handle them -- but it does seem to
be the right kind of coordination.

------
treve
'Coordination' in English ;)

~~~
jihadjihad
yeah, I guess the oklog folks have drunk the kool-aid put out by the New
Yorker: [http://www.newyorker.com/culture/culture-desk/the-curse-
of-t...](http://www.newyorker.com/culture/culture-desk/the-curse-of-the-
diaeresis)

~~~
rspeer
Diaeresis: a typographical accent used to remind you that you are reading the
New Yorker.

