
AOL Moloch: open-source, large scale, packet-capturing, indexing database system - bottle2
https://molo.ch/
======
jlgaddis
The Filesystem Hierarchy Standard [0] has been around for ~25 years but it
still took quite a long time before most Linux distros decided to adhere to it
(for the most part; some still do "non-standard" things at times).

Now that we're to that point, please stop screwing it up and coming up wih
your own locations for application binaries, data, etc.

To be clear, the "/data" directory -- under which Moloch's pre-built packages
apparently install to, according to the README [1] -- is not part of the FHS.

\---

[0]:
[https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard](https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard)

[1]:
[https://raw.githubusercontent.com/aol/moloch/master/release/...](https://raw.githubusercontent.com/aol/moloch/master/release/README.txt)

~~~
zwaps
This post 'gonna cost me, but what you write is endogenous to the FHS making
little sense to most people who don't have a unix background.

"Where ARE my programs / settings that I installed, I always have to google
it!"

is interrelated with someone else asking

"Where should I PUT this program / setting / data? I guess I'll chose one of
those three or so locations where it may fit, or several"

For example: Why isn't there a data directory to be found? Where does the data
go? Why not replace nondescript and unhelpful names like "opt" and "usr"
(which is btw. not where the user data is!)?

~~~
geocar
/usr is absolutely where user data was. That's what it stands for "users".

First when / ran out of space, new programs were put on the other disk /usr in
/usr/bin so everyone could assume programs were in /bin except ones that were
newly installed so they would be in /usr/bin

When /usr ran out of space, a new disk was added as /home and things that were
easier to move (user directories) were moved first leaving things that
everyone's script was depending on (#!/usr/bin...) where they were.

This was done out of necessity, not out of good taste.

~~~
elros
/usr stands for Unix System Resources

~~~
geocar
More marketing I'm afraid. See:

[https://www.bell-labs.com/usr/dmr/www/notes.html](https://www.bell-
labs.com/usr/dmr/www/notes.html)

See also, from:

[https://www.tldp.org/LDP/Linux-Filesystem-
Hierarchy/html/usr...](https://www.tldp.org/LDP/Linux-Filesystem-
Hierarchy/html/usr.html)

 _In the original Unix implementations, /usr was where the home directories of
the users were placed (that is to say, /usr/someone was then the directory now
known as /home/someone)._

~~~
mysterydip
Why didn't it go away once /home became common?

~~~
gerdesj
It still had /usr/bin and /usr/sbin in it and scripts, muscle memory and
general inertia to contend with. Yes of course it should have gone away, perl
should have been at v6 and python at v3 years ago.

I'm off to saddle up my piggie squadron for a flypast.

------
yingw787
This is cool, thanks for sharing! I'll be honest, I didn't think AOL did
anything...interesting...but this is interesting to me! What are some open-
source alternatives / analogues to this product?

~~~
gyehuda
Sorta like [https://www.wireshark.org/](https://www.wireshark.org/). But
Moloch is a very active project, used by many, and used internally at Verizon
Media. Aol is part of Verizon Media (which brought AOL and Yahoo together).
Open source is very active here. ;-)

Compare also to [https://enterprise.verizon.com/products/security/advanced-
th...](https://enterprise.verizon.com/products/security/advanced-threat-
analytics-and-detection/network-detection-response/) (formerly known as
Protectwise), which is not open source.

~~~
solarkraft
Ignorant question for you: What does AOL do these days (I can't view the
website for the same reason I don't read TechCrunch articles)?

Is it fully about online publications now?

~~~
gyehuda
You can see [https://www.verizonmedia.com/our-
brands](https://www.verizonmedia.com/our-brands) to see the collection of
online brands in the family. You might use lots of these brands today without
really noticing. There's a lot of internet content you get via
[https://www.verizondigitalmedia.com/](https://www.verizondigitalmedia.com/)
which is also part of the same company. Aol is still a thing, people do use
it. Many people use lots of these brands as part of their internet experience.

If you are here, you probably care about internet security. Verizon Media runs
a pretty significant Bug Bounty program
[https://hackerone.com/verizonmedia](https://hackerone.com/verizonmedia) which
you can read about here [https://www.protocol.com/hackerone-bug-bounty-
virtual-verizo...](https://www.protocol.com/hackerone-bug-bounty-virtual-
verizon) and their blog here
[https://www.verizonmedia.com/technology/security#/bug-
bounty](https://www.verizonmedia.com/technology/security#/bug-bounty)

You might even be looking for a job as an information security professional.
You can join "The Paranoids" team (now that's a good name, don't you think!)
by checking out some of their jobs.
[https://www.verizonmedia.com/careers/search.html?q=paranoids](https://www.verizonmedia.com/careers/search.html?q=paranoids)

------
gyehuda
According to my sources, DARPA wasn't involved in any of the child sacrifices,
but they might use Moloch to help secure their networks.

~~~
dguido
Yep! As far as I'm aware, a number of military services (I think the USAF)
have been public about using Moloch on their networks. It was money well spent
by DARPA.

~~~
gyehuda
Moloch is free. :-)

~~~
cryptoquick
Well, Moloch is free now. DARPA had to make a lot of sacrifices to make that
happen, though.

~~~
gyehuda
They do an amazing job keeping it secret too. Can't even find them on the
contributor list
[https://github.com/aol/moloch/graphs/contributors](https://github.com/aol/moloch/graphs/contributors)

~~~
cryptoquick
Those are those who contributed code. Unfortunately, defense contract money
doesn't have a tab on GitHub. Trust me, I've used palantir/tslint

~~~
burpsnard
Just search for the project/deliverable/system/protocol on LinkedIn ;) ;) ;)

------
dguido
Fun fact, Moloch was initially created with funding from DARPA's famous Cyber
Fast Track program! It's great to see that Moloch is still going strong since
~2013.

There is some previous discussion of Moloch when it was released in this older
thread:
[https://news.ycombinator.com/item?id=20586005](https://news.ycombinator.com/item?id=20586005)

~~~
awick
Not sure if you are joking, but Moloch was never part of DARPA's short lived
Cyber Fast Track program. :) We do welcome contributions from everyone, and
lots of different folks use Moloch. If interested join us over at
[https://molo.ch](https://molo.ch)

~~~
dguido
That's odd. Are you one of the original authors? The CFT project list had
Moloch on it. I'll try and dig it up, it's probably floating around my Google
Drive. You may want to speak with Eoin Miller, as I believe he was the point
of contact for the project in the document I'm thinking of.

~~~
dguido
Fuckkkkk I think I found the source of my confusion. I am wrong, you are
right.

I DID find documents about Moloch floating around my Google Drive from
~2013-ish. I believe I invited your co-author Eion to present at a conference
I was running, THREADS, in 2014 and that he was not able to make it. The focus
the _year prior_ was exclusively on DARPA CFT. I combined those two events in
my head and thought your project got some seed funding from DARPA too. I'm
sorry!

Here is the conference:

THREADS 2014 when you were invited:
[https://github.com/trailofbits/threads/tree/master/2014](https://github.com/trailofbits/threads/tree/master/2014)

THREADS 2013 was a retrospective on DARPA CFT:
[https://github.com/trailofbits/threads/tree/master/2013](https://github.com/trailofbits/threads/tree/master/2013)

------
dang
A thread from last year:
[https://news.ycombinator.com/item?id=20586005](https://news.ycombinator.com/item?id=20586005)

------
seemslegit
This should work nicely with Google Behemoth and Facebook Beelzebub

~~~
burpsnard
See also : SATAN
[https://en.m.wikipedia.org/wiki/Security_Administrator_Tool_...](https://en.m.wikipedia.org/wiki/Security_Administrator_Tool_for_Analyzing_Networks)

------
narrator
You'd think with a name like that that this would be a daemon that would take
as input the cores of killed child processes.

------
yters
There should be a competition for most useful software with most offensive
name. Then we can make all the software stacks of top companies likely to
result in jail time when discussed in public. That would be funny.

------
X6S1x6Okd1st
Is there anyone that uses something like this in a home/personal setting?

------
jzer0cool
I read the instructions but was not clear. Is this just a pcap viewer or
something more? The way I read it, it appears to be a running daemon which
listens on all port which saves the pcap file, which then exposes API's for
accessing such data.

If you had an application http server running, is traffic sent to Moloch
first, and forwarded to the http server like a proxy?

~~~
jlgaddis
When running something like this on a large scale to capture all traffic going
across a network, you'd typically use a "network packet broker" (cf. Google)
that sends a copy of all traffic to the machine(s) running this software.

Your hypothetical application server would not even be aware that this was
taking place.

------
nodaemon
Seriously, AOL named this after the god of child sacrifice, you might want to
put a new marketeer on this project, I won't be touching anything with a name
like that.

------
voz_
Pardon my ignorance - Is this the same AOL as America Online?

~~~
gyehuda
When this project was first published, the name of the company that published
it was AOL. Before that, the company was called America Online. Subsequently
it was named Oath, and now it's Verizon Media.

Names change. The common theme: names are not easy, sometimes they are beloved
brands, sometimes they fall out of favor. Sometimes they were just bad ideas
from the start, but happened anyway.

This gives rise to an interesting challenge for open source projects when you
have an open source project in a github org, and the name of your company
changes (or your company gets acquired), should you move the project? The
problem is real since you don't want to lose your community, but you don't
want to be stuck in the past.

------
frequentnapper
interesting. didn't realize till now that .ch was a TLD for switzerland!

~~~
simlevesque
It means Confœderatio Helvetica

------
hprotagonist
This gets posted once in a while and I always, without fail, think of
Ginsburg.

 _Moloch whose mind is pure machinery! Moloch whose blood is running money!
Moloch whose fingers are ten armies! Moloch whose breast is a cannibal dynamo!
Moloch whose ear is a smoking tomb!_

~~~
bigiain
And whenever anything here reminds me of Ginsberg, I always, without fail,
remember:

“The best minds of my generation are thinking about how to make people click
ads.” –Jeff Hammerbacher

~~~
mdonahoe
For the uninitiated (like myself)

Allen Ginsberg's poem "Howl" starts with the line:

"I saw the best minds of my generation destroyed by madness"

------
ignoramous
Not the greatest choice of name: _Moloch[a] (also Molech, Mollok, Milcom, or
Malcam) is the biblical name of a Canaanite god associated with child
sacrifice, through fire or war._

From:
[https://en.wikipedia.org/wiki/Moloch](https://en.wikipedia.org/wiki/Moloch)

~~~
striking
Perhaps they were going for an association with Allen Ginsburg's Howl
([https://www.poetryfoundation.org/poems/49303/howl](https://www.poetryfoundation.org/poems/49303/howl))
instead?

~~~
lsb
Reminds me of when someone saw that the NSA's public key
([http://www.cypherspace.org/adam/hacks/lotus-nsa-
key.html](http://www.cypherspace.org/adam/hacks/lotus-nsa-key.html)) to be
included in Lotus Notes had an organizational name of "MiniTruth", and a
common name of "Big Brother".

Moloch isn't the good guy in Howl.

 _Moloch! Solitude! Filth! Ugliness! Ashcans and unobtainable dollars!
Children screaming under the stairways! Boys sobbing in armies! Old men
weeping in the parks!_

 _Moloch! Moloch! Nightmare of Moloch! Moloch the loveless! Mental Moloch!
Moloch the heavy judger of men!_

 _Moloch the incomprehensible prison! Moloch the crossbone soulless jailhouse
and Congress of sorrows! Moloch whose buildings are judgment! Moloch the vast
stone of war! Moloch the stunned governments!_

~~~
peterwwillis
_Moloch the dude in the cubicle across from mine that always eats his lunch at
his desk and chews with his mouth open!_

 _Moloch the meddling micromanager from Maine!_

 _Moloch the Junior Architect!_

 _Moloch the 3-space indenter!_

------
dang
Url changed from
[https://github.com/aol/moloch](https://github.com/aol/moloch), which points
to this.

------
kfrzcode
The amount of professional-grade hand-wringing virtue-signaling in this
comment section makes me feel ill.

Why get bent out of shape over the name of a software project? It's virtually
a meaningless factor in day-to-day life.

What about hearing phrases "Sacrificing a Chicken to Moloch," the "spirit
cooking" culture and related symbolism rampant in elite political circles?
Shouldn't we be more interested in that?

Lots of low-hanging fruit to pick, I guess. Someday I might unlock the
"downvote" ability on this platform. Until then my opinions don't carry weight
here. Also, uhh who decides the "threshold" for downvoting? Hint: nobody
knows. [0]

This platform has become more of an echo chamber than a host of rational
discussion based on merit. I suppose that's a problem with growth.

There's a lot of conversation around "Moloch" as a name and the subsequent
emotional responses... but not a lot of discussion about the tech at-hand. And
it's a repost.

Where's the value?

[0]:
[https://news.ycombinator.com/newsfaq.html](https://news.ycombinator.com/newsfaq.html)

Edit: while this is high-ranking comment, I'd point out that if I had the
ability I would have just downvoted the comments I didn't like. Take that for
whatever it's worth.

~~~
TimTheTinker
If you have a strong, negative emotional response to the name of a project, I
think letting the project author know can be helpful.

I’m a rational guy to the best of my ability, but FWIW I would avoid using
this project solely because of its name (just the thought of that name evokes
a sick feeling to me - that’s how strong the negative association is... I
won’t explain why as this isn’t an appropriate venue for that).

We are human, and sometimes visceral responses can’t be ignored regardless of
their irrationality.

I could invoke Godwin’s law to give a more universal example of words with
negative associations, but I’ll refrain.

~~~
simlevesque
Do you think your reaction is professional ?

~~~
TimTheTinker
In a professional environment, I’d keep that to myself unless someone asked me
for feedback or I had a hand in the decision But I think once the point is
brought up in an online forum, that’s an appropriate venue to give feedback
like that.

Like it or not, people will have internal reactions to things—and I for one
would prefer if others let me know if they found a project name I chose to be
difficult in some way.

------
awsanswers
AOL github account's location is Dulles, VA. Thinking of what this does, the
name's meaning and its location in 'the corridor' is giving me the creeps

~~~
gyehuda
I fixed that for you [https://github.com/aol](https://github.com/aol)

~~~
awsanswers
Yeah much less creepy

