
Packetbeat: Open-source application monitoring - mobiplayer
http://packetbeat.com
======
greenyoda
_" Packetbeat agents sniff the traffic between your application processes,
parse on the fly protocols like HTTP, MySQL or REDIS and correlate the
messages into transactions.

For each transaction, the agents insert a JSON document into Elasticsearch
where they are stored and indexed."_

How much performance degradation could be expected from constantly inspecting
every packet on your network with libpcap, correlating the packets into
transactions and then storing information on every transaction in
Elasticsearch?

Also: I noticed that the link to Elasticsearch on the "Getting Started" page
is broken (it's missing the ":" after "http"). Here's a working link:
[http://www.elasticsearch.org](http://www.elasticsearch.org)

~~~
packetbeats
Depending on the traffic, the CPU usage of the agent could indeed be
significant. We're thinking to implement sampling to reduce the effect of
this. On the other hand, the agent is not "in-line" so it cannot introduce any
latency, only steal CPU time from them.

So, unless you have a single CPU core, it's pretty safe to run in production.
It cannot affect the application performance by much.

Thanks for noticing the broken link, we fixed it.

~~~
zentrus
What happens if more traffic is coming in than Packetbeat can process? Either
you are dropping that traffic or you are (perhaps unintentionally) blocking
it, right?

~~~
packetbeats
It will drop traffic in that case.

------
SEJeff
I wonder if this would work with PF_RING[1] from the ntop guys? Regular
libpcap drops packets quite a bit at 10G+ speeds when you have adaptive
coalescing disabled on the network card.

[1]
[http://www.ntop.org/products/pf_ring/](http://www.ntop.org/products/pf_ring/)

~~~
packetbeats
PF_RING is an option but it requires a kernel module and specific network
cards. Before then, the memory map sniffing feature of the Linux kernel [1]
can improve libpcap performance. Not quite to 10G speeds, though, but I think
enough for most cases :-)

[1]:
[http://www.cs.fsu.edu/~baker/devices/lxr/http/source/linux/D...](http://www.cs.fsu.edu/~baker/devices/lxr/http/source/linux/Documentation/networking/packet_mmap.txt)

~~~
SEJeff
Any plans to support it? Part of my job (historically) has been building line
rate pcap appliances that do pcaps at nanosecond resolution (using endace
cards, Mellanox, or solarflare). PF_RING is a bit tricky to get working, but
is completely flawless when you get it running.

This software looks fantastic however.

------
mobiplayer
Hey guys, I'm the one that sent the link to HN, but I'm in no way related to
the Packetbeat gals/guys, but I was so excited about this that couldn't help
myself :) congratulations to the devs, this is amazing!

I've been lately playing with logstash+elasticsearch+Kibana and I think this
is could be huge combined with Packetbeat agents. Can't wait to see what the
people come up with!

~~~
packetbeats
Many thanks for posting it! :)

------
nls
This looks great! Would be awesome if Packetbeat received Postgresql support
too.

------
Nate75Sanders
I believe [http://www.extrahop.com/](http://www.extrahop.com/) is a commercial
service/product in this space.

This looks very cool, though, and I want to play with it.

------
steedsofwar
This looks interesting. I'm actually looking forward to trying this on a game
server i'm working on at the moment. Thanks for making this freely available!

------
knodi
Wow such a brilliant idea. Can't wait to try it.

------
ausjke
what's new here comparing to snmp and all those oss monitoring
solutions(nagios,etc)?

------
aba_sababa
An open source Boundary?

~~~
ianmcgowan
An open source Splunk?

~~~
lewaldman
This would be kibana+elasticsearch+logstash :P

