Hacker News new | past | comments | ask | show | jobs | submit login
Tcpdump is amazing (jvns.ca)
567 points by ingve on Mar 17, 2016 | hide | past | web | favorite | 102 comments

Tcpdump is amazing. The quickness that you can have with it over Wireshark is awesome. I love that it is a command line tool instead of a GUI tool, since I needed to analyze TCP packets for quick debugging purposes.

But for the people who are new(ish) to tcpdump have you heard of libnet and libpcap? You can basically build your own tcpdump! :D

I was amazed at the speed packets fire when you program it yourself in C.

See: https://github.com/the-tcpdump-group/libpcap <-- CAPturing packets https://github.com/sam-github/libnet <-- sending packets

Libnet tutorial that I used religiously: https://repolinux.wordpress.com/2011/09/18/libnet-1-1-tutori...

Wireshark also comes with "tshark" which is a CLI tool that gives you access to display filters and some of the GUI features. In particular, it's great for scripting if you're trying to pull some stats while troubleshooting.

Tcpdump is cool, tshark is cool, but nothing beats raw libpcap for grabbing raw data off the wire. It is simply so stable at high throughput it's often the only option.

Wireshark starts to break down at a certain point. When that happens, I've found scapy (Python pcap parser) very useful. Once that breaks down, a good option reading the hex directly in C.

Once that breaks down... Well, there be dragons.

Got to add dpkt to this list for parsing pcap streams - allows you to write some awesome python analysis tools - e.g. For this 5GB packet capture find me all the streams where tcp packets were retransmitted more than twice.

Dpkt is also a very good, although I've had some bugs with it on Mac. Probably just odd network issues.

I use tshark for capturing a lot, it's great. One of the big advantages for me is that it allows you to use the more expressive (and familiar) wireshark filter syntax, e.g.

tshark -i eth0 -Y ip.addr==

Instead of having to remember tcpdump's own fiddly syntax. Given that most of my protocol debugging work is done inside Wireshark, I'm much more fluent in that filter language.

Note that the above snippet is a display filter, so it's capturing all the packets on the wire, doing a full dissection, and then running them through a filter before printing them; if you're trying to capture on a saturated 1G link you might need to fall back to the capture filters which are simpler and faster (and I believe also use tcpdump's syntax).

Wireshark syntax is also fiddly. It's just more familiar for some folks. The first several years of my experience troubleshooting network problems happened when Wireshark didn't exist, but tcpdump did. I still have better recollection and comprehension of tcpdump syntax than Wireshark. I have to read the docs to use either these days (since I don't work at that layer of the network much anymore), but I have to read longer to make Wireshark do what I want.

Sometimes "ease of use" is really just "what I'm used to". (Not saying Wireshark isn't more expressive or more powerful...it probably is. But, I usually have a very basic use case, and tcpdump can do it with a couple of flags and an address or port.)

If you use VM's for this sort of thing, having images laying around like: https://security-onion-solutions.github.io/security-onion/ gives you most of the tools you'd need (wireshark, scapy, etc. in one place). Seems to be the best way to go for me (Mac laptop user).

Have you used sysdig? https://github.com/draios/sysdig

Its creator was one of the guys behind wireshark. They're pitching sysdig as strace + tcpdump + htop + iftop + lsof + ...awesome sauce.

They have csysdig which is similar to tshark

I never knew, thanks :)

> I was amazed at the speed packets fire when you program it yourself in C.

For sure. There's no better way of learning the protocols than writing your own network sniffer. Even writing a crappy one like the one that a few folks and I put together in college will teach you a lot about how each protocol behaves and what to look for: https://github.com/carlosonunez/nbfm-sniffer

And tcpdump already exists on every server. I frequently pipe the output of tcpdump over ssh into wireshark for analysis on my local machine.

ssh root@HOST tcpdump -U -s0 -w - 'not port 22' | wireshark -k -i -

>> Also if you understand how to reason about the overhead of using tcpdump ("below 2 MB/s is always ok"?), I would REALLY REALLY LOVE TO KNOW. Please tell me.

I think when Dick Sites from Google says he has a rule of "stay below 1% of overhead" when analyzing traffic on datacenter nodes he wants you to select the right tool for the job. In the context of tcpdump you can run it with so many options that makes it very powerful. But that power is dangerous in the hand of a novice user. A simple error in how you run it (maybe missing filters or too wide an address space) can cause you to shoot well above the theoretical 1% limit. But that's not the tools fault IMO.

Anyway this seems more theoretical, because in practice I'd prefer a hardware based network tap and analyze that without creating any risk to the live traffic

Using something like "time tcpdump -c 1000 -w file.pcap ..." which stops after 1000 packets is always a good idea. It helps catch a too greedy BPF expression (PCAP filter), before using to many resources. Also always output to a file first, and then you can check the output afterwards with "tcpdump -nr file.pcap | less".

The -c option is indeed a good idea.

However, one of the worst things that tcpdump does is to put the NIC into promisc mode. On a physical NIC, this can be VERY expensive and may involve bouncing the link (behind your back) and dropping packets. At the very least, it can wreak havoc with steering filters on some NICs. To prevent this, use the -p option to prevent tcpdump from putting the NIC into promisc mode.

Another issue with tcpdump on an endstation is caused by stateless offloads like checksum offload and offloads like TSO on the send side, and GRO / LRO on the receive side. Because the BPF filters are applied between the network stack and the device driver, you may noticed tcpdump / wireshark complaining about bad checksums on transmit -- this is likely due to checksum offload. And you may see gigantic (way larger than MTU) sized frames. This is due to GRO/LRO on receive, and TSO on transmit. You can disable stateless offloads (ethtool -K on linux, ifconfig on bsd), but that will slow the entire system down.

> Anyway this seems more theoretical, because in practice I'd prefer a hardware based network tap and analyze that without creating any risk to the live traffic

I'll ask amazon to install it when I need it...? They probably won't mind.

Unless I misunderstood her ... I think the argument she makes is analyzing traffic in a datacenter and she talks about analyzing network traffic on a scale. Also she mentions Dick Sites who in his talk mentions datacenter technologies and not how to debug as an end user in some cloud.

Sure. I'm just pointing out that it's becoming increasingly rare to be able to replace a given use case of tcpdump with a physical tap, so you can't always sidestep the nastiness of calculating overhead—especially on a live system.

thinking about it further I wouldn't want even a 1% fluctuation caused by analysis since it distorts the data captured in statistics. And as a datacenter guy who hasn't written the software I might not know the sensitivity level that some of my software is depending upon (timeout, max-latency, etc) which could spin it well above 1% not because tcpdump but because of a domino effect.

So if it would be my datacenter I'd have a rule of no analysis of non-tap data.

A hardware tap becomes more and more meaningless in modern SDN networks behind dynamic virtualisation platforms.

Sure, you can find out where your VM is currently physically hosted, then tap the 4x10Gbit that come out of that hypervisor. Your network stream is probably somewhere in there.

But it is mixed with a lot of other data, and you could miss the bits that go directly to other VMs on the same hypervisor. Also it is encapsulated, split up in multiple streams, multiple virtual networks etc. If the VMs is moved (for load-balancing or maintenance or whatever reason) your tap becomes useless and you have to chase your VM.

>I'd have a rule of no analysis of non-tap data.

This is a ridiculous rule, you are cutting off your nose to spite your face. The saying "perfect is the enemy of good" comes to mind.

To your above point

>maybe missing filters or too wide an address space

There is merit in just capturing a shitload more than you need (provided of course, you're not trying to cap a full 10Gbit) because it is often the filters that are the cause of any "performance" issues, however you define that.

the rule depends on context which I failed to mention. I agree with you in context of non critical infrastructure. Though I'm not talking about web apps in AWS but places where you still want to have your own datacenter like in a banking or telecoms, or insurance or medical environments.

EDIT: whether you allow any access to a node for whatever purpose other than the software that was meant to run on that node would probably depend on what damage is done if that node goes down. If the damage is a blip in statistic and you can live with that fine but that's not always the case

You're still overthinking things and dealing in hypotheticals it sounds like to me.

Almost the only time you would want a physical tap is if you need a permanent tap capturing everything over a long term - often for the purpose of running through an IDS/IPS, an even then SPAN/RSPAN/ERSPAN works pretty well.

Even in those industries you mention, most of the other time you are doing a capture is to troubleshoot something, so you don't need to run things for a long time, nor does any "perf issue", which is overstated, probably matter, since things are possibly already half way to fucked. And the compliance argument doesn't hold either vis-a-vis installing tcpdump - your processes and policies would be written as such to allow for debugging (or should be)

If you had 100 computers, <1% overhead could mean "capture every packet going over a 1 Gbit network interface going to only 1 computer, even if it has a 100% overhead on that computer".

As a network engineer, tcpdump and friends are on my list of most frequently used applications.

If you're not a network engineer, you'd be amazed at how many times we get issues escalated to us exclaiming that "it's the firewall" and demanding that we fix it.

It's usually not the firewall, though, and unfortunately it falls on us to prove that that's the case. It's not always easy but, luckily, tcpdump and friends allow me to show that and I can punt the issues back to where they came from.

(Several years ago, I was able to prove it was a customer's on-premise firewall -- managed by them -- and not our firewall based upon the packet timestamps and this little thing known as "the speed of light".)

I'm not a network engineer and oftentimes have to use tcpdump and mtr on behalf of network engineers combined with Chrome devtools to show application developers that there's nothing wrong with AWS or even the internal enterprise network and that it's closer to them. The hard part is trying to make these diagnostic workflows scale for tens of thousands of application developers when you have maybe a handful of people that can help them in this manner.

Sorry for the stupid question, but by 'friends' you are referring to a utility or just peer support? If its the former, could you post a link? Thanks.

It's a pity the article doesn't touch on BPF. BPF is the magical bit in tcpdump.


She mentioned it, just not explicitly. BPF is the super efficient optimized compiler that tptacek tweeted about.

No, BPF is the extremely simple and limited virtual machine. The optimized compiler is in libpcap and does some neat stuff to eliminate redundant code. There are implementations of the pcap filtering language that do not compile to BPF, and there are other languages compiling to BPF which do not benefit from the compiler in libpcap.

It's worth noting that optimizer is also absolutely necessary since the design of the pcap language is such that very simple filters are easy, anything even remotely complicated becomes very verbose and repetitive.

I'm pretty sure the three things (BPF, tcpdump, and the pcap compiler) were developed in tandem, or, at least, the latter two were.

I had a job, about 15 years ago, hacking on a customized version of libpcap (mostly to do filter merges). There is a surprising amount of stuff going on there.

Indeed. There's a great talk about those early days at https://www.youtube.com/watch?v=XHlqIqPvKw8 ; I'd recommend anyone who is interested in the design and implementation of little languages to watch that. Few things will make a language designer throw out everything and restart from scratch, but I guess having Van Jacobson dismiss the initial design as unusable would do it!

But after that common starting point they quickly developed lives of their own, and should not be treated as a single unit. You'll get things like pflua as a completely distinct implementation of the filtering language compiling to lua instead of BPF, pfmatch with language extensions that are not really compilable to BPF, seccomp and other uses of BPF in the Linux kernel for things that have nothing at all to do with packet processing, a (e)BPF code generator backend in LLVM, and so on.

BPF and libpcap are certainly separable, but tcpdump is practically just the test driver program for libpcap. :)

> BPF is the extremely simple and limited virtual machine

No longer true in modern Linux.


Yes it is. Going from 2 to 10 registers and getting map data structures and compilation to machine code doesn't change that BPF is intentionally simple and even disallows loops. "simple and limited" is a good thing.

It's simple, sure. And it's limited, in that there are sane limits in place, because you don't want an application running in the kernel to loop for forever, but 'extremely simple and limited' I don't think is fair.

You can do a hell of a lot with BPF, and it's not like work to extend it's functionality is slowing down, either. We've had tons of features implemented in the 4.x family, stack walking is likely to be implemented in the future, etc.

When you have people like Alexei Starovoitov and Brendan Gregg saying it can do all sorts of things including "crazy stuff", I think we've moved beyond 'extremely simple and limited'

You're reading a negativity into that phrase that isn't really there. It's harshly limited, but in a careful way. Despite having few parts and not being turing complete it has a massive number of use cases.

BPF is sort of boring compared to libpcap.

Actually, you can decode a stream as HTTP traffic in Wireshark. It assembles related TCP packets and puts together the conversation from the viewpoint of the application layer.

even with SSL you can MITM yourself with it


Nitpicky, but: this isn't MITM, this is the client side of the connection writing out the keys necessary to decrypt the wire traffic. This is important because the process doesn't change alter the network data in any way, whereas a MITM proxy would.

You're right. Not MITM but decoding tap.

If you want a true MITM proxy then I recommend Burp


unfortunately this is only for http

You can use netsh on Windows without need to install anything external (beside capture file viewer - Microsoft's Message Analyzer - but that can be done on your workstation rather than servers)

  netsh trace start capture=yes IPv4.Address=

  netsh trace stop
The last command will output path to .etl file containing captured packets.



What's really irritating about the windows stack is how difficult it is to dump local host traffic compared to the same problem on Linux. I believe it's because the loopback interface isn't as completely implemented.

I read the title and think "Ok, lets see what tcpdump has that wireshark doesn't" and what do I find inside ?

Article about wireshark :)

I guess the most obvious thing is what it not has, which is a massive GUI. It is much faster to work with when you know what you are looking for in my opinion. In most cases running wireshark without tcpdump first is really inadequate.

For wireshark, there is tshark for a nice command-line based version with none of the GUI overhead. But usually, I'd rather just use tcpdump.


My usage or Wireshark is rather sporadic, so I appreciate the traffic drill down I can do w/o any knowledge in advance about the protocols I have captured.

True, honestly I expected to see more TCPDUMP tips :(

I really love ngrep[1] for debugging network traffic (for smaller problems).

Give it a filter (BPF) and the pattern you're looking for, and off you go: $ ngrep -W byline -d en0 "INVITE" port sip and host sip.phone.tld

[1]: http://ngrep.sourceforge.net/usage.html

What's the advantage of using ngrep over tcpdump?

It is based around filtering content, not metadata. Breaks down with TLS though.

Wireshark is great, but it's fundamentally an interactive tool; you fire it up when you have a problem to look into and close it down afterwards. Running wireshark/tcpdump all the time seems like it would generate too much data. I'm not sure how well Wireshark handles dumps larger than available RAM, either.

The author mentions the pcap filter language, but one of the misfeatures of Wireshark is that it has a different filter language for the GUI filter box.

The capture filters and display filters are interfaces to two sort of unrelated pieces. Maybe the misfeature is that wireshark doesn't do good job of communicating that.

BTW, If you want constant monitoring for your applications, you can buy a shark appliance that you can always go back to to investigate issues.

There are commercial products that you can leave running all the time to generate data from your packets on the fly, such as ExtraHop (http://extrahop.com). There are also continuous PCAP tools, but they need massive amounts of storage and in larger environments the lookback you get is limited.

Capture filters not being similar to GUI filters have always been the #1 issue I've had with Wireshark.

Some Linux distros (Debian family) separate tshark from wrireshark which is great because that means you can install tshark without X-Windowsetc... I wish the other predominant distro family (Fedora) did the same.

I have not used tcpdump a lot, but I believe there is nothing it can do that tshark cannot.

Tshark can capture only the portion of traffic you want via filters, which can reduce the size of your pcap files considerably, and possibly haves less of an impact on performance.

When using tshark, make sure you capture the traffics to a file too, so you can go back to look at something that happened x seconds or minutes ago.

I understand the security concerns, but forms being able to debug with tshark (or an equivalent tool) is a very good reason for not using HTTPS internally.

> I have not used tcpdump a lot, but I believe there is nothing it can do that tshark cannot.

How about go two months without a security vulnerability in packet processing logic? Drop slightly fewer packets? (Yes, dumpcap too)

> good reason for not using HTTPS internally.


> I understand the security concerns, but forms being able to debug with tshark (or an equivalent tool) is a very good reason for not using HTTPS internally.

One of the nice things about HTTPS is that it's just a SSL connection and the HTTP goes over it. It's not hard to use socat to create something which listens to TCP connections on one port, and wraps it in a SSL layer and sends that on to another host:port. This allows you to remove the S from HTTPS, and you then speak HTTP to it, and hence can use tcpdump (etc) to dump.

Tcpdump use the BPF syntax to filter packages. In the current linux kernel,BPF was implemented and extended as a kernel virtual machine,when cooperated with the perf module,they can be used for collect trace info of the system. see https://lwn.net/Articles/599755/

No, tcpdump uses libpcap to filter, and libpcap compiles down to BPF.

Thanks for your explanation,I'll dig it deeper.

One of my favs, a tcpdump filter for HTTP including request headers:

sudo tcpdump -s 0 -A 'tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420'

This is useful for troubleshooting outbound requests that your backends are making. I've had the interesting logic explained to me but can't remember the details.

It's good to have things like that in your toolbox, but it's probably also a good idea to figure out how they work and what their limitations are going to be, otherwise you'll may hit situations where they don't work and you don't understand why.

So, start with the magik number. If you look up those 8-bit ASCII codes you'll see that it spells out GET followed by a space, which should give a clue as to how it's working. So it will capture a lot of HTTP requests, but it may not be getting them all.

Aren't tcpdump and wireshark some pretty interfaces to libpcap? Also linking the home page of tcpdump/libpcap [1], and check the documentation about the packet capturing.

[1] http://www.tcpdump.org/

They're intelligent interfaces to libpcap. If you have a pretty interface, you get nice opaque data you'll have to understand yourself. With intelligent interface you can do high-level processing like "I only want to see GET requests".

But they're still fairly stateless... without some extra scripting you cannot do a query for "TCP connections with more than 3 restarts".

What I meant with pretty, is pretty for human eyes, unless you are able to read TCP/IP headers ( for example ) directly in hexa.

Regarding ther other topic, nothing stops someone to capture the packets to a generic DB ( sqlite or berkleydb ) to operates such queries. Looks like a weekend project!

I started really using tcpdump when I discovered "-X" switch. It displays all the traffic as it happens in hexadecimal, so you can then use other standard unix tools (grep, less, redirection...) to check the traffic. For inspection recording is still better, but nothing beats "tcpdump -X" when you just want to know if the packages are arriving at some port.

You also may want to use the "-A" switch if you are looking for ASCII content, the command ngrep..

tcpflow is another useful tool. It's similar to tcpdump, but reassembles connections by sequence number. I've found it easier to use for application-layer analysis, where you care more about the data being sent than what literally appears on the wire.

One thing that you have to watch out for with tcpdump in the field (and especially in production) is that it doesn't rotate its capture files by default, and so you'll eventually end up with very large capture files which aren't loadable in Wireshark (which struggles to load capture files of size ~= avaliable RAM).

See http://superuser.com/questions/904786/tcpdump-rotate-capture... for an example of the required syntax.

Note that tcpdump's default behaviour here is better than Wireshark's; last I checked Wireshark just crashes when your capture file exceeds the available RAM. Again, you can enable file rotation, but many don't realize this until they have been bitten by this attempting to do an overnight capture of a production issue...

I have used tcpdump in the past to capture traffic when I had physical access and ability to implement a hardware tap and analyze packets after feeding in packets with tcpreplay from another network's pcap file (IIRC). The point was to configure an IDS like Snorby using rules I derived from the packets I analyzed in Wireshark (from the resulting pcap file).

However, I haven't seen a need to use tcpdump in awhile since my problem domains have been quite different in that my focus back then was primarily network monitoring. Usually performance problems where I have worked have been easy enough to identify at a higher layer (e.g. n+1 select issues with SQL).

Julia Evans' blog is pretty great overall.

Her talks are great too, I particularly enjoyed her strace talk [0].

[0] https://youtu.be/4pEHfGKB-OE

Just yesterday tcpdump saved me an indeterminate amount of time working with a Horribly Problematic vendor's support team by allowing me to sniff the loopback interface to pinpoint which of two (closed-source) components was introducing a sizeable processing delay in our real-time network event management system.

I am grateful for the tool.

You can capture with tcpdump while analyzing it with wireshark at the same time (https://wiki.wireshark.org/CaptureSetup/Pipes#Remote_Capture).

If you are interested in tcpdump and use it for debugging, you might potentially also be interested in the Bro network monitoring system (http://bro.org).

It gives you very deep visibility in the supported protocols, dumps easy to parse log-files by default (see e.g. https://www.bro.org/sphinx-git/httpmonitor/index.html for HTTP information) - and it is fully scriptable.

(Disclaimer: I am involved with the project.)

Here's my primer for anyone interested:


Tcpdump and wireshark are built on top of libpcap. If you are interested in learning how to write programs in C or Go using libpcap check out these posts.



We use tcpdump all the time to capture traffic that we later analyse with Wireshark, so I'm a bit surprised by some comments trying to confront the two.

I apologize if someone already mentioned it, but you can capture packets from Wireshark directly as well, without the need for tcpdump or tshark. However, this is only useful if you want to capture packets on the machine you are running Wireshark, obviously.

true, for machines without a monitor you normally use tcpdump and tshark then do local analysis with wireshark, plus wireshark is a little heavy for real time capturing sometimes.

This person's blog in general seems pretty great. This was posted a while back from there: https://news.ycombinator.com/item?id=8167546

With tshark used to be possible to capture packets using display filters, which was pretty handy in lots of situations. When privilege separation was implemented, the feature was lost(bug 2234).

I understand the importance of privilege separation, but I miss the feature.

The problem is that encryption (https everywhere) spoils its usefulness for the larger part.

If you have the private key, you can decrypt it in Wireshark.

Inaccurate. Most browsers will now be using ephemeral key exchange. You pretty much have to configure one of the end points to dump session keys to a log file, then load that in to wireshark alongside the packet dump.

It would be nice if there was some automated way to do this.

You can use mitmproxy [1] to dump the TLS Master Secrets for all connnections it intercepts [2]. The dumpfile goes straight into WireShark.

Obligatory disclaimer: I'm one of the mitmproxy authors - happy to answer any questions.

[1] https://mitmproxy.org/

[2] http://docs.mitmproxy.org/en/stable/dev/sslkeylogfile.html

tcpdump is fantastic; I often use it in lieu of Wireshark if I can. It's also a bit faster, which kind-of doesn't matter for me since I usually have it output the trace to a file and then use less to go through it.

windump is the windows version of tcpdump: https://www.winpcap.org/windump/

I haven't used it yet but it's libpcap based so I can't imagine it being too different. It has to be at least 2000x better than the piece of shit Microsoft Network Monitor (it's like Wireshark, except so much worse...oh, and it doesn't do promiscuous mode)

I love that the PCAP format has become the defacto standard for analyzer tools to use. I just saw an HN article about a new analyzer based on PCAP.

If you are on Windows, I highly recommend starting with Netmon (Network Monitor) instead. It is 100 times easier to use than tcpdump and Wireshark.

It's also great combined with something like CURL you can upload the captures to CloudShark and skip using Wireshark entirely.

What do you see as the advantage of using CloudShark over Wireshark? I see CloudShark costs between $200-$9000 a year and Wireshark is, of course, free so I'm wondering why CloudShark is preferable. Is it the collaborative aspect?

love this, I always thought tshark is similar to tcpdump for CLI, did not realize it knows more protocol than tcpdump, learned something new today. Thanks!

Didn't use -n in any of the examples...please...

particular advantages over wireshark?

tcpdump is just a way of capturing the traffic, it's not a great analysis tool. It's simple, command-line, and part of the base install for many distributions. Generally very useful when you're remote debugging where you don't have a GUI: the workflow is tcpdump -> download .pcap to local machine -> analyze offline with wireshark. Alternately, tcpdump -> analyze in situ with tshark. (You can also use tshark directly, but then you don't get a do-over if you want to change your display filters and such). Also, easily scriptable.

it's really amazing

Tcpdump for president!

For a moment I read Tcpdump as Trump

Registration is open for Startup School 2019. Classes start July 22nd.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact