Tracing HTTP Requests with tcpflow

[ananthakumaran]

https://ananthakumaran.in/2022/11/12/trace-http-requests.html

[imiric]

For a tool that understands TCP and HTTP conversations, the filter syntax is awfully similar to tcpdump's, and having to use `grep -A 15` to filter specific requests seems clunky.

It's good knowing this tool exists, but I think I'll stick to tcpdump and Wireshark. In Wireshark it's trivial to use the `http.request` filter to do this, and following the TCP conversation with decoded bodies, or specifying a TLS cert, is equally simple.

[denysvitali → imiric]

For a CLI Wireshark, use tshark

[jimmyed]

There is also netpeek[1] which has better filtering capabilities and UX overall. It supports ngrep like bpf filters which we found useful.

1: https://github.com/darshanime/netpeek

[thegabez]

This assumes you have shell and those tools installed via your Dockerfile. Best practice is to have a multi stage build and just have the working binary, keeping the image as light as possible.

So when do you make a fat image with debug tools and when do you keep it skinny?

[mozman → thegabez]

Or you’re running on a VM and can install these tools for debugging and then blow it away when you’re done

[ystad]

A thought. I would usually start with my application logs. Reverting to a tcp tool for a http app problem seems cobtrived

[0x445442 → ystad]

Yeah the app looked like it was delegating to another service and needed a bearer token for that. The app logs should have showed this straight away. Furthermore, I’ve been burned enough to know that if I have a service that’s dependent on other services then I setup tasks in my service to ping the health of those services periodically and log warnings/errors if they’re unreachable. This allows me to divert blame as quickly as possible if alerted in the middle of the night.

[nhoughto]

This looks handy, better ergonomics than strace to just hook into http requests of a running process.

I wonder how it works with TLS if it’s working at the socket level..

[xtreak29 → nhoughto]

There seems to be an issue open for this https://github.com/simsong/tcpflow/issues/58

[youngtaff → nhoughto]

I’m guessing it doesn’t work with TLS as it’d need the keys to decrypt the traffic

[LinuxBender → youngtaff]

I recall seeing a thread somewhere saying tcpflow would not add this capability and they point people to ssldump [1][2] and even that has some limitations.

[1] - https://github.com/adulau/ssldump

[2] - https://linux.die.net/man/1/ssldump

[fmajid → youngtaff]

With TLS 1.3 and Perfect Forward Safety, even knowing the secret key is no longer enough. Otherwise, ssldump is a tool that would dump sessions if you knew the server private key and were somehow able to force negotiating a non-PFS TLS 1.2 cipher suite.

[_ZeD_]

in this space I have successfully used mitmproxy, charles and, with the best experience, fiddler

mitmproxy.org www.charlesproxy.com https://www.telerik.com/fiddler/fiddler-classic

[rwestergren → _ZeD_]

+1 for Fiddler, though it doesn't sound like the author was in a position to forward traffic to an HTTP proxy at the time in prod

[29athrowaway]

There's also termshark, a TUI for tshark inspired by Wireshark.

You also have mitmproxy which can be useful if TLS is involved.

[aftbit]

Just because I know them better, I would have used `netstat -putln` and `tcpdump -i lo -n -A port 8000` to do the same thing. I'll take a look at tcpflow and ss though - it's always nice to know more tools.