
Using tcpflow to debug HTTP API libraries - mathie
http://woss.name/2011/03/06/using-tcpflow/
======
tedunangst
Funny that it mentions twisted (briefly). I last used tcpflow not long ago to
debug some issues in twisted proxy. Twisted has two quirks that make it
slightly less compatible/forgiving with the web at large than most browsers.

1\. It lowercases headers, such that Cookie: becomes cookie:. Technically, the
standard says they are case insensitive.

2\. When reading responses, it strictly looks for \r\n line endings. \n will
not cut. Standard does say \r\n is the only acceptable ending.

Guess which web server violates both of these rules? The one in arc that
powers Hacker News. Figuring out why my proxy worked with every website except
HN probably wouldn't have been possible without tcpflow.

------
jedsmith
Wireshark's _Follow TCP Stream_ capability does roughly the same thing.
Wireshark isn't the prettiest on OS X, but it works, and the ability to do
this after-the-fact with tcpdump-created dumps from a remote box is useful.

[http://www.wireshark.org/docs/wsug_html_chunked/ChAdvFollowT...](http://www.wireshark.org/docs/wsug_html_chunked/ChAdvFollowTCPSection.html)

Charles is also extraordinarily handy. I used it extensively while developing
an iPhone app in the simulator, since figuring out what NSURLConnection is
doing is _hard_.

<http://www.charlesproxy.com/>

------
yoda_sl
I typically use tcpflow or Charles on Mac OS X. They are both useful tools
that any back end engineer should know how to use. It helps diagnose any
problem your app/code is facing and it's another good way to learn how
existing apps interact with their back end web service. I often check how iOS
app interact with the server by usually setting up a proxy on my laptop and
configure on the iOS device settings to use my local proxy. Then simply fire
away tcpflow or Charles and watch all the HTTP calls being made.

~~~
psadauskas
Your first statement is a bit ambiguous, but I have to ask if you've gotten
tcpflow to work on OSX. I can never get it to output anything when trying to
monitor connections to localhost.

~~~
mathie
If you're looking to monitor traffic to localhost, bind the the `lo0`
interface. For example, when I'm monitoring traffic to a local Rails app in
development, I'll do something along the lines of:

    
    
        sudo tcpflow -c -n lo0 src or dst post 3000
    

and it'll pick up the traffic.

(Thanks for the tips on alternative ways of monitoring TCP traffic flows,
folks!)

------
wpeterson
This is why you need to hire full stack engineers, or have at least one on
your team.

------
andrewcooke
How does tcpflow compare with ngrep?

~~~
tedunangst
tcpflow is rather simpler, but probably a lot better for examining multiple
connections. You specify a tcpdump/pcap filter and it saves the traffic to
files (two per connection, one for each direction). You review the data at
your leisure, it's not all jumbled up in your terminal.

~~~
nailer
Can tcpflow set up fake TLS certificates like Charles?

~~~
tedunangst
No. It takes data from the network and puts it into files. That's all. It
doesn't inject, interpret, or decrypt data.

