

ISPs tracking online habits of at least 100,000 users via packet inspection. - timr
http://www.washingtonpost.com/wp-dyn/content/article/2008/04/03/AR2008040304052.html

======
tptacek
This is not a battle between cipherpunks and the NSA. It's a battle between
BitTorrent and cable Internet providers.

I assert without evidence or significant analysis, but with some past
experience: the products WaPo is talking about are used almost exclusively for
two purposes:

* To enable ISPs to build "fast-path" offerings for "premium" Internet access for streaming media.

* To ratchet down the incredibly painful impact that BitTorrent and other "always-on" file sharing applications are having on ISP networks.

The irony is, the high-end products WaPo is talking about are the least
applicable to NSA-style spying. The code handling the packets that AT&T
shunted to NSA is crappy vanilla pcap; any of us could write it.

------
pmorici
Doesn't the ISP open themselves up to more lawsuits from the RIAA and the like
when they start inspecting traffic because they are no longer just pipe?

------
bprater
What kind of software is used to do this?

~~~
tptacek
The leaders in the space are Sandvine and Cisco, and Arbor (my old employer)
just bought their way in by nabbing Ellacoya.

The answer in the general case is, "stuff that looks substantially like
libpcap". It's all written in C, surprise!

On the very high end, or in very specialized cases, you'll find:

* FPGA regexers, OEM'd from a couple common vendors, that compile DFAs into gates.

* Multicore "network processors" with MIPS cores and fast custom memory busses.

* Blades that connect into the backplanes of (ubiquitous) Cat6k switches to intercept traffic.

By and large, this is a systems design problem, not an algorithms challenge.
To appreciate that, you have to get the context for this WaPo story, which
includes the fact that the story is totally overblown and ISPs are not reading
your email or web traffic, but rather trying to figure out how to
commercialize a "fast-path" network product for bulk P2P/streaming media
customers.

~~~
anupamkapoor
shouldn't narus-networks be up there somewhere ?

edit: also, packet classification is quite cool from research perspective. for
example, check out the following : "packet classification on multiple fields,
gupta, mckewon". also, network algorithmics devotes an entire chapter on
various strategies for efficient packet-classfication.

~~~
tptacek
Network Algorithmics is an awesome book; it's been recommended here before.

But I don't think most of the DPI products use advanced packet classification
algorithms. No product I've worked on has; it's pretty much, "that's port 80,
so use the HTTP decode". There's classification done for binning and
accounting, but it's pretty brute force.

Narus is a DPI vendor, but not of the type WaPo is talking about. They provide
"lawful" (read, "unlawful") intercept for traffic that has already been
classified and diverted.

