Of course you're going to try to avoid storing all traffic, but to decide what's interesting it has at least to be captured first. And on big sites, 40 Gb/s is still only an already random-sampled or pre-filtered subset of all traffic.
The inability to analyze traffic at this rate is a serious problem. How do you study it to see how protocols can be improved? A lab environment cannot compare to real world traffic. How do you detect attacks (not DoS!) if it's hidden in a link operating at this capacity?
Even if you can capture the traffic at wire speed, the CPU doesn't have the power to analyse the stream. I thought that traffic analysers had to be done with FPGA/ASIC because of that.
My manager did his thesis on this. Endace NICs, split traffic up and send to a cluster of IDS servers. Allows you to actually do line rate analysis. No need for FPGA/ASIC.
when monitoring a network, and faced with choices of where to tap the network, a tap which captures a wider view can be advantageous. For instance, capturing at a WAN link can provide a better view of attackers.