In what use case would iptables give a performance penalty ?
I have iptable-setups running with hundreds of rules and 800-1000 concurrent(!) users. During traffic peaks times my 5-6(?) year old Xeon does a good job, cpu usage barely touches 3-4%. Throughput downstream at that point around 350Mbit/s (admitted, downstream just matched by conntrack:matched/established) and upstream around 20Mbit/s, working through hundreds of rules.
iptables runs in the kernel-space and is very, very, VERY performant. (well, I lied, iptables itself is just a configuration tool for the kernel - but it is incredibly fast).
Following that kind of traffic with tcpdump (userland) just drops about half of the packets because it maxes out that poor CPU instantly. (yes, that depends on the args). And don't even thinkg about using iptraf :-)
(That reminds me: please, some one, send me better server hardware :-x)
1. Don't use connection tracking
2. If you need to use it, make judicious use of -j NOTRACK
I use this on a few high traffic servers, with good results:
iptables -t raw -A PREROUTING -i lo -j NOTRACK
iptables -t raw -A OUTPUT -o lo -j NOTRACK
iptables -A INPUT -i eth0 -p tcp -m tcp --dport 80 -j ACCEPT
iptables -A INPUT -i eth0 -p tcp -m tcp --dport 443 -j ACCEPT
iptables -t raw -A OUTPUT -o eth0 -p tcp -m tcp --sport 443 -j NOTRACK
iptables -t raw -A OUTPUT -o eth0 -p tcp -m tcp --sport 80 -j NOTRACK
iptables -t raw -A PREROUTING -i eth0 -p tcp -m tcp --dport 443 -j NOTRACK
iptables -t raw -A PREROUTING -i eth0 -p tcp -m tcp --dport 80 -j NOTRACK