Disabling connection tracking on bridge interfaces created by libvirt

Today I got bitten by a problem I've already encountered in the past... and as I didn't document it properly, I had to google it again! Let this blog entry be a more permanent documentation than the previous one.

Early in the morning, the supervision system has started alerting me that the response time of one of the virtualization hosts we use at Matta is going through the roof; making everything 'feel' slow.

Connecting to the system through SSH is already taking tens of seconds, indicating that something is indeed seriously wrong. Modern linux systems use resource isolation in the form of cgroups and are quite resilient to obnoxious programs chewing up resources; The traditional fork-bomb is a lot less effective than it used to be thanks to these improvements.

The following command can be used to see how the processes are grouped by your init system:

ps xawf -eo pid,user,cgroup,args

At this stage, when I've finally obtained a command prompt on the remote server, I knew that the cause of the problem was kernel-related. So my first command was dmesg

...
nf_conntrack: table full, dropping packet.
...

This message is familiar; being a penetration testing company, we are using and abusing of advanced TCP trickery, confusing the hell out of any stateful firewall in the way. Tonight, the culprit was Nmap conducting a SYN scan...

No matter how much resources I could allocate to the tracking table, it will never be big enough... and fundamentally, there is no reason to filter the traffic bridged from the VM to the network using a stateful firewall. In the past I've decided to deal with that specific problem by setting a rule to tell netfilter to specifically disregard the traffic flowing through the bridged devices. This was achieved with the following command:

iptables --table raw -A PREROUTING -m physdev --physdev-is-bridged -j NOTRACK

It worked well; up until the point where a new rule was introduced, negating its effects. The improved solution reads:

echo net.bridge.bridge-nf-call-iptables=0 >> /etc/sysctl.conf
sysctl -p

Live and learn as they say!