Debian network bridge setup and 10Gb networking tips


I don’t like Network Manager. I know I probably just need to learn it, but why? It seems to half use config files. And I don’t want it automatic! Sure, maybe on desktops, I suppose Network Manager is actually easy and works fine. But it always gets in my way on servers.

One thing driving me nuts the last few days is that the network interface would just drop. And it wouldn’t come back up. I discovered that once it tried to to get a new lease it would fail. Network Manager said something about “exit-code” and there wan’t an interface. Except there was. I may actually have a hardware problem still, but NM made it way more of a hassle to take the network up/down and troubleshoot. Just take my darn config files and leave me alone please!

Anyway on my new Debian load for my dev server I just wanted DHCP on a bridged 10Gb ethernet NIC for the host and guests. Here’s what I did.

Setup the config files.

/etc/network/interfaces.d/enp36s0

auto enp36s0
iface enp36s0 inet manual
        mtu 9000

/etc/network/interfaces.d/br0

auto br0
iface br0 inet dhcp
    bridge_ports enp36s0
    bridge_stp off
    bridge_fd 0
    mtu 9000

Then disable Network Manager and restart networking:

systemctl disable --now NetworkManager
systemctl restart networking

Bam! There’s my networking!

I’ll see if I run into issues still. Like I said, I may have some other problems. But at least its easier to work with!

Here’s a couple other quick tips I’ve run into regarding 10Gb networking. On Linux, for years I would disable TCP Window Scaling. I ran into so many issues with rsync’s and SSH between client sites and someone at our provider suggested it may be Window Scaling. With that disabled I had much better SSH data transfers and on the LAN I couldn’t tell the difference. This goes way back to Ubuntu 12.04, 14.04 days actually, but it stuck on all my servers since. For 10Gb networking though, that caused me a ton of issues and very slow transfers over the LAN. Make sure its not disabled if on 10Gb LAN. Also, I’m not seeing any difference with SSH syncing over the internet these days, so cool!

Also, I’ve seen advice from RHEL about different TCP Offloading options for the network. I was limited to only 300Mbps iperf on 10Gb LAN until I did A) enable TCP Window Scaling, and B) change the offloading using ethtool. I setup and /etc/rc.local file on CentOS and it also seems to work on Debian. I need to double check my tests, but originally these changes upped my speeds with iperf to about 800Mbps.

/etc/rc.local

#!/bin/bash


ethtool -K enp36s0 tx off sg off tso off gso off gro off
ethtool -K enp43s0 tx off sg off tso off gso off gro off

Obviously change to your interfaces.