Background

My NAS died this year from a dust bunny short on the main board. When I replaced it, I wanted long-term growth for my home lab.  I included a 10GbE card on the replacement NAS to feed the home lab.  To distribute the isolated storage network I bought a TP-Link TL-SX105 5 port unmanaged switch.  This storage network is to feed my Proxmox system and my DGX Sparc.  I was inspired this week to build an ambitious project and I was surprised to find something was slow on my builds.  Yes, I know the real bottleneck will always be the disks.  However, when I started basic troubleshooting, I was surprised to see sub gigabit network speeds.

The storage network being slow was a surprise.  To get better performance metrics, I setup iperf3 on the three members of the network.  I also tested direct connections between members (skipping the switch).  It was a long evening of troubleshooting, but rewarding.  I found out about three surprises to share with others.

First: Is it the physical or data layer?

I started with removing the switch because I was observing the arp table were not being populated as I expected.  After fussing with the physical layer, I could prove that the physical layer was not the problem.  However, I still wasn’t seeing proper arp tables on all nodes.

At the data layer, Ubuntu on the DGX had a problem.  The first problem, is I found a typo.  While the wrong address was still in the network, it explained why one connection test wasn’t working.  This lead to finding out about an existing bug.

The network in use was 192.168.200.192/26.  The broadcast address for this specific fourth block happens to end in .255 and the subnet math should have lined up. However, the system configuration script explicitly passed the noprefixroute flag alongside a forced, manual broadcast parameter.

Apparently, when Linux sees a manually forced broadcast address alongside noprefixroute, it indicates that the system’s local network manager (NetworkManager) is not generating the implicit local link route inside the kernel routing table  As such the local link route (192.168.200.0/26) was never dynamically spawned by the kernel.  Whenever the system tried to communicate on the storage network, it would look at its routing table and find no local interface route that knew how to reach that IP.  When a Linux kernel lacks a local interface route for an IP, its default behavior is to use the default gateway.  Thus, the system was taking high-speed storage traffic and trying to route it out the 1Gb front end network.

I didn’t know about this specific problem with noprefixroute, however this is the class of problem I expected to find.  As such, I fixed the typo and changes all nodes to make the network a /24.  I expected I’d be back to work soon.  Alas, it got much more interesting.

Some useful commands to remember:

# show network neighbor
ip neigh show
# clear arp table
ip neigh flush all
# allow system to share arp table with neighbors
sysctl -w net.ipv4.conf.all.proxy_arp=1
# but probably better on a per interface basis (don’t want to leak info that your system could be use as a bridge
sysctl -w net.ipv4.conf.<interface>.proxy_arp=1
# arping
arping -I <interface> <target ip address>
# turn on promiscuous mode
ip link set <interface> promisc on
# tcpdump for just arp traffic
tcpdump -I <interface> arp
# check network interface statistics
ethtool -S 

Second: The DGX still doesn’t see arp response

I could see arp one way, but not the other.  Tcpdump running on both sides of the DGX and Proxmox, I could see nothing on DGX and I see receive and response activity on the Proxmox.  Lots of wasted time checking on firewalls.  Plus, Proxmox uses a virtual bridge, so it adds another variable to the equation.  As I was getting to lower layers on the DGX to understand where the miss is happening, luckily the internet found the answer.

On the ASUS GX10 platform, the network adapter runs an out-of-tree Realtek driver.  As the system runs strict memory virtualization, the NVIDIA SMMU (System MMU / IOMMU) treats incoming Layer 2 network broadcasts as suspicious memory access.  Nvidia’s ubuntu was silently isolating the network card’s shared DMA memory pools, dumping raw discovery broadcasts into a black hole.  

The solution: I created /etc/default/grub.d/iommu.cfg and added `GRUB_CMDLINE_LINUX=”$GRUB_CMDLINE_LINUX iommu.passthrough=1″`   This forced the SMMU to map to memory the card’s broadcast rings and fix arp.

# manually set a static neighbor MAC address
ip neigh replace <neighbor IP> lladdr <neighbor MAC> dev <network interface>
# check ring paramters for network interface
ethtool -g <interface>
# if you didn’t know, remove a network interface
ip addr del <ipaddres>/<CIDR> dev <device>

Third: DMA Ring Meltdown

Ok, everyone on the storage network now has layer 1-4 working as expected right from boot time.  However, the Proxmox system is still tragically slow for an isolated network using jumbo frames.  I moved to testing the network with iperf3 via udp.  Jitter got up in to 5ms and 59% loss.  I know the network is fine because the other two members of the network are BLAZING fast.  It’s just the Proxmox system.

As I worked my way, moving the levers and dials of the Debian Proxmox system, I was moving into smaller and smaller dials, but not making progress until I saw “InDroppedDma”.  I didn’t know what that meant.  This was the clue that lead to a deep read on how the network hardware and drivers work.

The culprit is the Aquantia AQC113 10GbE network card on the proxmox system.  It’s using an older driver and there aren’t many counters on this driver but there is InDroppedDma.  What that means is incoming packets were successfully received by the physical network card over the wire, but they were dropped before they could be successfully copied into the Proxmox’s system memory.  This is a bit like when software modems came out (if you remember modems).  This card has a “host- dependent” chipset.  In other words, it needs the PCIe bus, the CPU and system memory to operate.

The card on the Proxmox system doesn’t have the power to do things on its own, and can’t really handle via system resources the flood coming at it.  The x86-64 architecture uses 4KB pages sizes (memory blocks), but my ARM system uses 64KB page sizes and this can affect networking.  The card is too busy packet chunking.  The card’s response to getting overwhelmed is to flood the network back with pause frames.  The unmanaged switch acts the same as direct connect.  There is no means on the switch to filter pause frames.  The other members of the networks have better chipsets and can mitigate, and handle the storm and choke their speeds down to accommodate, explaining the transmit ring stall.

In this case, the packets are rejected even before the OS can see them.  You will not see the traffic reflected in system CPU or memory usage.

Since the driver counters (ethtool -S ) only give a clue, I fell down another rabbit hole to learn how to see this storm.  To physically see the raw frames traversing the wire, you can use tcpdump.  However, standard network drivers strip out hardware pause frames before passing them to the Linux packet analyzer.  One must explicitly instruct tcpdump to sniff the raw, unmapped Ethernet Control MAC pseudo-address or use the specific Layer 2 EtherType protocol code.  I didn’t know this until about 4am in the morning, but IEEE 802.3x Flow Control pause frames always use a standardized multicast destination MAC address (01:80:c2:00:00:01).  They also use EtherType hex code (0x8808).

Side story

Long ago, I was troubleshooting a problem with a Veritas cluster that was unstable.  As I received it, instead of the standard practice of buying a cheap switch to physically isolate the heartbeat network, they chose to plug it into the catalyst and put them on a separate vlan.  In the end, this was source of the problem.  However, along the way, to troubleshoot, we plugged into a solaris server into that same network so we could run snoop.  Yes, this was a long time ago.  Snoop is the precursor to tcpdump.  As we were looking at the datagrams, I was getting distracted and forgot these aren’t packets.  I kept seeing 0xCAFE.  Is this some sort of new Java networking thing?  What is this?  Eventually I realized – it’s a hex number describing the unique frame type Veritas used for the network heartbeat – and (at least at the time) the catalyst would drop it some of the time.  Eventually, we were allowed to get an old 10Mb switch from the junk closet and fixed the problem.

Note, much of the time I was distracted with using and not using the Proxmox virtual bridge.  In this case, it didn’t matter.  The packets were getting dropped by the card, not the OS’es TCP stack.  Host- dependent network chipset just cannot handle the packets built on 64KB page size to put into 4KB page x86-64 system.

# to see the pause storm for yourself
tcpdump -i <interface> -nn -e ether dst 01:80:c2:00:00:01
# or use the frame type to see
tcpdump -i <interface> -nn -e ether proto 0x8808
# iperf3 bandwidth test, client, target, udp, 1Gb bandwith, 10 parallel streams
iperf3 -c 192.168.200.200 -u -b 1g -P 10 

Solution

            Get a card with silicon that can handle the speed.  I know the card SAYS it’s 10Gb and it can negotiate 10Gb – but it can’t handle it.  It’s a bit like having a Dodge Challenger Hellcat, and putting spare tire on all four wheels.  I put in an order for an Intel X550-T2 10GbE card.  What is ironic is I never noticed until now.  This is because my previous VMs were on iSCSI and they were tiny and undemanding.  This all goes back to when I re-arranged my storage.  I wanted to improve performance on my NAS, so I added M.2 solid state drives and I wanted them for both read and write caching.  On the synology, I can only attach the SSD cache to one volume.  Thus, I moved live traffic (not backup or archival or media) to the same volume.  I got into this performance mess by investing in improving performance.

            That was my adventure.  If you read this far, cheers and condolence to you!  This and my entire blog is really about capturing some notes to share and help each other a bit.  Thank you to the bits of the internet that index and share these little obscure corners of the internet that help with these arcane problems.