On a previous episode, I found odd behavior and after spending so much time and money on a good storage network to support my hypervisor and nvidia sparc, I was getting 1996 speeds.  Ultimately, the TP-link TX401 couldn’t do it.  Here’s what happen tonight, on a very special episode of 10Gb Joy.

As a reminder, here’s the network layout with problems:

What I did tonight:

  • Replace the 10Gb nic with grown-up Intel X550T2 (used, because it’s a home lab)
  • Check the ring buffers on all the cards on this unmanaged 10Gb switch
  • Set on systems where I could rx and tx ring buffers to 4096

On the big boy Intel card, it can have rx and tx size of 32768.  This is 32,768 descriptors, each of which is a memory to hold a frame or packet.  The equivalent on my intel mac is 256rx/128tx.  The default was 512.  I made it match the Synology with 4096.  No more pause frames.  Searching the internet seemed to have a consensus on 10Gb and jumbo frames, 4096 ring buffers because it’s enough to handle 10Gb bursts, but not so much to create buffer bloat latency.

Consistent iperf3 tests now.  Screaming on TCP.  UDP was slow – just UDP and just from proxmox to storage.  Turn on -P 4 for iperf3 and it’s fine.  There’s a single threaded thing going on with UDP.  I didn’t dig into it as I’ll have tcp enabled for NFS anyway because of the wicked good TCP offload engine.  Plus, the synology has a 400GB R/W M.2 SSD cache and 32gb of ram.  It’s fast enough.

Here’s the mount options I put in /etc/pve/storage.cfg to maximize the nfs storage for the proxmox VMs

options vers=3,hard,proto=tcp,nconnect=4,rsize=131072,wsize=131072,noatime

I then used fio to test random IO and sequential.  I did the tests again with with 2 streams of 16GB to really prove it.

What are the results?

  • Sustained Sequential Write (918 MB/s / 876 MiB/s): Writing 32 GB continuously across two parallel streams averaged 918 MB/s, peaking at 1.25 GB/s (1,284,096 KiB/s) before settling into the raw sustained write speed of your hard drive RAID pool.
  • Sustained Sequential Read (698 MB/s / 666 MiB/s): Reading all 32 GB back straight from the spinning platters averaged 698 MB/s with zero cache assistance. For a multi-drive spinning RAID array over NFS, ~700 MB/s sustained uncached read and ~920 MB/s sustained write is top-tier mechanical disk performance.
  • End-to-End Bottleneck Eliminated: Before the Intel X550-T2 upgrade and nfs nconnect=4 tuning, the network throttled the array well below its physical drive speeds. Now the 10Gb network gives the Synology’s hard drives and 400 GB NVMe cache full headroom to run at their physical maximums.

Lessons Learned

  • Don’t buy or use a TP-Link TX401 for 10Gb
    • It can probably survive 5Gb and thrive at 2.5Gb
  • 10Gb is still edgy for prosumers
  • You can’t skimp on network cards for 10Gb
  • At least the TP-Link TL-SX105 seems solid.

Ok.  I fixed the foundation, so I can get on to other lovely projects. 🙂

Here is the after diagram: