Network Speed Discrepancy

[undaunt]

I'm trying to ascertain why my primary docker jail has seemingly poor network speed versus two other jails. Here are some speedtests. Morty is my docker host, the other two are only running adguard home locally. Morty is running adguard home in docker. All 3 are listed as DNS servers for all clients on my home network and resolve queries with dig.

root@rick[/var/empty]# jlmkr exec birdperson speedtest
Retrieving speedtest.net configuration...
Testing from Comcast Cable (xxx)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Ridge Wireless (Cupertino, CA) [61.41 km]: 14.116 ms
Testing download speed................................................................................
Download: 62.02 Mbit/s
Testing upload speed......................................................................................................
Upload: 22.11 Mbit/s
root@rick[/var/empty]# jlmkr exec mr-meeseeks speedtest
Retrieving speedtest.net configuration...
Testing from Comcast Cable (xxx)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Wave (San Francisco, CA) [1.94 km]: 14.058 ms
Testing download speed................................................................................
Download: 60.84 Mbit/s
Testing upload speed......................................................................................................
Upload: 22.30 Mbit/s
root@rick[/mnt/enterprise/jailmaker/jails]# jlmkr exec morty speedtest
Retrieving speedtest.net configuration...
Testing from Comcast Cable (xxx)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Pigs Can Fly Labs LLC (San Francisco, CA) [1.94 km]: 71.54 ms
Testing download speed................................................................................
Download: 16.43 Mbit/s
Testing upload speed......................................................................................................
Upload: 4.81 Mbit/s
# The next one was run with seccomp=0 for comparison
root@rick[/var/empty]# jlmkr exec morty speedtest      
Retrieving speedtest.net configuration...
Testing from Comcast Cable (xxx)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Tekify Fiber & Wireless (Fremont, CA) [46.83 km]: 15.451 ms
Testing download speed................................................................................
Download: 5.70 Mbit/s
Testing upload speed......................................................................................................
Upload: 4.99 Mbit/s

For reference, my non TrueNAS server (baremetal Ubuntu 22.04 LTS) gets the following results:

undaunt@morty:~$ speedtest
Retrieving speedtest.net configuration...
Testing from Comcast Cable (x)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Ridge Wireless (Cupertino, CA) [61.41 km]: 14.952 ms
Testing download speed................................................................................
Download: 312.65 Mbit/s
Testing upload speed......................................................................................................
Upload: 33.64 Mbit/s

I'd like to see all three perform faster, but the Morty jail is particularly ill equipped to perform right now.

Here are my templates. All three are bridge networking to br0 as opposed to macvlan.

morty

startup=1
gpu_passthrough_intel=0
gpu_passthrough_nvidia=0
# Turning off seccomp filtering improves performance at the expense of security
seccomp=1

# Network and mount configuration for the container
systemd_nspawn_user_args=--network-bridge=br0
        --resolv-conf=bind-host
        --system-call-filter='add_key keyctl bpf'
        --bind='/mnt/enterprise/username:/mnt/username
# Additional binds snipped for brevity but there are 22, if that matters.

# Script to run on the HOST before starting the jail
pre_start_hook=#!/usr/bin/bash
        set -euo pipefail
        echo 'PRE_START_HOOK'
        echo 1 > /proc/sys/net/ipv4/ip_forward
        modprobe br_netfilter
        echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
        echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables

# Only used while creating the jail
distro=debian
release=bookworm

# Install docker inside the jail and other initial setup including environment settings
initial_setup=#!/usr/bin/bash
        set -euo pipefail

        # Perform system updates and install essential packages
        apt-get update
        apt-get -y install ca-certificates curl nano screen wget openssh-server lsof rsync dnsutils python3

        # Docker installation
        install -m 0755 -d /etc/apt/keyrings
        curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
        chmod a+r /etc/apt/keyrings/docker.asc

        echo \
        "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
        $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
        tee /etc/apt/sources.list.d/docker.list > /dev/null
        apt-get update
        apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

        # Define the user and home directory
        USER_NAME="undaunt"
        HOME_DIR="/home/${USER_NAME}"

        # User setup
        useradd -u 1000 -m -s /bin/bash ${USER_NAME}
        if [ $(getent group docker) ]; then
        usermod -aG docker ${USER_NAME}
        fi
        usermod -aG sudo ${USER_NAME}

        # Sudoers setup
        echo "${USER_NAME} ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/${USER_NAME}
        chmod 0440 /etc/sudoers.d/${USER_NAME}

# systemd-nspawn specific default arguments
systemd_run_default_args=--property=KillMode=mixed
        --property=Type=notify
        --property=RestartForceExitStatus=133
        --property=SuccessExitStatus=133
        --property=Delegate=yes
        --property=TasksMax=infinity
        --collect
        --setenv=SYSTEMD_NSPAWN_LOCK=0

systemd_nspawn_default_args=--keep-unit
        --quiet
        --boot
        --bind-ro=/sys/module
        --inaccessible=/sys/module/apparmor

birdperson / mr-meeseeks

startup=1
gpu_passthrough_intel=0
gpu_passthrough_nvidia=0
# Turning off seccomp filtering improves performance at the expense of security
seccomp=1

# Use macvlan networking to provide an isolated network namespace,
# so docker can manage firewall rules
# Alternatively use --network-bridge=br1 instead of --network-macvlan
# Ensure to change eno1/br1 to the interface name you want to use
# You may want to add additional options here, e.g. bind mounts
systemd_nspawn_user_args=--network-bridge=br0
        --resolv-conf=bind-host

# Script to run on the HOST before starting the jail
# Load kernel module and config kernel settings required for docker
pre_start_hook=#!/usr/bin/bash
        set -euo pipefail
        echo 'PRE_START_HOOK'
        echo 1 > /proc/sys/net/ipv4/ip_forward
        modprobe br_netfilter
        echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
        echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables

# Only used while creating the jail
distro=debian
release=bookworm

initial_setup=#!/bin/bash
        set -euo pipefail

        apt-get update && apt-get -y install ca-certificates curl
        apt-get install -y nano screen curl wget openssh-server

        USER_NAME="undaunt"
        useradd -u 1000 -m -s /bin/bash $USER_NAME
        usermod -aG sudo $USER_NAME

        # Install AdGuard Home
        curl -s -S -L https://raw.githubusercontent.com/AdguardTeam/AdGuardHome/master/scripts/install.sh | sh -s -- -v

        # Create and configure AdGuard Home DNS configuration
        mkdir -p /etc/systemd/resolved.conf.d
        touch /etc/systemd/resolved.conf.d/adguardhome.conf
        echo "[Resolve]" >> /etc/systemd/resolved.conf.d/adguardhome.conf
        echo "DNS=127.0.0.1" >> /etc/systemd/resolved.conf.d/adguardhome.conf
        echo "DNSStubListener=no" >> /etc/systemd/resolved.conf.d/adguardhome.conf

        # Disable systemd-resolved and configure DNS resolution
        mv /etc/resolv.conf /etc/resolv.conf.backup
        ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf

# You generally will not need to change the options below
systemd_run_default_args=--property=KillMode=mixed
        --property=Type=notify
        --property=RestartForceExitStatus=133
        --property=SuccessExitStatus=133
        --property=Delegate=yes
        --property=TasksMax=infinity
        --collect
        --setenv=SYSTEMD_NSPAWN_LOCK=0

systemd_nspawn_default_args=--keep-unit
        --quiet
        --boot
        --bind-ro=/sys/module
        --inaccessible=/sys/module/apparmor

Thanks for any insight!

[undaunt]

I did some testing with blank/standard templates, macvlan vs bridge vs host. I even killed my bridge and my bond and tested macvlan directly with my two NICs. I thought it could've been an LACP issue.

I then swapped my DNS servers to cloudflare and immediately got 600+ mbps in the main docker jail. I then rebuilt my br0, same result. I'm now reconfiguring LACP and going to nest bond0 below br0 to test. If it's the same result, I may just force TrueNAS and the jails to use Cloudflare DNS. I have 3 Adguard home DNS servers running in jails and my other computers/devices at home have no issue, nor does my baremetal host, so perhaps there is some odd interplay between the jails networking and DNS that is causing this sub-10 Mbps issue.

[undaunt]

Having the host point to Cloudflare and force the jails to use local Adguard DNS allows me to maintain good speeds while letting my local containers within the docker host jails resolve my private FQDN split-DNS records.

I'm not sure why this is an issue but I've worked around it sufficiently, I think. As far as I can tell it's not related to LACP and the issue appears on any of the 3 levels of binding (eth0/bond0/br0) with Macvlan as well as br0 with bridge.

[Lockszmith-GH]

See if you can test another network interface, a different manufacturer, might be a driver issue of your NIC.

[undaunt]

Same results on a different machine also running Scale, Intel vs Supermicro.

[Jip-Hop]

If you have the means and want to get to the bottom of this you could copy the jails directory (with rsync or creating a tar file) while preserving ownership and permissions to a vanilla debian 12 server (preferably on the same hardware you already tested). Then run the jails using plain systemd-run and systemd-nspawn commands (the command to use is shown by jailmaker when you start the jail, so make note of this also before copying). By taking jailmaker and TrueNAS SCALE out of the equation we can better pinpoint the issue. If it also happens there you may even try a distro with the latest version of systemd... When running into this problem still you may want to open an issue in the systemd GitHub repo.