USB passthrough to jail - how to?

[sakcaj]

I've followed the set-up guide by Stux and got the jlmkr configured with dockage, got one container running after a bit of back and forth - learning curve of linux... my another container to migrate from Charts is Zigbee2MQTT (z2m) - I got it running but not in a consistent way and I cannot figure out what's causing the issue - it's the USB Zigbee stick mounting point that's not always present causing the container to fail at startup. This was not a problem with TrueCharts app - it was hardrock stable thus I do not believe this has anything to do with the stick itself, USB port etc.

Error from z2m container's terminal:

z2m  | Using '/app/data' as data directory
z2m  | [2024-06-14 20:40:02] info:      z2m: Logging to console, file (filename: log.log)
z2m  | [2024-06-14 20:40:02] info:      z2m: Starting Zigbee2MQTT version 1.38.0 (commit #fe048e6)
z2m  | [2024-06-14 20:40:02] info:      z2m: Starting zigbee-herdsman (0.49.2)
z2m  | [2024-06-14 20:40:02] error:     zh:zstack:znp: Failed to determine if path is valid: 'Error: ENOENT: no such file or directory, lstat '/dev/ttyUSB0''
z2m  | [2024-06-14 20:40:02] info:      zh:zstack:znp: Opening SerialPort with {"path":"/dev/ttyUSB0","baudRate":115200,"rtscts":false,"autoOpen":false}
z2m  | [2024-06-14 20:40:02] error:     z2m: Error while starting zigbee-herdsman
z2m  | [2024-06-14 20:40:02] error:     z2m: Failed to start zigbee
z2m  | [2024-06-14 20:40:02] error:     z2m: Check https://www.zigbee2mqtt.io/guide/installation/20_zigbee2mqtt-fails-to-start.html for possible solutions
z2m  | [2024-06-14 20:40:02] error:     z2m: Exiting...
z2m  | [2024-06-14 20:40:02] error:     z2m: Error: Error: No such file or directory, cannot open /dev/ttyUSB0
z2m exited with code 0

I've been troubleshooting this for a few days, changing various settings in the the jail docker config, re-plugging my stick at various points, setting up symlinks etc and cannot figure it out.

Here's my jlmkr docker config:

startup=1
gpu_passthrough_intel=0
gpu_passthrough_nvidia=0
# Turning off seccomp filtering improves performance at the expense of security
seccomp=0 #set to 0 to take it out of the picture for now
docker_compatible=1 #found in a similar thread which I cannot locate now, does not seem to make any difference
# Use macvlan networking to provide an isolated network namespace,
# so docker can manage firewall rules
# Alternatively use --network-macvlan=eno1 instead of --network-bridge
# Ensure to change eno1/br1 to the interface name you want to use
# You may want to add additional options here, e.g. bind mounts
systemd_nspawn_user_args=
        --network-bridge=br01
        --resolv-conf=bind-host
        --system-call-filter='add_key keyctl bpf'
        --capability=all #found in a similar thread which I cannot locate now, does not seem to make any difference
        --bind-ro=/run/udev
        --bind='/mnt/SSD Storage/docker/data:/mnt/data'
        --bind='/mnt/SSD Storage/docker/stacks:/opt/stacks'
        --bind=/dev/ttyUSB0
        --bind='/mnt/HDD/HDD_CAMS_SMB:/mnt/media'

Here's my z2m compose:

version: "3.8"
services:
  z2m:
    container_name: z2m
    image: koenkk/zigbee2mqtt
    restart: unless-stopped
    group_add:
      - dialout
    user: 0:1000
    volumes:
      - /mnt/data/z2m/data:/app/data
      - /run/udev:/run/udev:ro
    privileged: true
    ports:
      # Frontend port
      - 10103:8080
    environment:
      - TZ=Europe/Berlin
    devices:
      # Make sure this matched your adapter location
      - /dev/ttyUSB0:/dev/ttyUSB0
networks: {}

z2m config:

permit_join: false
mqtt:
  base_topic: zigbee2mqtt
  server: mqtt://192.168.0.176
  user: username
  password: password
serial:
  port: /dev/ttyUSB0
advanced:
  network_key:
(...)

Here's my USB stick, I do not use any other USB devices so it consistently appears on the same mount. You can see the permissions are extended, that's part of the troubleshooting I've done.

ls -la /dev/serial/by-id/usb-ITead_Sonoff_Zigbee_3.0_USB_Dongle_Plus_e053af01b919ec1193eb37cc47486eb0-if00-port0
lrwxrwxrwx 1 root root 13 Jun 14 18:09 /dev/serial/by-id/usb-ITead_Sonoff_Zigbee_3.0_USB_Dongle_Plus_e053af01b919ec1193eb37cc47486eb0-if00-port0 -> ../../ttyUSB0

So the issue that I think I see is lack of the tty devices being passed over to jail - is that the case? If so, how would I do that for the docker jail? Here's ls -la /dev/ in the docker jail

root@docker:~# ls -la /dev/
total 9
drwxr-xr-x  8 root root  460 Jun 14 20:38 .
drwxr-xr-x 17 root root   21 Jun  8 07:27 ..
drwxr-xr-x  2 root root  180 Jun 14 20:38 char
lrwxrwxrwx  1 root root    5 Jun 14 20:38 console -> pts/0
lrwxrwxrwx  1 root root   11 Jun 14 20:38 core -> /proc/kcore
lrwxrwxrwx  1 root root   13 Jun 14 20:38 fd -> /proc/self/fd
crw-rw-rw-  1 root root 1, 7 Jun 14 20:38 full
drwxr-xr-x  2 root root    0 Jun 14 20:38 hugepages
lrwxrwxrwx  1 root root   12 Jun 14 20:38 initctl -> /run/initctl
lrwxrwxrwx  1 root root   28 Jun 14 20:38 log -> /run/systemd/journal/dev-log
drwxrwxrwt  2 root root   40 Jun 14 20:38 mqueue
drwxr-xr-x  2 root root   60 Jun 14 20:38 net
crw-rw-rw-  1 root root 1, 3 Jun 14 20:38 null
lrwxrwxrwx  1 root root    8 Jun 14 20:38 ptmx -> pts/ptmx
drwxr-xr-x  2 root root    0 Jun 14 20:38 pts
crw-rw-rw-  1 root root 1, 8 Jun 14 20:38 random
drwxrwxrwt  2 root root   40 Jun 14 20:38 shm
lrwxrwxrwx  1 root root   15 Jun 14 20:38 stderr -> /proc/self/fd/2
lrwxrwxrwx  1 root root   15 Jun 14 20:38 stdin -> /proc/self/fd/0
lrwxrwxrwx  1 root root   15 Jun 14 20:38 stdout -> /proc/self/fd/1
crw-rw-rw-  1 root root 5, 0 Jun 14 20:38 tty
crw-rw-rw-  1 root root 1, 9 Jun 14 20:38 urandom
crw-rw-rw-  1 root root 1, 5 Jun 14 20:38 zero

I found similar thread but for iocage, where it seems there's a few settings to be set which allow for usb devices to appear under /dev/ - https://www.truenas.com/community/threads/access-usb-device-from-jail-state-of-the-art-method.107972/ so it is possible.

I do not see any errors in the jails logs:

root@truenas[~]# jlmkr log docker
Jun 09 09:37:05 truenas systemd[1]: Starting jlmkr-docker.service - My nspawn jail docker [created with jailmaker]...
Jun 09 09:37:05 truenas .ExecStartPre[1635456]: PRE_START_HOOK
Jun 09 09:37:05 truenas systemd-nspawn[1635458]: systemd 252.22-1~deb12u1 running in system mode (+PAM +AUDIT +SELINUX >
Jun 09 09:37:05 truenas systemd-nspawn[1635458]: Detected virtualization systemd-nspawn.
Jun 09 09:37:05 truenas systemd-nspawn[1635458]: Detected architecture x86-64.
Jun 09 09:37:05 truenas systemd-nspawn[1635458]: Detected first boot.
Jun 09 09:37:05 truenas systemd-nspawn[1635458]:
Jun 09 09:37:05 truenas systemd-nspawn[1635458]: Welcome to Debian GNU/Linux 12 (bookworm)!
Jun 09 09:37:05 truenas systemd-nspawn[1635458]:
Jun 09 09:37:05 truenas systemd-nspawn[1635458]: Initializing machine ID from container UUID.
Jun 09 09:37:05 truenas systemd-nspawn[1635458]: Populated /etc with preset unit settings.
Jun 09 09:37:05 truenas systemd-nspawn[1635458]: Queued start job for default target graphical.target.
Jun 09 09:37:05 truenas systemd-nspawn[1635458]: [  OK  ] Created slice system-getty.slice - Slice /system/getty.
Jun 09 09:37:05 truenas systemd-nspawn[1635458]: [  OK  ] Created slice system-modprobe.slice - Slice /system/modprobe.
Jun 09 09:37:05 truenas systemd-nspawn[1635458]: [  OK  ] Created slice user.slice - User and Session Slice.
Jun 09 09:37:05 truenas systemd-nspawn[1635458]: [  OK  ] Started systemd-ask-password-consol…quests to Console Directo>
Jun 09 09:37:05 truenas systemd-nspawn[1635458]: [  OK  ] Started systemd-ask-password-wall.p… Requests to Wall Directo>
Jun 09 09:37:05 truenas systemd-nspawn[1635458]: [  OK  ] Reached target cryptsetup.target - Local Encrypted Volumes.
Jun 09 09:37:05 truenas systemd-nspawn[1635458]: [  OK  ] Reached target integritysetup.targe…Local Integrity Protected>
Jun 09 09:37:05 truenas systemd-nspawn[1635458]: [  OK  ] Reached target paths.target - Path Units.
Jun 09 09:37:05 truenas systemd-nspawn[1635458]: [  OK  ] Reached target remote-cryptsetup.target - Remote Encrypted Vo>
Jun 09 09:37:05 truenas systemd-nspawn[1635458]: [  OK  ] Reached target remote-fs.target - Remote File Systems.
Jun 09 09:37:05 truenas systemd-nspawn[1635458]: [  OK  ] Reached target remote-veritysetup.t…- Remote Verity Protected>
Jun 09 09:37:05 truenas systemd-nspawn[1635458]: [  OK  ] Reached target slices.target - Slice Units.
Jun 09 09:37:05 truenas systemd-nspawn[1635458]: [  OK  ] Reached target swap.target - Swaps.
Jun 09 09:37:05 truenas systemd-nspawn[1635458]: [  OK  ] Reached target veritysetup.target - Local Verity Protected Vo>
Jun 09 09:37:05 truenas systemd-nspawn[1635458]: [  OK  ] Listening on systemd-initctl.socket… initctl Compatibility Na>
Jun 09 09:37:05 truenas systemd-nspawn[1635458]: [  OK  ] Listening on systemd-journald-dev-l…ocket - Journal Socket (/>
Jun 09 09:37:05 truenas systemd-nspawn[1635458]: [  OK  ] Listening on systemd-journald.socket - Journal Socket.

[sakcaj]

So I'm digging deeper and deeper into this rabbit hole and my assumption is that the devices is not visible within the jail due to devfs rules config. I've found few sources where similar config is applied, but each seems outdated as things like service devfs restart do not seem to work in TrueNAS Scale right now. Does anyone have the wits to advise what and where has to be applied to make it work? I can do the startup script once I know it works...

What I found: https://www.truenas.com/community/threads/apply-devfs-rules-in-a-jail.26589/ and https://forums.freebsd.org/threads/devfs-restart-causes-warning-mounting-usbs-is-awful-experience.76892/

Here's a guide which lists steps to create a script which re-creates devfs rule at startup as its lost during host reboot - which I think could be the issue I've seen i.e. along the way I've applied a rule which allowed for the device to appear in jail, was happy, started working on a next task and after one of the follow-up restarts noticed the passthrough is not working anymore, unfortunately I do not know what rule I've applied at that time...

[Jip-Hop]

Those articles don't apply to TrueNAS SCALE. SCALE is Linux, the articles you've linked are about FreeBSD. Jailmaker doesn't actually make jails, that's a FreeBSD thing. I misused the term jails, sorry for the confusion. I don't use USB devices on SCALE so I can't help you with your USB passthrough issue. Perhaps post your question in the forum too. I'm sure there are people using USB passthrough successfully with jailmaker.

[wisekki]

Just dropping my thoughts; I'm using USB passthrough for usb-zigbee device that is used by HA container and these two work nicely. I'm passing /dev/serial/by-id/device and from what I can see from your docker config the bind is not complete. Not sure if your paste was incomplete but maybe check that one out. My docker config is like this:
--bind /dev/serial/by-id/usb-Silicon_Labs_slae.sh_cc2652rb_stick_xxx:/dev/serial/by-id/usb-Silicon_Labs_slae.sh_cc2652rb_stick_xxx
and then inside the docker host I'm mapping this same /dev/serial to the HA container.

[sakcaj]

Hey, thanks for your time! The stick's id got somehow cut off during copy/paste from terminal. I've raised a topic on truenas forums as advised by @Jip-Hop - https://forums.truenas.com/t/linux-jails-sandboxes-containers-with-jailmaker/417/284?u=sakcaj lets try to talk there and if I finally figure it out I will update this thread.