[BUG] Address resolution failed while creating pot with flavours

[Jomy10]

Describe the bug
I can't create a pot using flavours. Creating a pot normally does work.

To Reproduce
Consider these files:

• test

set-attr -A no-rc-script -V YES
set-attr -A persisten -V NO
set-rss -C 1

• test-cmd

set-cmd -c "python -m http.server"

• test.sh

#!/bin/sh

[ -w /etc/pkg/FreeBSD.conf ] && sed -i '' 's/quarterly/latest/' /etc/pkg/FreeBSD.conf
ASSUME_ALWAYS_YES=yes pkg bootstrap
touch /etc/rc.conf
sysrc sendmail_enable="NONE"
pkg install -y python
pkg clean -y

Now running pot create -p test1 -b 14.1 -N public-bridge -t single -f test -f test-cmd will result in the following error message:

Mon Sep 23 15:59:40 UTC 2024
test.sh -> /opt/pot/jails/test1/m/tmp/test.sh
Bootstrapping pkg from pkg+https://pkg.FreeBSD.org/FreeBSD:14:amd64/latest, please wait...
pkg: Error fetching https://pkg.FreeBSD.org/FreeBSD:14:amd64/latest/Latest/pkg.txz: Host does not resolve
Address resolution failed for https://pkg.FreeBSD.org/FreeBSD:14:amd64/latest.
Consider changing PACKAGESITE.
sendmail_enable: NONE -> NONE
Bootstrapping pkg from pkg+https://pkg.FreeBSD.org/FreeBSD:14:amd64/latest, please wait...
pkg: Error fetching https://pkg.FreeBSD.org/FreeBSD:14:amd64/latest/Latest/pkg.txz: Host does not resolve
Address resolution failed for https://pkg.FreeBSD.org/FreeBSD:14:amd64/latest.
Consider changing PACKAGESITE.
Bootstrapping pkg from pkg+https://pkg.FreeBSD.org/FreeBSD:14:amd64/latest, please wait...
pkg: Error fetching https://pkg.FreeBSD.org/FreeBSD:14:amd64/latest/Latest/pkg.txz: Host does not resolve
Address resolution failed for https://pkg.FreeBSD.org/FreeBSD:14:amd64/latest.
Consider changing PACKAGESITE.
###>  create: flavour test failed (script)

Expected behavior
Not expecting any errors.

System configuration - if possible

• /usr/local/etc/pot/pot.conf

# pot configuration file

# All datasets related to pot use the some zfs dataset as parent
# With this variable, you can choose which dataset has to be used
POT_ZFS_ROOT=sys/pot

# It is also important to know where the root dataset is mounted
POT_FS_ROOT=/opt/pot

# This is the cache used to import/export pots
POT_CACHE=/var/cache/pot

# This is where pot is going to store temporary files
POT_TMP=/tmp

# This is the group owning POT_FS_ROOT
# POT_GROUP=pot

# This is the suffix added to temporary files created using mktemp,
# X is a placeholder for a random character, see mktemp(1)
# POT_MKTEMP_SUFFIX=.XXXXXXXX

# Define the max length of the hostname inside the pot
# POT_HOSTNAME_MAX_LENGTH=64

# Internal Virtual Network configuration

# IPv4 Internal Virtual network
# POT_NETWORK=10.192.0.0/10

# Internal Virtual Network netmask
# POT_NETMASK=255.192.0.0

# The default gateway of the Internal Virtual Network
# POT_GATEWAY=10.192.0.1

# The name of the network physical interface, to be used as default gateway
POT_EXTIF=vtnet0

# The list of extra network interface, to make other network segments accessible
# POT_EXTRA_EXTIF=vlan20 vlan50
# for each extra interface, a variable is used to sepcify its network segment
# POT_NETWORK_vlan20=192.168.100.0/24
# POT_NETWORK_vlan50=10.50.50.0/24

# Do not allow bridge-based pots to forward traffic to each other
# POT_ISOLATE_VNET_POTS=true

# DNS on the Internal Virtual Network

# name of the pot running the DNS
# POT_DNS_NAME=dns

# IP of the DNS
# POT_DNS_IP=10.192.0.2

# VPN support

# name of the tunnel network interface
# POT_VPN_EXTIF=tun0
# POT_VPN_NETWORKS=192.168.0.0/24 192.168.10.0/24

• System

```                        `       [email protected]
  ` `.....---.......--.```   -/    -------------
  +o   .--`         /y:`      +.   OS: FreeBSD 14.1-RELEASE amd64
   yo`:.            :o      `+-    Uptime: 5 days, 21 hours, 39 mins
    y/               -/`   -o/     Packages: 128 (pkg)
   .-                  ::/sy+:.    Shell: zsh 5.9
   /                     `--  /    Terminal: /dev/pts/0
  `:                          :`   CPU: Intel Xeon (Skylake, IBRS, no TSX) (2) @ 2.294GHz
  `:                          :`   GPU: Virtio 1.0 GPU
   /                          /    Memory: 3571MiB / 3957MiB
   .-                        -.
    --                      -.
     `:`                  `:`
       .--             `--.
          .---.....----.

This is an instance running on a Hetzner VPS.

** If network related **

• cat /etc/pf.conf

# Firewall
# /etc/pf.conf
# vim: set ft=pf

##########
# Macros #
##########

### PODMAN ###
# Change these to the interface(s) with the default route
v4egress_if = "vtnet0"
v6egress_if = "vtnet0"
#v4egress_if = "ix0"
#v6egress_if = "ix0"
### END PODMAN ###

## Set public interface ##
ext_if = "vtnet0"

## Set server public IP address
ext_if_ip = "<server-ip>"

## Set and drop IP ranges on public interface ##
martians = "{ 127.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, \
	      10.0.0.0/8, 169.254.0.0/16, 192.0.2.0/24, \
	      0.0.0.0/8, 240.0.0.0/4 }"

## Set http (80) and https (443) ports ##
webports = "{http, https}"

## enable services ##
int_tcp_services = "{domain, ntp, smtp, www, https, ftp, ssh}" # mail also goes here
int_udp_services = "{domain, ntp}"

##########
# Tables #
##########

### PODMAN ###
table <cni-nat>
### END PODMAND ###

###########
# Options #
###########

## Skip loop back interface ##
set skip on lo

## Sets the interface for which PF should gather statistics such as bytes in/out and packets passed/blocked ##
set loginterface $ext_if

######################
# Ethernet filtering #
######################

#########################
# Traffic normalization #
#########################

# Deal with attacks based on incorrect handling of packet fragments
scrub in all

############
# Queueing #
############

###############
# Translation #
###############

### PODMAN ###
nat on $v4egress_if inet from <cni-nat> to any -> ($v4egress_if)
nat on $v6egress_if inet6 from <cni-nat> to !ff00::/8 -> ($v6egress_if)

rdr-anchor "cni-rdr/*"
nat-anchor "cni-rdr/*"
### END PODMAN ###

### POT ###
nat-anchor pot-nat
rdr-anchor "pot-rdr/*"
### END POT ###

####################
# Packet Filtering #
####################

# action [direction] [log] [quick] [on interface] [af] [proto protocol]
#        [from src_addr [port src_port]] [to dst_addr [port dst_port]]
#        [flags tcp_flags] [state]

## Set default policy ##
block return in log all
block out all

# Drop all Non-Routable Addresses
block drop in quick on $ext_if from $martians to any
block drop out quick on $ext_if from any to $martians

## Blocking spoofed packets
antispoof quick for $ext_if

# Allow SSH from any IP address
pass in inet proto tcp to $ext_if port ssh

# allow tcp and udp traffic from 10.88.0.0/16
pass in inet proto tcp from 10.88.0.0/16 to $ext_if
pass in inet proto udp from 10.88.0.0/16 to $ext_if

# Allow Ping-Pong stuff. Be a good sysadmin
pass inet proto icmp icmp-type echoreq

# All access to our Nginx/Apache/Lighttpd Webserver ports
pass proto tcp from any to $ext_if port $webports

# Allow essential outgoing traffic
pass out quick on $ext_if proto tcp to any port $int_tcp_services
pass out quick on $ext_if proto udp to any port $int_udp_services

# Podman dns
pass in on cni-podman0

• potnet show -v

16:09:46 [ INFO] Insert network 10.192.0.0/10
16:09:46 [ INFO] Insert broadcast 10.192.0.0/10
16:09:46 [ INFO] Insert gateway 10.192.0.1
16:09:46 [ INFO] Insert dns 10.192.0.2
Network topology:
	network : 10.192.0.0/10
	min addr: 10.192.0.0
	max addr: 10.255.255.255

Addresses already taken:
	10.192.0.0
	10.192.0.1	default gateway
	10.192.0.2	dns
	10.255.255.255

Debug information
PotSystemConfig {
    zfs_root: "sys/pot",
    fs_root: "/opt/pot",
    network: 10.192.0.0/10,
    netmask: 255.192.0.0,
    gateway: 10.192.0.1,
    ext_if: "vtnet0",
    dns: Some(
        PotDnsConfig {
            pot_name: "dns",
            ip: 10.192.0.2,
        },
    ),
}

Additional context
The system is running on a Hetzner VPS.

[grembo]

Hi,

What‘s in your jailhosts /etc/resolv.conf?

By default, pot inherits this setting from the jailhost. You can override it, see pot help create (or pot create help?).

I see that you have pot anchors in your pf.conf, but didn’t study it in detail if there might be a nat issue. So if the dns setting doesn’t solve it, place something like sleep 1000 in the flavor script and then jexec into the jail to debug nat (run something like host google.com 8.8.8.8 and seeing that works).

[Jomy10]

@grembo /etc/resolv.conf is:

# Generated by resolvconf
nameserver 185.12.64.1
nameserver 185.12.64.2

The weird thing is that creating a pot without a flavour does work (for instance pot create -p test -b 14.1 -t single). When I go into the pot, I can install packages, so internet access does work.

I will see if I can debug any possible nat issues.

[Jomy10]

I went into the jail as you suggested. I looked at the pf logs and saw a block on bridge1. So I added pass in on bridge1 to pf.conf. Now it works.

[grembo]

Based on your description it seems like no NAT rules are placed in the anchor while creating a flavor. This could be considered both a bug or a feature - it's certainly unexpected and should be addressed IMHO. Probably would make sense to have a flag to control behavior.