[BUG] OpenShift Local setup fails in restricted network

[Emmettlsc]

General information

• OS: RHEL 8.6
• Hypervisor: KVM
• Did you run crc setup before starting it (Yes/No)? No (fails here)
• Running CRC on: Laptop

CRC version

CRC version: 2.31.0+6d23b6
OpenShift version: 4.14.7
Podman version: 4.4.4

CRC status

# Put `crc status --log-level debug` output here

CRC config

- consent-telemetry                     : no

Host Operating System

NAME="Red Hat Enterprise Linux"
VERSION="8.6 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.6"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.6"

Steps to reproduce

1. download crc_libvirt_4.14.7_amd64.crcbundle from OpenShift Mirror and place it where crc expects
2. run crc setup in a restricted network

Expected

crc setup to find the .crcbundle and extract it without the need to perform any downloads

Actual

crc setup reports that it cannot find the cached bundle, then tries and fails to download the bundle

Logs

$ crc setup --log-level debug
DEBU CRC version: 2.31.0+6d23b6
DEBU OpenShift version: 4.14.7
DEBU Podman version: 4.4.4
DEBU Running 'crc setup'
INFO Using bundle path /home/tim/.crc/cache/crc_libvirt_4.14.7_amd64.crcbundle
DEBU Checking if systemd-resolved.service is running
DEBU Running 'systemctl status systemd-resolved.service'
DEBU Command failed: exit status 3
DEBU stdout: * systemd-resolved.service - Network Name Resolution
   Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:systemd-resolved.service(8)
           https://www.freedesktop.org/wiki/Software/systemd/resolved
           https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
           https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
DEBU stderr:
INFO Checking if running as non-root
INFO Checking if running inside WSL2
INFO Checking if crc-admin-helper executable is cached
DEBU Running '/home/tim/.crc/bin/crc-admin-helper-linux --version'
DEBU Found crc-admin-helper-linux version 0.5.2
DEBU crc-admin-helper executable already cached
INFO Checking if running on a supported CPU architecture
DEBU GOARCH is amd64 GOOS is linux
INFO Checking if crc executable symlink exists
INFO Checking minimum RAM requirements
DEBU Total memory of system is 134096687104 bytes
INFO Checking if Virtualization is enabled
DEBU Checking if the vmx/svm flags are present in /proc/cpuinfo
DEBU CPU virtualization flags are good
INFO Checking if KVM is enabled
DEBU Checking if /dev/kvm exists
DEBU /dev/kvm was found
INFO Checking if libvirt is installed
DEBU Checking if 'virsh' is available
DEBU 'virsh' was found in /usr/bin/virsh
DEBU Checking 'virsh capabilities' for libvirtd/qemu availability
DEBU Running 'virsh --readonly --connect qemu:///system capabilities'
DEBU Found x86_64 hypervisor with 'hvm' capabilities
INFO Checking if user is part of libvirt group
DEBU Checking if current user is part of the libvirt group
DEBU Current user is already in the libvirt group
INFO Checking if active user/process is currently part of the libvirt group
INFO Checking if libvirt daemon is running
DEBU Checking if libvirtd service is running
DEBU Running 'systemctl status virtqemud.socket'
DEBU Command failed: exit status 3
DEBU stdout: * virtqemud.socket - Libvirt qemu local socket
   Loaded: loaded (/usr/lib/systemd/system/virtqemud.socket; disabled; vendor preset: disabled)
   Active: inactive (dead)
   Listen: /run/libvirt/virtqemud-sock (Stream)
DEBU stderr:
DEBU virtqemud.socket is neither running nor listening
DEBU Running 'systemctl status libvirtd.socket'
DEBU libvirtd.socket is running
INFO Checking if a supported libvirt version is installed
DEBU Checking if libvirt version is >=3.4.0
DEBU Running 'virsh -v'
INFO Checking if crc-driver-libvirt is installed
DEBU Checking if crc-driver-libvirt is installed
DEBU Running '/home/tim/.crc/bin/crc-driver-libvirt version'
DEBU Found crc-driver-libvirt version 0.13.6
DEBU crc-driver-libvirt is already installed
INFO Checking crc daemon systemd service
DEBU Checking crc daemon systemd service
DEBU Checking if crc-daemon.service is running
DEBU Running 'systemctl --user status crc-daemon.service'
DEBU Command failed: exit status 3
DEBU stdout: * crc-daemon.service - CRC daemon
   Loaded: loaded (/home/tim/.config/systemd/user/crc-daemon.service; static; vendor preset: enabled)
   Active: inactive (dead)
DEBU stderr:
DEBU crc-daemon.service is neither running nor listening
DEBU Checking if crc-daemon.service has the expected content
INFO Checking crc daemon systemd socket units
DEBU Checking crc daemon systemd socket units
DEBU Checking if crc-http.socket is running
DEBU Running 'systemctl --user status crc-http.socket'
DEBU crc-http.socket is listening
DEBU Checking if crc-http.socket has the expected content
DEBU Checking if crc-vsock.socket is running
DEBU Running 'systemctl --user status crc-vsock.socket'
DEBU crc-vsock.socket is listening
DEBU Checking if crc-vsock.socket has the expected content
INFO Checking if systemd-networkd is running
DEBU Checking if systemd-networkd.service is running
DEBU Running 'systemctl status systemd-networkd.service'
DEBU Command failed: exit status 4
DEBU stdout:
DEBU stderr: Unit systemd-networkd.service could not be found.
DEBU systemd-networkd.service is not running
INFO Checking if NetworkManager is installed
DEBU Checking if 'nmcli' is available
DEBU 'nmcli' was found in /usr/bin/nmcli
INFO Checking if NetworkManager service is running
DEBU Checking if NetworkManager.service is running
DEBU Running 'systemctl status NetworkManager.service'
DEBU NetworkManager.service is already running
INFO Checking if /etc/NetworkManager/conf.d/crc-nm-dnsmasq.conf exists
DEBU Checking NetworkManager configuration
DEBU NetworkManager configuration is good
INFO Checking if /etc/NetworkManager/dnsmasq.d/crc.conf exists
DEBU Checking dnsmasq configuration
DEBU dnsmasq configuration is good
INFO Checking if libvirt 'crc' network is available
DEBU Checking if libvirt 'crc' network exists
DEBU Running 'virsh --connect qemu:///system net-info crc'
DEBU Checking if libvirt 'crc' definition is up to date
DEBU Running 'virsh --connect qemu:///system net-dumpxml --inactive crc'
DEBU libvirt 'crc' network has the expected value
INFO Checking if libvirt 'crc' network is active
DEBU Checking if libvirt 'crc' network is active
DEBU Running 'virsh --connect qemu:///system net-info crc'
DEBU libvirt 'crc' network is already active
INFO Checking if CRC bundle is extracted in '$HOME/.crc'
INFO Checking if /home/tim/.crc/cache/crc_libvirt_4.14.7_amd64.crcbundle exists
DEBU error getting bundle info for crc_libvirt_4.14.7_amd64.crcbundle: could not find cached bundle info in /home/tim/.crc/cache/crc_libvirt_4.14.7_amd64: stat /home/tim/.crc/cache/crc_libvirt_4.14.7_amd64: no such file or directory
DEBU could not find cached bundle info in /home/tim/.crc/cache/crc_libvirt_4.14.7_amd64: stat /home/tim/.crc/cache/crc_libvirt_4.14.7_amd64: no such file or directory
INFO Getting bundle for the CRC executable
DEBU Ensuring directory /home/tim/.crc/cache exists
INFO Downloading bundle: /home/tim/.crc/cache/crc_libvirt_4.14.7_amd64.crcbundle...
unable to get verified hash for default bundle: Get "https://mirror.openshift.com/pub/openshift-v4/clients/crc/bundles/openshift/4.14.7/sha256sum.txt.sig": dial tcp: lookup mirror.openshift.com on 127.0.0.1:53: server misbehaving

note: crc_libvirt_4.14.7_amd64.crcbundle exists

$ ls -l /home/tim/.crc/cache/crc_libvirt_4.14.7_amd64.crcbundle
-rw-r-----. 1 tim tim 5031941801 Feb  1 15:45 /home/tim/.crc/cache/crc_libvirt_4.14.7_amd64.crcbundle

I am unsure if this is intentional. I did not see anywhere in OpenShift Local's documentation that stated network access was required for setup.

[cfergeau]

Can you try with crc setup --bundle ... or crc config set bundle ... to see if this works?
I would move the .crcbundle out of ~/.crc/cache for these tests.

This would only be a workaround, but this could narrow down the problem.

[Emmettlsc]

Yup, when I move the .crcbundle out of ~/.crc/cache and update the bundle config property with the new path, setup completes without any issues. The bundle property's default path was where I had placed the .crcbundle originally so I did not expect changing the property would change the behavior.

[gbraad]

and place it where crc expects

you most likely missed the checksum file. This is necessary to determine if the download succeeded/usable. The cache folder is not for the user, but rather an application-controlled storage.

... after this, the functionality worked as expected? What other actions are needed?

[Emmettlsc]

I don't think this was an issue with the checksum file, I think the first step of the download process when crc setup fails to find the .crcbundle is to fetch the checksum from the OpenShift mirror which fails due to the restricted network. Originally I had the checksum file and the checksum signature in the same path as the .crcbundle (~/.crc/cache/) but running crc setup would still try and fetch it from the OpenShift mirror.

The fix for me was moving the bundle anywhere but the default bundle config property:
* bundle Bundle path/URI - absolute or local path, http, https or docker URI (string, like 'https://foo.com/crc_libvirt_4.14.7_amd64.crcbundle', 'docker://quay.io/myorg/crc_libvirt_4.14.7_amd64.crcbundle:2.31.0' default '/home/tim/.crc/cache/crc_libvirt_4.14.7_amd64.crcbundle' )

so I could move it anywhere including within the .crc directory and it would be extracted as long as the path was not equal to the default path.