bootstrap.iso for rhel 7.7 having some issues #8616

aviratna · 2020-02-09T08:21:43Z

We tried with rhel 7.7 & vic 1.5.4 we are getting same issue.

Summary:
bootstrap.iso file builds successfully and containers could startup however, tether doesn't start (refer debug attached) and container fails with "docker: Error response from daemon: Server error from portlayer: unable to wait for process launch status: container VM has unexpectedly powered off."

vSphere and vCenter Server version
vSphere 6.7u2

VIC version
VIC Version: 1.5.4

@YanzhaoLi
@DanielXiao
@malikkal
@hickeng

aviratna · 2020-02-09T08:43:49Z

Please find the logs below:
++ drivers+=("vmw_pvscsi" "vmxnet3" "nfnetlink" "iptable_filter" "xt_conntrack" "nf_nat_ipv4" "iptable_nat" "nf_conntrack" "nf_conntrack_ipv4" "nf_defrag_ipv4" "ipt_REJECT" "xt_state")
++ for i in '${drivers[@]}'
++ modprobe vmw_pvscsi
[ 4.987530] piix4_smbus 0000:00:07.3: SMBus Host Controller not enabled!

tether tmpfs size before copying libraries:
++ df -k /mnt/containerfs/.tether
Filesystem 1K-blocks Used Available Use% Mounted on
tmpfs 230400 191272 39128 84% /mnt/containerfs/.tether

++ cp /sbin/rngd /mnt/containerfs/.tether/bin/rngd
++ cp -Ln /lib64/libcom_err.so.2 /lib64/libcom_err.so.2.1 /lib64/libcrypt-2.17.so /lib64/libcrypt.so.1 /lib64/libcrypto.so.1.0.2k /lib64/libcrypto.so.10 /lib64/libcryptsetup.so.12 /lib64/libcryptsetup.so.12.3.0 /lib64/libcryptsetup.so.4 /lib64/libcryptsetup.so.4.7.0 /lib64/libgcrypt.so.11 /lib64/libgcrypt.so.11.8.2 /lib64/libk5crypto.so.3 /lib64/libk5crypto.so.3.1 /lib64/libc.so.6 /lib64/libcurl.so.4 /lib64/libcurl.so.4.3.0 /lib64/libdl.so.2 /lib64/libfreebl3.chk /lib64/libfreebl3.so /lib64/libfreeblpriv3.chk /lib64/libfreeblpriv3.so /lib64/libgpg-error.so.0 /lib64/libgpg-error.so.0.10.0 /lib64/libgssapi_krb5.so.2 /lib64/libgssapi_krb5.so.2.2 /lib64/libidn.so.11 /lib64/libidn.so.11.6.11 /lib64/libkeyutils.so.1 /lib64/libkeyutils.so.1.5 /lib64/libkrb5.so.3 /lib64/libkrb5.so.3.3 /lib64/libkrb5support.so.0 /lib64/libkrb5support.so.0.1 /lib64/liblber-2.4.so.2 /lib64/liblber-2.4.so.2.10.7 /lib64/libldap-2.4.so.2 /lib64/libldap-2.4.so.2.10.7 /lib64/libldap_r-2.4.so.2 /lib64/libldap_r-2.4.so.2.10.7 /lib64/liblzma.so.5 /lib64/liblzma.so.5.2.2 /lib64/libm.so.6 /lib64/libnspr4.so /lib64/libnss3.so /lib64/libnss_compat-2.17.so /lib64/libnss_compat.so.2 /lib64/libnss_db-2.17.so /lib64/libnss_db.so.2 /lib64/libnss_dns-2.17.so /lib64/libnss_dns.so.2 /lib64/libnss_files-2.17.so /lib64/libnss_files.so.2 /lib64/libnss_hesiod-2.17.so /lib64/libnss_hesiod.so.2 /lib64/libnss_myhostname.so.2 /lib64/libnss_mymachines.so.2 /lib64/libnss_nis-2.17.so /lib64/libnss_nis.so.2 /lib64/libnss_nisplus-2.17.so /lib64/libnss_nisplus.so.2 /lib64/libnssckbi.so /lib64/libnssdbm3.chk /lib64/libnssdbm3.so /lib64/libnsspem.so /lib64/libnsssysinit.so /lib64/libnssutil3.so /lib64/libpcre.so.1 /lib64/libpcre.so.1.2.0 /lib64/libplc4.so /lib64/libplds4.so /lib64/libpthread.so.0 /lib64/libresolv.so.2 /lib64/librt.so.1 /lib64/libsasl2.so.3 /lib64/libsasl2.so.3.0.0 /lib64/libselinux.so.1 /lib64/libsmime3.so /lib64/libssh2.so.1 /lib64/libssh2.so.1.0.1 /lib64/libssl.so.1.0.2k /lib64/libssl.so.10 /lib64/libssl3.so /lib64/libsysfs.so.2 /lib64/libsysfs.so.2.0.1 /lib64/libxml2.so.2 /lib64/libxml2.so.2.9.1 /lib64/libz.so.1 /lib64/libz.so.1.2.7 /mnt/containerfs/.tether/lib64/
cp: error writing '/mnt/containerfs/.tether/lib64/libxml2.so.2.9.1': No space left on device
cp: failed to extend '/mnt/containerfs/.tether/lib64/libxml2.so.2.9.1': No space left on device
cp: error writing '/mnt/containerfs/.tether/lib64/libz.so.1': No space left on device
cp: failed to extend '/mnt/containerfs/.tether/lib64/libz.so.1': No space left on device
cp: error writing '/mnt/containerfs/.tether/lib64/libz.so.1.2.7': No space left on device
cp: failed to extend '/mnt/containerfs/.tether/lib64/libz.so.1.2.7': No space left on device

cat: write error: No space left on device
++ echo 'tether tmpfs size after copying libraries: '
tether tmpfs size after copying libraries:
++ df -k /mnt/containerfs/.tether
Filesystem 1K-blocks Used Available Use% Mounted on
tmpfs 230400 230400 0 100% /mnt/containerfs/.tether

++ echo 'switching to the new mount'
switching to the new mount
++ systemctl
Failed to get D-Bus connection: Operation not permitted
+++ readlink /usr/sbin/switch_root
++ [[ '' == \t\o\y\b\o\x ]]
++ exec switch_root /mnt/containerfs /.tether/tether
all stderr redirected to debug logFeb 9 2020 08:04:13.390Z INFO Registering tether extension Attach
Feb 9 2020 08:04:13.426Z INFO Registering tether extension Toolbox
Feb 9 2020 08:04:13.426Z INFO Registering tether extension Entropy
Feb 9 2020 08:04:13.428Z DEBUG [BEGIN] [vic/lib/tether.(*tether).Start:542] main tether loop
Feb 9 2020 08:04:13.428Z DEBUG [BEGIN] [vic/lib/tether.(*tether).setup:140] main tether setup
Feb 9 2020 08:04:13.428Z DEBUG [BEGIN] [main.(*operations).Log:57] operations.Log
Feb 9 2020 08:04:13.428Z INFO opening /dev/ttyS1 for debug log
Feb 9 2020 08:04:13.429Z DEBUG [ END ] [main.(*operations).Log:57] [364.061µs] operations.Log
Feb 9 2020 08:04:13.433Z INFO Started reaping child processes
Feb 9 2020 08:04:13.443Z DEBUG writing "::1 localhost localhost.localdomain localhost6 localhost6.localdomain6\n" to /.tether/etc/hosts
Feb 9 2020 08:04:13.443Z DEBUG writing "127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4\n" to /.tether/etc/hosts
Feb 9 2020 08:04:13.443Z ERROR Unable to flush file content to /.tether/etc/hosts717335263: write /.tether/etc/hosts717335263: no space left on device
Feb 9 2020 08:04:13.443Z DEBUG writing "fe00:: ip6-localnet\n" to /.tether/etc/hosts
Feb 9 2020 08:04:13.443Z DEBUG writing "ff00:: ip6-mcastprefix\n" to /.tether/etc/hosts
Feb 9 2020 08:04:13.443Z DEBUG writing "ff02::1 ip6-allnodes\n" to /.tether/etc/hosts
Feb 9 2020 08:04:13.444Z DEBUG writing "ff02::2 ip6-allrouters\n" to /.tether/etc/hosts
Feb 9 2020 08:04:13.444Z DEBUG writing "127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4\n" to /.tether/etc/hosts
Feb 9 2020 08:04:13.444Z DEBUG writing "::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 ip6-localhost ip6-loopback\n" to /.tether/etc/hosts
Feb 9 2020 08:04:13.444Z ERROR Unable to flush file content to /.tether/etc/hosts109424818: write /.tether/etc/hosts109424818: no space left on device
Feb 9 2020 08:04:13.444Z ERROR Failed tether setup: write /.tether/etc/hosts109424818: no space left on device
Feb 9 2020 08:04:13.444Z DEBUG [ END ] [vic/lib/tether.(*tether).setup:140] [16.23942ms] main tether setup
Feb 9 2020 08:04:13.444Z ERROR Failed to run setup: write /.tether/etc/hosts109424818: no space left on device
Feb 9 2020 08:04:13.445Z DEBUG [ END ] [vic/lib/tether.(*tether).Start:542] [18.64275ms] main tether loop
Feb 9 2020 08:04:13.445Z ERROR write /.tether/etc/hosts109424818: no space left on device
Feb 9 2020 08:04:13.445Z INFO Powering off the system
[ 5.878074] Power down.

YanzhaoLi · 2020-02-11T08:46:19Z

@aviratna It seems that the tmpfs /mnt/containerfs/ was used up. And it looks like a bug that the computed size is not large enough in

vic/isos/bootstrap.sh

Line 157 in 9becae4

echo ${size} > $(rootfs_dir $PKGDIR)/.tempfs_size

I'll try to fix it. And a workaround is to increase the size like doubling it and build the bootstrop.iso again.

And could you help do a test with this pr: #8618

malikkal · 2020-02-16T13:47:12Z

Thank you @YanzhaoLi

Making custom bootstrap iso
building rhel-7.7
Preparing systemd for bootstrap
Total tempfs size: 241
Constructing initramfs archive
1760669 blocks
Embedding build version v1.5.5-rc1-0-9becae4 (use BUILD_NUMBER environment variable to override)

will try the new bootstrap.iso and update here..

aviratna · 2020-02-17T03:15:40Z

@YanzhaoLi We are getting below error after building with above changes mentioned.

Error in file:
[ 4.795077] piix4_smbus 0000:00:07.3: SMBus Host Controller not enabled!

[ 4.921772] Error: Driver 'pcspkr' is already registered, aborting...

++ mount -t tmpfs -o size=m tmpfs /mnt/containerfs/.tether/
++ cp -Ln /lib64/libsysfs.so.2 /lib64/libsysfs.so.2.0.1 /lib64/libm.so.6 /lib64/libm-2.17.so '/lib64/libgcc_s*' /lib64/libip4tc.so.0 /lib64/libip4tc.so.0.1.0 /lib64/libip6tc.so.0 /lib64/libip6tc.so.0.1.0 /lib64/libiptc.so.0 /lib64/libiptc.so.0.0.0 /lib64/libxtables.so.10 /lib64/libxtables.so.10.0.0 /lib64/libdl-2.17.so /lib64/libdl.so.2 /lib64/libc.so.6 /lib64/libc-2.17.so /mnt/containerfs/.tether/lib64/
cp: cannot stat '/lib64/libgcc_s*': No such file or directory

/bin/repoinit: line 36: cannot create temp file for here-document: No space left on device

Failed to get D-Bus connection: Operation not permitted

Feb 17 2020 02:38:51.685Z ERROR Starting entropy failed with "fork/exec : no such file or directory"

Feb 17 2020 02:38:51.686Z ERROR Failed to start extension Entropy: fork/exec : no such file or directory
Feb 17 2020 02:38:51.694Z ERROR Failed to run setup: fork/exec : no such file or directory

Feb 17 2020 02:38:51.696Z ERROR fork/exec : no such file or directory

Please find the detailed tether.debug log below, not able to upload the file:

export PATH=/usr/local/bin:/usr/bin:/sbin
PATH=/usr/local/bin:/usr/bin:/sbin
echo '######## Loading repo customizations ########'
######## Loading repo customizations ########
. /bin/repoinit
++ set -x
echo '######## Done repo customizations ######## -- exit 0'
######## Done repo customizations ######## -- exit 0
mount -t proc none /proc
mount -t sysfs none /sys
mount -t devtmpfs none /dev
mknod -m 622 /dev/console c 5 1
mknod: '/dev/console': File exists
mknod -m 666 /dev/null c 1 3
mknod: '/dev/null': File exists
mknod -m 666 /dev/zero c 1 5
mknod: '/dev/zero': File exists
mknod -m 666 /dev/ptmx c 5 2
mknod: '/dev/ptmx': File exists
mknod -m 666 /dev/tty c 5 0
mknod: '/dev/tty': File exists
mknod -m 444 /dev/random c 1 8
mknod: '/dev/random': File exists
mknod -m 444 /dev/urandom c 1 9
mknod: '/dev/urandom': File exists
chown root:tty /dev/console /dev/ptmx /dev/tty
mkdir -p /dev/pts /dev/shm
mount -t devpts -o gid=5,mode=620,ptmxmode=666 none /dev/pts
mount -t tmpfs none /dev/shm
UDEV_PID=82
/usr/lib/systemd/systemd-udevd
udevadm trigger
starting version 219
echo 'Bring up loopback interface'
Bring up loopback interface
ip link show lo
1: lo: mtu 65536 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
ip link set lo up
echo '######## Launching bootstrap entrypoint ########'
######## Launching bootstrap entrypoint ########
. /bin/bootstrap
++ set -x
++ echo 1
[ 4.795077] piix4_smbus 0000:00:07.3: SMBus Host Controller not enabled!
++ echo 1
++ echo 'Loading drivers'
Loading drivers
++ drivers+=("vmw_pvscsi" "vmxnet3" "nfnetlink" "iptable_filter" "xt_conntrack" "nf_nat_ipv4" "iptable_nat" "nf_conntrack" "nf_conntrack_ipv4" "nf_defrag_ipv4" "ipt_REJECT" "xt_state")
++ for i in '${drivers[@]}'
++ modprobe vmw_pvscsi
++ for i in '${drivers[@]}'
++ modprobe vmxnet3
++ for i in '${drivers[@]}'
++ modprobe nfnetlink
++ for i in '${drivers[@]}'
++ modprobe iptable_filter
++ for i in '${drivers[@]}'
++ modprobe xt_conntrack
++ for i in '${drivers[@]}'
++ modprobe nf_nat_ipv4
++ for i in '${drivers[@]}'
++ modprobe iptable_nat
++ for i in '${drivers[@]}'
++ modprobe nf_conntrack
++ for i in '${drivers[@]}'
++ modprobe nf_conntrack_ipv4
++ for i in '${drivers[@]}'
++ modprobe nf_defrag_ipv4
++ for i in '${drivers[@]}'
++ modprobe ipt_REJECT
++ for i in '${drivers[@]}'
++ modprobe xt_state
++ udevadm control --reload-rules
[ 4.921772] Error: Driver 'pcspkr' is already registered, aborting...
++ udevadm trigger --type=subsystems --action=add
++ udevadm trigger --type=devices --action=add
[ 4.980697] sd 0:0:0:0: [sda] Assuming drive cache: write through
++ MOUNTPOINT=/mnt/containerfs
++ mkdir -p /mnt/containerfs
++ echo 'Waiting for rootfs'
Waiting for rootfs
++ '[' '!' -e /dev/disk/by-label/containerfs ']'
++ sleep 0.1
[ 5.049421] Error: Driver 'pcspkr' is already registered, aborting...
++ '[' '!' -e /dev/disk/by-label/containerfs ']'
++ sleep 0.1
++ '[' '!' -e /dev/disk/by-label/containerfs ']'
++ sleep 0.1
++ '[' '!' -e /dev/disk/by-label/containerfs ']'
++ timeout -s KILL 2m mount -t ext4 /dev/disk/by-label/containerfs /mnt/containerfs
++ mkdir -p /mnt/containerfs/.tether /mnt/containerfs/dev /mnt/containerfs/proc /mnt/containerfs/sys
++ '[' -f /.tempfs_size ']'
+++ cat /.tempfs_size
++ tsize=
++ mount -t tmpfs -o size=m tmpfs /mnt/containerfs/.tether/
++ ln -s lib64 /mnt/containerfs/.tether/lib
++ mkdir -p /mnt/containerfs/.tether/lib64 /mnt/containerfs/.tether/usr/lib/iptables /mnt/containerfs/.tether/run
++ echo 'Publishing modules within container'
Publishing modules within container
++ mkdir -p /mnt/containerfs/lib/modules
++ mkdir -p /mnt/containerfs/.tether/lib/modules
++ mount --bind /mnt/containerfs/.tether/lib/modules /mnt/containerfs/lib/modules
++ cp -pr /lib/modules/3.10.0-1062.12.1.el7.x86_64 /mnt/containerfs/lib/modules/
++ echo 'prepping for switch to container filesystem'
prepping for switch to container filesystem
++ cp /bin/tether /mnt/containerfs/.tether/tether
++ cp /bin/unpack /mnt/containerfs/.tether/unpack
++ ln -s tether /mnt/containerfs/.tether/tether-debug
++ echo 'tether tmpfs size before copying libraries: '
tether tmpfs size before copying libraries:
++ df -k /mnt/containerfs/.tether
Filesystem 1K-blocks Used Available Use% Mounted on
tmpfs 0 0 0 - /mnt/containerfs/.tether
++ install-iptables /mnt/containerfs/.tether
++ mkdir -p /mnt/containerfs/.tether/bin /mnt/containerfs/.tether/lib64
++ mkdir -p /mnt/containerfs/.tether/usr/lib64
++ cp -Ln /lib64/ld-linux-x86-64.so.2 /mnt/containerfs/.tether/lib64/
++ cp -L /sbin/iptables /mnt/containerfs/.tether/bin/iptables
++ ln -s bin/iptables /mnt/containerfs/.tether/iptables
++ cp -Ln /lib64/libsysfs.so.2 /lib64/libsysfs.so.2.0.1 /lib64/libm.so.6 /lib64/libm-2.17.so '/lib64/libgcc_s' /lib64/libip4tc.so.0 /lib64/libip4tc.so.0.1.0 /lib64/libip6tc.so.0 /lib64/libip6tc.so.0.1.0 /lib64/libiptc.so.0 /lib64/libiptc.so.0.0.0 /lib64/libxtables.so.10 /lib64/libxtables.so.10.0.0 /lib64/libdl-2.17.so /lib64/libdl.so.2 /lib64/libc.so.6 /lib64/libc-2.17.so /mnt/containerfs/.tether/lib64/
cp: cannot stat '/lib64/libgcc_s': No such file or directory**
++ cp -a /lib64/xtables /mnt/containerfs/.tether/lib64/
++ cp -Ln /lib64/libsysfs.so.2 /lib64/libsysfs.so.2.0.1 /lib64/libm.so.6 /lib64/libm-2.17.so '/lib64/libgcc_s' /lib64/libip4tc.so.0 /lib64/libip4tc.so.0.1.0 /lib64/libip6tc.so.0 /lib64/libip6tc.so.0.1.0 /lib64/libiptc.so.0 /lib64/libiptc.so.0.0.0 /lib64/libxtables.so.10 /lib64/libxtables.so.10.0.0 /lib64/libdl-2.17.so /lib64/libdl.so.2 /lib64/libc.so.6 /lib64/libc-2.17.so /mnt/containerfs/.tether/usr/lib64/
cp: cannot stat '/lib64/libgcc_s': No such file or directory**
++ cp -a /lib64/xtables /mnt/containerfs/.tether/usr/lib64/
++ cat -
/bin/repoinit: line 88: cannot create temp file for here-document: No space left on device
++ chmod a+x /mnt/containerfs/.tether/bin/iptables-wrapper
++ install-entropy /mnt/containerfs/.tether
++ mkdir -p /mnt/containerfs/.tether/opt/config /mnt/containerfs/.tether/bin/lib64 /mnt/containerfs/.tether/lib64
++ cp -Ln /lib64/ld-linux-x86-64.so.2 /mnt/containerfs/.tether/lib64/
++ cp -Ln /lib64/libc.so.6 /mnt/containerfs/.tether/lib64/
++ cp /sbin/rngd /mnt/containerfs/.tether/bin/rngd
++ cp -Ln /lib64/libcom_err.so.2 /lib64/libcom_err.so.2.1 /lib64/libcrypt-2.17.so /lib64/libcrypt.so.1 /lib64/libcrypto.so.1.0.2k /lib64/libcrypto.so.10 /lib64/libcryptsetup.so.12 /lib64/libcryptsetup.so.12.3.0 /lib64/libcryptsetup.so.4 /lib64/libcryptsetup.so.4.7.0 /lib64/libgcrypt.so.11 /lib64/libgcrypt.so.11.8.2 /lib64/libk5crypto.so.3 /lib64/libk5crypto.so.3.1 /lib64/libc.so.6 /lib64/libcurl.so.4 /lib64/libcurl.so.4.3.0 /lib64/libdl.so.2 /lib64/libfreebl3.chk /lib64/libfreebl3.so /lib64/libfreeblpriv3.chk /lib64/libfreeblpriv3.so /lib64/libgpg-error.so.0 /lib64/libgpg-error.so.0.10.0 /lib64/libgssapi_krb5.so.2 /lib64/libgssapi_krb5.so.2.2 /lib64/libidn.so.11 /lib64/libidn.so.11.6.11 /lib64/libkeyutils.so.1 /lib64/libkeyutils.so.1.5 /lib64/libkrb5.so.3 /lib64/libkrb5.so.3.3 /lib64/libkrb5support.so.0 /lib64/libkrb5support.so.0.1 /lib64/liblber-2.4.so.2 /lib64/liblber-2.4.so.2.10.7 /lib64/libldap-2.4.so.2 /lib64/libldap-2.4.so.2.10.7 /lib64/libldap_r-2.4.so.2 /lib64/libldap_r-2.4.so.2.10.7 /lib64/liblzma.so.5 /lib64/liblzma.so.5.2.2 /lib64/libm.so.6 /lib64/libnspr4.so /lib64/libnss3.so /lib64/libnss_compat-2.17.so /lib64/libnss_compat.so.2 /lib64/libnss_db-2.17.so /lib64/libnss_db.so.2 /lib64/libnss_dns-2.17.so /lib64/libnss_dns.so.2 /lib64/libnss_files-2.17.so /lib64/libnss_files.so.2 /lib64/libnss_hesiod-2.17.so /lib64/libnss_hesiod.so.2 /lib64/libnss_myhostname.so.2 /lib64/libnss_mymachines.so.2 /lib64/libnss_nis-2.17.so /lib64/libnss_nis.so.2 /lib64/libnss_nisplus-2.17.so /lib64/libnss_nisplus.so.2 /lib64/libnssckbi.so /lib64/libnssdbm3.chk /lib64/libnssdbm3.so /lib64/libnsspem.so /lib64/libnsssysinit.so /lib64/libnssutil3.so /lib64/libpcre.so.1 /lib64/libpcre.so.1.2.0 /lib64/libplc4.so /lib64/libplds4.so /lib64/libpthread.so.0 /lib64/libresolv.so.2 /lib64/librt.so.1 /lib64/libsasl2.so.3 /lib64/libsasl2.so.3.0.0 /lib64/libselinux.so.1 /lib64/libsmime3.so /lib64/libssh2.so.1 /lib64/libssh2.so.1.0.1 /lib64/libssl.so.1.0.2k /lib64/libssl.so.10 /lib64/libssl3.so /lib64/libsysfs.so.2 /lib64/libsysfs.so.2.0.1 /lib64/libxml2.so.2 /lib64/libxml2.so.2.9.1 /lib64/libz.so.1 /lib64/libz.so.1.2.7 /mnt/containerfs/.tether/lib64/
++ cat -
/bin/repoinit: line 36: cannot create temp file for here-document: No space left on device
++ echo 'tether tmpfs size after copying libraries: '
tether tmpfs size after copying libraries:
++ df -k /mnt/containerfs/.tether
Filesystem 1K-blocks Used Available Use% Mounted on
tmpfs 0 0 0 - /mnt/containerfs/.tether
++ iptables -N VIC
++ iptables -P INPUT DROP
++ iptables -P OUTPUT DROP
++ iptables -P FORWARD DROP
++ iptables -A INPUT -j VIC
++ iptables -A OUTPUT -j VIC
++ iptables -A INPUT -i lo -j ACCEPT
++ iptables -A OUTPUT -o lo -j ACCEPT
++ iptables -A FORWARD -i lo -o lo -j ACCEPT
++ echo 262144
++ '[' -d /sys/fs/cgroup ']'
++ mount -t cgroup -o all cgroup /sys/fs/cgroup
+++ ls -1 /dev/disk/by-label
+++ wc -l
+++ ls -1 /sys/block/
+++ wc -l
+++ grep -v 'ram*'
+++ grep -v 'loop*'
++ [[ 2 -eq 2 ]]
++ [[ 82 != '' ]]
++ kill 82
++ echo 'switching to the new mount'
switching to the new mount
++ systemctl
Failed to get D-Bus connection: Operation not permitted
+++ readlink /usr/sbin/switch_root
++ [[ '' == \t\o\y\b\o\x ]]
++ exec switch_root /mnt/containerfs /.tether/tether
all stderr redirected to debug logFeb 17 2020 02:38:51.560Z INFO Registering tether extension Attach
Feb 17 2020 02:38:51.614Z INFO Registering tether extension Toolbox
Feb 17 2020 02:38:51.615Z INFO Registering tether extension Entropy
Feb 17 2020 02:38:51.616Z DEBUG [BEGIN] [vic/lib/tether.(*tether).Start:542] main tether loop
Feb 17 2020 02:38:51.617Z DEBUG [BEGIN] [vic/lib/tether.(*tether).setup:140] main tether setup
Feb 17 2020 02:38:51.618Z DEBUG [BEGIN] [main.(*operations).Log:57] operations.Log
Feb 17 2020 02:38:51.619Z INFO opening /dev/ttyS1 for debug log
Feb 17 2020 02:38:51.620Z DEBUG [ END ] [main.(*operations).Log:57] [1.797905ms] operations.Log
Feb 17 2020 02:38:51.621Z INFO Started reaping child processes
Feb 17 2020 02:38:51.643Z DEBUG writing "127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4\n" to /.tether/etc/hosts
Feb 17 2020 02:38:51.644Z DEBUG writing "::1 localhost localhost.localdomain localhost6 localhost6.localdomain6\n" to /.tether/etc/hosts
Feb 17 2020 02:38:51.646Z DEBUG writing "::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 ip6-localhost ip6-loopback\n" to /.tether/etc/hosts
Feb 17 2020 02:38:51.648Z DEBUG writing "fe00:: ip6-localnet\n" to /.tether/etc/hosts
Feb 17 2020 02:38:51.649Z DEBUG writing "ff00:: ip6-mcastprefix\n" to /.tether/etc/hosts
Feb 17 2020 02:38:51.650Z DEBUG writing "ff02::1 ip6-allnodes\n" to /.tether/etc/hosts
Feb 17 2020 02:38:51.651Z DEBUG writing "ff02::2 ip6-allrouters\n" to /.tether/etc/hosts
Feb 17 2020 02:38:51.652Z DEBUG writing "127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4\n" to /.tether/etc/hosts
Feb 17 2020 02:38:51.653Z INFO unmounting /etc/hosts
Feb 17 2020 02:38:51.654Z DEBUG path is not currently a bindmount target
Feb 17 2020 02:38:51.655Z INFO bind-mounting /.tether/etc/hosts on /etc/hosts
Feb 17 2020 02:38:51.656Z INFO Starting extension Toolbox
Feb 17 2020 02:38:51.656Z INFO toolbox: waiting for initialization
2020/02/17 02:38:51 dispatching power op "OS_PowerOn"
Feb 17 2020 02:38:51.680Z INFO toolbox: service is ready (power on event received)
Feb 17 2020 02:38:51.681Z INFO Starting extension Entropy
Feb 17 2020 02:38:51.682Z INFO Starting entropy
Feb 17 2020 02:38:51.683Z INFO Starting entropy daemon
Feb 17 2020 02:38:51.685Z ERROR Starting entropy failed with "fork/exec : no such file or directory"
Feb 17 2020 02:38:51.686Z ERROR Failed to start extension Entropy: fork/exec : no such file or directory
Feb 17 2020 02:38:51.689Z DEBUG [ END ] [vic/lib/tether.(*tether).setup:140] [71.478996ms] main tether setup
Feb 17 2020 02:38:51.685Z DEBUG Inspecting children with status change
Feb 17 2020 02:38:51.691Z DEBUG Reaped process 288, return code: 0
Feb 17 2020 02:38:51.691Z INFO Reaped zombie process PID 288
Feb 17 2020 02:38:51.692Z DEBUG Inspecting children with status change
Feb 17 2020 02:38:51.693Z DEBUG No more child processes to reap
Feb 17 2020 02:38:51.694Z ERROR Failed to run setup: fork/exec : no such file or directory
Feb 17 2020 02:38:51.695Z DEBUG [ END ] [vic/lib/tether.(*tether).Start:542] [79.272395ms] main tether loop
Feb 17 2020 02:38:51.696Z ERROR fork/exec : no such file or directory
Feb 17 2020 02:38:51.698Z INFO Powering off the system
[ 5.936949] Power down.

@malikkal

YanzhaoLi · 2020-02-17T04:52:19Z

@aviratna @malikkal It seems the size is still not large enough. BTW, Is the rhel-7.7 same as cento-7, or is there only difference between kernel ?

malikkal · 2020-02-17T07:15:20Z

same as CentOS 7.7 including the kernel version. CentOS 7.7. is built from RHEL 7.7. SRPMs minus branding.

BTW, the build containers that I am using are non-privileged. Should I be building from privileged containers?

YanzhaoLi · 2020-02-18T08:40:10Z

the

@malikkal I don't think it matters.
BTW, I tried out centos-7 and it worked. I'll continue to debug with centos-7.7.

malikkal · 2020-02-21T03:32:25Z

@YanzhaoLi any updates on this, please?

YanzhaoLi · 2020-02-21T03:42:17Z

@malikkal Still in investigation and to find the solution might not be a trivial.
The size of generated Centos7.7 iso image is 354MB which is too larger than that of centos7, which results in the exhaustion of disk space when booting.
I don't know the reason but it might be the poor dependency of management in the rpms.
So first we have to figure out the extra and unnecessary packages for centos7.7.

malikkal · 2020-02-21T10:24:16Z

@YanzhaoLi okay, thanks mate. In parallel, let me also revert to 7.5 or older and see if it helps reducing the size.

malikkal · 2020-03-04T11:34:08Z

rhel7.4 (bootstrap.iso is 273 MB)
3.10.0-693.21.1.el7.x86_64 #1 SMP Fri Feb 23 18:54:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

rhel7.5 (bootstrap.iso is 310 MB)
3.10.0-862.14.4.el7.x86_64 #1 SMP Fri Sep 21 09:07:21 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
for rhel7.5, the tmpfs for /.tether mount is remaining at only 5%. :( I will proceed with 7.5 for now.

hickeng · 2020-07-07T18:43:41Z

Looking at this after call last week.

tmpfs for /.tether mount is remaining at only 5%.

The tmpfs is supposed to be very nearly full - it's size it calculated based on the size of the files needed. Ideally we'd be able to hit 100% perfectly so as to use the minimum number of memory pages, but some tolerance is useful.

As @YanzhaoLi noted, 354MB is way more than I'd expect for a bootstrap iso.

This line looks wrong to me - there's zero reason for us to have curl in the bootstrap, nor the key utils. Certainly not as dependencies of rngd which, in most cases these days, should just be calling RDRAND:
https://github.com/vmware/vic/blob/master/isos/base/repos/centos-7/init.sh#L31

I think it likely that a mistake was made when determining the library dependencies for rgnd that has bloated this image and any based on it.

The probable reason for the out of space issues is that there are a lot of new libraries introduced by updates for centos7.5 and those libraries are not matched by the list used to calculate the necessary tmpfs size:

vic/isos/bootstrap.sh

Line 19 in 1967d0c

tempfs_target_list=('/lib/modules/*' \

This should not be fixed just by adding the new libraries to the list used to calculate the tmpfs size - it should be fixed by determining the actual runtime dependencies used by rngd and iptables for the operations the tether requires.

Additionally there is a double copy of the iptables libraries into /.tether/usr/lib64:

vic/isos/base/repos/centos-7/init.sh

Line 82 in 6729b55

    
           cp -Ln /lib64/lib{sysfs*,m.*,m-*,gcc_s*,ip*tc*,xtables*,dl*,c.so*,c-*} $1/usr/lib64/

I assume this is because something has a hardcoded library path suffix but this would be better addressed with a symlink from /.tether/usr/lib64 to /.tether/lib64 as is the same set of libraries. Currently this will be doubling the footprint of any library binaries which definitely will not be accounted for in the tmpfs size calculation.

YanzhaoLi linked a pull request Feb 11, 2020 that will close this issue

Include additional files to compute tmpfs size #8618

Open

YanzhaoLi assigned michmike Mar 5, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bootstrap.iso for rhel 7.7 having some issues #8616

bootstrap.iso for rhel 7.7 having some issues #8616

aviratna commented Feb 9, 2020 •

edited

Loading

aviratna commented Feb 9, 2020

YanzhaoLi commented Feb 11, 2020 •

edited

Loading

malikkal commented Feb 16, 2020 •

edited

Loading

aviratna commented Feb 17, 2020

YanzhaoLi commented Feb 17, 2020

malikkal commented Feb 17, 2020

YanzhaoLi commented Feb 18, 2020

malikkal commented Feb 21, 2020

YanzhaoLi commented Feb 21, 2020

malikkal commented Feb 21, 2020

malikkal commented Mar 4, 2020

hickeng commented Jul 7, 2020 •

edited

Loading

bootstrap.iso for rhel 7.7 having some issues #8616

bootstrap.iso for rhel 7.7 having some issues #8616

Comments

aviratna commented Feb 9, 2020 • edited Loading

aviratna commented Feb 9, 2020

YanzhaoLi commented Feb 11, 2020 • edited Loading

malikkal commented Feb 16, 2020 • edited Loading

aviratna commented Feb 17, 2020

YanzhaoLi commented Feb 17, 2020

malikkal commented Feb 17, 2020

YanzhaoLi commented Feb 18, 2020

malikkal commented Feb 21, 2020

YanzhaoLi commented Feb 21, 2020

malikkal commented Feb 21, 2020

malikkal commented Mar 4, 2020

hickeng commented Jul 7, 2020 • edited Loading

aviratna commented Feb 9, 2020 •

edited

Loading

YanzhaoLi commented Feb 11, 2020 •

edited

Loading

malikkal commented Feb 16, 2020 •

edited

Loading

hickeng commented Jul 7, 2020 •

edited

Loading