rd.live.overlay.overlayfs=1 is broken on Fedora 39 LiveOS #2645

[mahdiaqallal]

Describe the bug
A clear and concise description of what the error is.

rd.live.overlay.overlayfs=1 is supposed to provide a non-persistent (i.e. deleted after reboot of Live Image) and temporary storage in RAM at /run/overlayfs which is by default 32GiB unless another parameter is supplied : rd.live.overlay.size=<size_MiB> . See dracut documentation #Booting live images

Using rd.live.overlay.overlayfs=1 as kernel command line parameter has two issues

1. sudo dmesg | grep overlayfs includes an error:
   [ 13.317280] overlayfs: failed to resolve '/run/overlayfs': -2

2. Installing any rpm package once Fedora 39 LiveOS is running, results with the same errors
   error: lsetfilecon: (33 /usr/bin/make-dummy-cert;66130180, system_u:object_r:bin_t:s0) Permission denied
   error: Plugin selinux: hook fsm_file_prepare failed
   [...]
   Error unpacking rpm package
   Failed:
   Error: Transaction failed

Below an example with openssl

 sudo dnf install -y openssl`
Last metadata expiration check: 1:44:39 ago on Sun 07 Apr 2024 08:41:58 PM CEST.
Dependencies resolved.
===============================================================================
 Package                  Architecture            Version                            Repository               Size
===============================================================================
Installing:
 openssl                  x86_64                  1:3.1.1-4.fc39                     fedora                  1.0 M

Transaction Summary
===============================================================================
Install  1 Package

Total download size: 1.0 M
Installed size: 1.6 M
Downloading Packages:
openssl-3.1.1-4.fc39.x86_64.rpm                                                    1.4 MB/s | 1.0 MB     00:00    
-------------------------------------------------------------------------------------------------------------------
Total                                                                              802 kB/s | 1.0 MB     00:01     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                           1/1 
  Installing       : openssl-1:3.1.1-4.fc39.x86_64                                                             1/1 
error: lsetfilecon: (33 /usr/bin/make-dummy-cert;66130180, system_u:object_r:bin_t:s0) Permission denied
error: Plugin selinux: hook fsm_file_prepare failed

Error unpacking rpm package openssl-1:3.1.1-4.fc39.x86_64
  Verifying        : openssl-1:3.1.1-4.fc39.x86_64                                                             1/1 

Failed:
  openssl-1:3.1.1-4.fc39.x86_64                                                                                    

Error: Transaction failed

Distribution used
Which distribution was this behaviour seen in?
Fedora 39 LiveOS

Dracut version
Which dracut version was this behaviour seen in?
Fedora 39 dracut-059-16.fc39
Init system
Which init system is being used?
systemd
To Reproduce
Steps or code to reproduce the behavior.
Insert alongside your kernel command line parameter on Fedora 39 Live ISO (for example in a grub.cfg menuentry) rd.live.overlay.overlayfs=1 after the parameter rd.live.image
Expected behavior
A clear and concise description of what you expected to happen.

We should have with this parameter :

1. a root live filesystem with 32GiB of available RAM space instead of 4.9G is used and 1.5G is available space out of a total size of 6.4G if this parameter is not used

2. the ability to install rpm packages with sudo dnf install <package> until the available RAM is depleted

Additional context
Add any other context you like about the problem here.

a. Users of Fedora Linux have been reporting related bugs here: https://discussion.fedoraproject.org/t/fedora-liveos-root-system-and-available-ram/82531/1

CC @gregory-lee-bartholomew

(Cherry-picked commit from dracutdevs/dracut#2604 and dracut-ng #61)

b. The Booting Live Images documentation section could need be more "verbose" with more explanations, examples, use-cases. IMHO

[LaszloGombos]

This needs to be triage to have evidence that this is actionable for dracut-ng main version.

This might not be an issue with dracut, but with Fedora or https://github.com/livecd-tools/livecd-tools (which are all different repositories, in which case the bug should be filed not here).

CC @pvalena @Conan-Kudo @AdamWill @FGrose

[pvalena]

It'd be best to test with v60 first: https://bodhi.fedoraproject.org/updates/FEDORA-2024-de1619b856

[mahdiaqallal]

This needs to be triage to have evidence that this is actionable for dracut-ng main version.

This might not be an issue with dracut, but with Fedora or https://github.com/livecd-tools/livecd-tools (which are all different repositories, in which case the bug should be filed not here).

CC @pvalena @Conan-Kudo @AdamWill @FGrose

@LaszloGombos : I didn't use any of the livecd-tools to reproduce this issue. The Fedora 39 LiveOS was directly written into an usb using following command
sudo dd if=/<path-to-Fedora 39 LiveOS.iso> of=/dev/sdb

Then the usb drive was booted and it's grub.cfg modified according to my description above

FYI: livecd-tools can't be used as of Fedora 39 and since Fedora 35+ because the structure of the .iso files has been changed and an appropriate release still needs to be available from the official Fedora repos. see this issues #244 and #253 in livecd-tools repository

[mahdiaqallal]

Food for thoughts: couldn't it be related with SELinux rules because of the folders where /run/overlayfs stores all modified files and directories after thoses unpacked from squashfs in the live boot ?

I found this thread in Fedora Discussion and this and that bugzilla report

Can't be sure if it helps or not, it is well beyond my understanding

[mahdiaqallal]

Thanks @FGrose for the speedy upgrade of the man page on Booting a Live Image.
It might need some further enhancement most importantly more examples

So I can provide feedback on a Fedora 40beta Workstation LIve ISO USB drive, please indicate which are the MINIMAL kernetl/dracut command line parameters required to activate:

Temporary in RAM memory reboot volatile overlay
OR
Permanent on disk reboot resistant overlay

For now whatever combination of rd.live.overlayXXX parameters I try, I get either dmesg errors or no overlay feature (neither temporary nor permanent)

@FGrose : thanks for your reply . It might need some elaboration though:

Which kernel command line parameter is needed to activate on a Fedora Worsktation Live .ISO, the "Device-mapper snapshot based non-persistent overlays":

Device-mapper snapshot based non-persistent overlays are
sparse files in RAM that only consume content space as required blocks are
allocated. These snapshot based overlays default to an apparent size of 32 GiB
in RAM.

Regarding live-respins

The 2 problems you identified at the top are fixed in an update of Fedora 39 such as in the live-respin, https://dl.fedoraproject.org/pub/alt/live-respins/F39-WORK-x86_64-LIVE-20240401.iso

Are live-respins natively corrected for rd.live.overlay.overlayfs=1 ? Does this mean that on Fedora 40 Worsktation Live ISO the bug for rd.live.overlay.overlayfs=1 is fixed too ?

I'll try soon sudo mount -o remount,size=70% /run (on 32GiB RAM laptop) on Fedora 40 Workstation LIve ISO and will provide feedback

[mahdiaqallal]

Thanks @FGrose for the speedy upgrade of the man page on Booting a Live Image.
It might need some further enhancement most importantly more examples

So I can provide feedback on a Fedora 40beta Workstation LIve ISO USB drive, please indicate:

What are the minimal kernel command line parameters to activate (on a freshly created Fedora 40beta Workstation Live USB):

1. Temporary in RAM memory non reboot resistant overlay
   OR
2. Permanent boot resistant on disk overlay (provided a dedicated vfat or ext4 or btrfs partition has been created)

For now whatever combination of rd.live.overlayXXX parameters I try, I get either dmesg errors or no overlay feature (neither temporary nor permanent)

@FGrose : thanks for your reply . It might need some elaboration though:

Which kernel command line parameter is needed to activate on a Fedora Worsktation Live .ISO, the "Device-mapper snapshot based non-persistent overlays":

    Device-mapper snapshot based non-persistent overlays are
    sparse files in RAM that only consume content space as required blocks are
    allocated. These snapshot based overlays default to an apparent size of 32 GiB
    in RAM.

Regarding live-respins

    The 2 problems you identified at the top are fixed in an update of Fedora 39 such as in the live-respin, https://dl.fedoraproject.org/pub/alt/live-respins/F39-WORK-x86_64-LIVE-20240401.iso

Are live-respins natively corrected for rd.live.overlay.overlayfs=1 ? Does this mean that on Fedora 40 Worsktation Live ISO the bug for rd.live.overlay.overlayfs=1 is fixed too ?

I'll try soon sudo mount -o remount,size=70% /run (on 32GiB RAM laptop) on Fedora 40 Workstation LIve ISO and will provide feedback

[LaszloGombos]

So I can provide feedback on a Fedora 40beta Workstation LIve ISO USB drive, please indicate:

@mahdiaqallal . When you e.g. Google "Fedora bug database" that will lead you to https://fedoraproject.org/wiki/Bugzilla, which will lead you to https://bugzilla.redhat.com/ .

Reporting bugs in the wrong bug database has at least two consequences

• It does not get the right attention as people who can actually address the issue do not see it
• It gets the wrong attention from the wrong set of people and create noise in the project where it does not belong

Please file the bug in the Fedora bug database or provide some evidence why dracut-ng upstream is impacted.

[mahdiaqallal]

Thank your for your explanations @LaszloGombos

I will report bug(s) accordingly regarding Fedora

Il seems to me nonetheless that dracut-ng uptream is affected when using it's rd.live.overlay.overlayfs parameter to activate temporary in RAM memory non reboot resistant overlay by means of overlayfs while booting a live image.

I get the following message:

$ sudo dmesg | grep overlay
[    0.000000] Command line: BOOT_IMAGE=/images/pxeboot/vmlinuz root=live:CDLABEL=Fedora-WS-Live-40_B-1-10 rd.live.image quiet rhgb rd.live.overlay.overlayfs
[    0.074626] Kernel command line: BOOT_IMAGE=/images/pxeboot/vmlinuz root=live:CDLABEL=Fedora-WS-Live-40_B-1-10 rd.live.image quiet rhgb rd.live.overlay.overlayfs rd.live.overlay rd.live.overlay.thin
[   11.590183] overlayfs: failed to resolve '/run/overlayfs': -2
[   13.642866] evm: overlay not supported

Either that's a bug or I am not using the dracut-ng command line parameters correctly. If so, the documentation recently upgraded by @FGrose might still need more explanations and most importantly more examples IMHO:

What are the minimal kernel command line or which among rd.live.overlay/overlayfs/thin/size/cowfs parameters have to be used and their proper combinations to activate in each case (on a freshly created Fedora 40 Workstation Live USB) for :

1. A temporary in RAM memory non reboot resistant overlay

or for

2. A permanent boot resistant on disk overlay using overlayFS and not DeviceMapper (provided a dedicated vfat or ext4 or btrfs partition has been created). 

In the latter case it is still unclear in the doc what the parameter rd.live.overlay has to point to: an filesystem file image e.g. mypermoverlay.img or a directory ?

How should I handle this now that the present issue is closed ? Is it a new issue to file, requesting more details in man page or is it relevant a bug issue here ?