Update CI VMs to F40, F39, D13

[cevich]

Ref: containers/automation_images#349

Does this PR introduce a user-facing change?

None

[cevich]

@Luap99 PTAL when you get a moment. I'm seeing a bunch of pasta related "permission denied" failures on the debian integration tests (annotated log). Ed's script reports this should be passt version 2024-04-26. Is this anything you've seen/heard of before or know how to fix?

Edit: Nevermind, Paul responded in the related c/automation_images PR:

pasta is not working on debian, I bet apparmor is blocking access
containers/buildah#5440
Although the version looks new enough and given the old version worked this seems weird??

[cevich]

@Luap99 I noticed this is the same version of pasta in the new F40 CI VM image and no tasks failed there. I'm a bit uneasy thinking about it, but how would you feel if I:

1. Open a podman issue, referencing the buildah issue.
2. Disable the int podman debian-13 rootless host sqlite task with a link to that issue

Edit: Paul replied in the other PR

[Luap99]

Look like it is still not working... Not sure if the profile is wrong or if there is some special magic needed after changes

[cevich]

Look like it is still not working... Not sure if the profile is wrong or if there is some special magic needed after changes

Let me confirm the basics first, like is the file there, does it have correct permissions, etc.

@edsantiago I'm seeing these weird tar: Skipping to next header failures on Debian. Is that what prompted the restriction to 1.34?

Edit: in the 20240502t200454z-f40f39d13 Debian image, I find tar 1.35+dfsg-3 installed, so somehow/somewhere it must be updating 😞

Edit 2: Found #21373

Edit 3: Un-ping @edsantiago I found the source of the trouble and staged a fix in containers/automation_images#349

[cevich]

Keeping this a draft since I don't know if "Temporarily disable rootless debian e2e testing" will work.

[cevich]

@mheon Question when you get a chance. With this PR, CI is setup to test F39 with BoltDB. Is this even supported and/or should we continue? Reminder: We have CI running on release-branches daily. There's also the upgrade test w/ the comment:

              # 2024-02: as long as possible/reasonable, try to keep
              #   one version < 4.8 so we can test boltdb. v4.3.1 is
              #   the lowest we can go right now, builds before that
              #   have netavark <1.4 which hangs on f39 kernel (#21863).
              PODMAN_UPGRADE_FROM: v4.3.1

The other reason I ask is largely cosmetic and to simplify our scripts: Almost every Cirrus task name includes 'boltdb' or 'sqlite' at the end. Making them eyeball unfriendly. ISTM we could maybe get rid of that, and the $CI_DESIRED_DATABASE envar + special handling. Could also be done in a followup PR.

[mheon]

Yes, it's still sensible. We've only just disabled the creation of BoltDB databases in F40 (F39 never gets 5.0, so it can still use BoltDB without restrictions, even if it's not the default). We'll want to keep it around on F40 for folks upgrading, as well. Probably obligated to keep it around until next year, when we can consider what to do about the old DB code.

[Luap99]

LGTM

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cevich, Luap99

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [Luap99]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[edsantiago]

WTF????? Is this the pasta thing? Do we have a timeline for resolving this?

[edsantiago]

I mean, IIRC the bug had something to do with the move to /var/tmp, did anyone consider setting an explicit TMPDIR=/tmp in the debian CI?

[Luap99]

Yes my tmpfs PR will fix it by using /tmp/... Otherwise I need to dig into apparmor

[edsantiago]

Sorry, I can't approve this just yet. I prefer to bandaid tests, not disable them. Maybe #22533 will pass and merge very quickly, but maybe there will be problems, and I don't want a no-debian-testing window.

I submitted #22639 as an alternative. If that fails, I'll reconsider.

[edsantiago]

Pasta is completely broken on debian. I see no choice but to hold my nose and merge this. Good luck to everyone working on fixes.

/lgtm

[cevich]

Pasta is completely broken on debian

Ya, it was totally FUBAR. Paul's got a handle on it 😄