Skip to content

Update RockyLinux 8.10 inc. OpenHPC v2.10, disable act_pedit#994

Merged
sjpb merged 16 commits into
mainfrom
ft/bump_timestamps_2026-06-29
Jul 10, 2026
Merged

Update RockyLinux 8.10 inc. OpenHPC v2.10, disable act_pedit#994
sjpb merged 16 commits into
mainfrom
ft/bump_timestamps_2026-06-29

Conversation

@elelaysh

@elelaysh elelaysh commented Jun 29, 2026

Copy link
Copy Markdown
Contributor
  • Update system packages on RockyLinux 8.10 including to kernel-core-4.18.0-553.136.1.el8_10 to fix CVE-2026-46331 (pedit-COW). Note system packages for RockyLinux 9.8 are not updated as kernel-core-5.14.0-687.17.1.el9_8 breaks DOCA-OFED.
  • Disable kernel module act_pedit (on both OSes) to mitigate this CVE for RockyLinux 9.8.
  • Update to OpenHPC v2.10 on RockyLinux 8.10, including Slurm major upgrade from v23.11.10 -> v25.05.8.
  • Update stackhpc.openhpc role to v1.6.0, mostly moving appliance defaults into the role. However it fixes slurmdbd upgrade logic and does change defaults for the following slurm.conf parameters:
    • RebootProgram: unset -> /sbin/reboot: Allows scontrol reboot to be used to reboot nodes. If slurm controlled rebuild is configured this is overriden.
    • StateSaveLocation: /var/spool/slurm -> /var/spool/slurmctld: Slurmd state is lost on rebuild anyway so this is backwards-compatible.

@elelaysh elelaysh requested a review from a team as a code owner June 29, 2026 11:05
@elelaysh elelaysh force-pushed the ft/bump_timestamps_2026-06-29 branch from 20b80e2 to 94d43ba Compare June 29, 2026 12:27
@elelaysh elelaysh changed the title Bump all repo timestamps 2026-06-29 Bump all repo timestamps on 2026-06-29 Jun 29, 2026
@elelaysh

Copy link
Copy Markdown
Contributor Author

⚠️ the RL 9.8 kernel revision breaks nvidia DOCA 3.2.2

This line in Rocky changelog for 5.14.0-687.16.1.el9_8:

  • dpll: add fractional frequency offset to pin-parent-device (Ivan Vecera) [RHEL-175823]

introducing this commit in upstream linux

@elelaysh

Copy link
Copy Markdown
Contributor Author

RH errata referenced from Rocky Linux changelog: https://access.redhat.com/errata/RHSA-2026:27789

@elelaysh elelaysh force-pushed the ft/bump_timestamps_2026-06-29 branch from 94d43ba to ad291eb Compare July 1, 2026 12:29
@elelaysh elelaysh changed the title Bump all repo timestamps on 2026-06-29 Bump RL8 repo timestamps on 2026-07-01 Jul 1, 2026
@elelaysh elelaysh self-assigned this Jul 1, 2026
elelaysh added 2 commits July 2, 2026 14:47
Not updating RL98 because the 5.14.0-687.17.1.el9_8 kernel
breaks DOCA OFED (change interface of dpll_pin_ops.ffo_get
in commit 9c11fcb of linux)

Not updating OHPC timestamps: OHPC 2.10 has an upgraded Slurm 25.11
which is too much of an upgrade to carry for this PR.
@elelaysh elelaysh force-pushed the ft/bump_timestamps_2026-06-29 branch from b5817d0 to dc51b04 Compare July 2, 2026 12:48
@sjpb

sjpb commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Image build: https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/28871145182

Needed b/c #997 should have updated the image but didn't; took the opportunity to bump the RL8 timestamps too.

@sjpb

sjpb commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Slurm db backup failed for RL8 with
/bin/sh: 1: /usr/bin/python3.9: not found
The weird thing is this is delegated to localhost, no become, so that should be the runner's python 3.12 venv.
Debugging via last commit ...

@sjpb sjpb force-pushed the ft/bump_timestamps_2026-06-29 branch from e861aa7 to fffa68a Compare July 8, 2026 12:33
@sjpb

sjpb commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Ok so here's what's happening:

  1. On the remote host, we use python3.9 (i.e. not default system python) to install pymysql so we get a recent version:
    ansible_python_interpreter: /usr/bin/python3.9
  2. So when we want to use community.mysql.mysql_query during the upgrade, we need to use the same interpreter: https://github.com/stackhpc/ansible-role-openhpc/blob/c2aff51213e0b913f5fa905f09a7c113a0e44ffa/tasks/upgrade.yml#L6
  3. So we set the interpreter at the play level, to avoid having to modify the role (which TBH has moved on now anyway):
    ansible_python_interpreter: /usr/bin/python3.9
  4. But the actual backup task runs delegated to localhost, and should use the venv's python there so we get openstack CLI:
    # configure slurm database pre-upgrade backups:

What I'm not clear on is why this apparently worked ok for the RL9 / ohpc 3.x upgrade.

@sjpb

sjpb commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

So RL9 / openhpc 3.x upgrade was here.

The final RL9 run actually failed but unrelated, not sure why we merged TBH: https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/27609175731/job/81629755985

The previous run was here: https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/27609175731/job/81640923243 but skips the backup!

TASK [stackhpc.openhpc : Check if slurm database requires an upgrade] **********
...
Slurm Database already at the highest version '16'. No conversion needed.

Edit: OpenHPC confirmed they just added the warning re. db upgrade as a precaution. And the SchedMD upgrade docs don't actually mention anything about using slurmdbd -u. So, I am going to assume that actually there just was no DB upgrade needed between these versions, and hence we just didn't notice that our upgrade logic was broken.

@sjpb

sjpb commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

RL8 slurm db upgrade failed because pexpect was not installed. Fixing via openhpc role.

Image build: https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/29013896303

@sjpb sjpb changed the title Bump RL8 repo timestamps on 2026-07-01 Update RockyLinux 8.10 inc. OpenHPC v2.10, disable act_pedit Jul 9, 2026
@sjpb sjpb force-pushed the ft/bump_timestamps_2026-06-29 branch from 9cf9e6c to 9dedc0c Compare July 9, 2026 16:17
@sjpb sjpb force-pushed the ft/bump_timestamps_2026-06-29 branch from b759716 to 41da8cb Compare July 9, 2026 19:56

@wtripp180901 wtripp180901 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@sjpb sjpb merged commit aae120c into main Jul 10, 2026
34 checks passed
@sjpb sjpb deleted the ft/bump_timestamps_2026-06-29 branch July 10, 2026 08:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants