Upgrade prometheus stack to latest LTS versions

[elelaysh]

Fix issues found via grype; switch to (our fork of) the prometheus-community ansible collection.

• Updates all OS packages
• Cleanup of /tmp in ansible/cleanup.yml has been fixed
• Removes packages cockpit-system cockpit-bridge
• Replaces rclone binary with a no-op script to avoid vulnerabilities
• Alertmanager db now persisted in appliances state directory (so alert state e.g. silence will be persisted across control node rebuilds)
• Adds ignore list for grype vulnerability scanner, see .grype.yaml
• Upgrades to prometheus 3.5.3, alertmanager 0.32.1, node_exporter 1.11.1
• Pins workflows for compliance with organisation allowlist

Note the grype allowlist is at .grype.yaml and includes some vulnerabilities which have no fix at this time.

[elelaysh]

do we want to store checksums or are we happy to download them alongside binaries from github?

[elelaysh]

Shall I remove our cloudalchemy.node_exporter cloudalchemy.prometheus alertmanager in this PR?

[sjpb]

do we want to store checksums or are we happy to download them alongside binaries from github?
I think we should store checksums in ansible variables, along with versions.

Shall I remove our cloudalchemy.node_exporter cloudalchemy.prometheus alertmanager in this PR?
Yes I think so - for my reference the first two are just requirements.yml and alertmanager is an actual role.

Is all the actual config backward-compatible, across all 3x repos roles?

[elelaysh]

Build run: https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/24184216835

[elelaysh]

0690c0b build run https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/24235098749 from 0d874c7

[sjpb]

• Deployed cluster from main @ 602a830
• Ran hpctests:pingpong
• Updated to ade3802, ran setup-env
• Reimaged cluster to new image: ansible-playbook ansible/adhoc/rebuild.yml -e rebuild_image=openhpc-RL9-260410-0859-0d874c74
• Ran ansible compute -ba "touch /var/lib/ansible-init.done" b/c .stackhpc env configured for compute-init, and I'd accidently already started site.
• Ran site.yml - FAILS because it is trying to do:

- name: Configure alertmanager
      # TODO: compare with prometheus.prometheus.alertmanager
      ansible.builtin.include_role:
        name: alertmanager
        tasks_from: configure.yml

and that doesn't exist yet

• Confirmed pre-rebuild pingpong job shows in slurm job dashboard
• Confirmed do see network and CPU data when clicking thro to the job
• Ran hpctests: pingmatrix
• Confirmed job shows in slurm jobs dashboard
• Confirmed do see network and CPU data when clicking thro to the job

[sjpb]

@ 748f820, confirmed monitoring.yml does complete!

TODO: should check the prom propagate binaries approach - I think this seemed bad before for large clusters, with the old cloudalchemy roles and we sort of patched it out/disabled it? But I don't remember the details TBH, maybe something about multiple downloads. But hopefully whatever we're doing to just do the install during image build should make this ok, would just like to check.

[elelaysh]

Rebuild for 3086ebe: https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/24404000044

[elelaysh]

Is all the actual config backward-compatible, across all 3x repos roles?

It is backward-compatible for prometheus, node_exporter. alertmanager diverged more. See common/alertmanager.yml config.

[elelaysh]

TODO: should check the prom propagate binaries approach - I think this seemed bad before for large clusters, with the old cloudalchemy roles and we sort of patched it out/disabled it? But I don't remember the details TBH, maybe something about multiple downloads. But hopefully whatever we're doing to just do the install during image build should make this ok, would just like to check.

Please see our ansible-prometheus vs prometheus-community. They seem pretty much identical to me: download,checksum,unpack on ansible host, copy to all hosts. True: we wouldn't care if it was downloaded on each host due to the install.yml only run on the builder to produce the image.

[sjpb]

I think the initial PR comment should also note the following "incidental" changes:

• Updates all OS packages
• Cleanup of /tmp in ansible/cleanup.yml has been fixed
• Removes packages cockpit-system cockpit-bridge
• Alertmanager db now persisted in appliances state directory (so alert state e.g. silence will be persisted across control node rebuilds)
• Adds ignore list for grype vulnerability scanner

[sjpb]

As per separate discussion, I feel the python intepreter logic is really hard to reason about, and e.g. means pre.yml runs under different interpreters depending on how it is called, which seems like its going to cause difficult problems at some point.

Decided to just run mysql under 3.9, which is needed to avoid a vuln in in that package.

[sjpb]

CI failure for RL8:

TASK [Query grafana for expected hpctests jobs] ********************************
task path: /home/runner/work/ansible-slurm-appliance/ansible-slurm-appliance/ansible/ci/check_grafana.yml:14
Thursday 30 April 2026  09:39:32 +0000 (0:00:00.738)       0:00:00.782 ******** 
fatal: [slurmci-RL8-1030-control]: FAILED! => {}

MSG:

The conditional check '_expected_jobs | difference(_found_jobs) == []' failed. The error was: error while evaluating conditional (_expected_jobs | difference(_found_jobs) == []): {{ _slurm_stats_jobs.docs | map(attribute='JobName', default='(json error in slurmstats data)') }}: 'dict object' has no attribute 'docs'. 'dict object' has no attribute 'docs'. {{ _slurm_stats_jobs.docs | map(attribute='JobName', default='(json error in slurmstats data)') }}: 'dict object' has no attribute 'docs'. 'dict object' has no attribute 'docs'

But on that control node:

[root@slurmci-RL8-1030-control rocky]# sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
extra        up 60-00:00:0      0    n/a 
standard*    up 60-00:00:0      2   idle slurmci-RL8-1030-compute-[0-1]
rebuild      up      30:00      2   idle slurmci-RL8-1030-compute-[0-1]
[root@slurmci-RL8-1030-control rocky]# sacct
JobID           JobName  Partition    Account  AllocCPUS      State ExitCode 
------------ ---------- ---------- ---------- ---------- ---------- -------- 
1            pingpong.+   standard                     2  COMPLETED      0:0 
1.batch           batch                                1  COMPLETED      0:0 
1.0               orted                                2  COMPLETED      0:0 
2            rebuild-s+    rebuild       root          1  COMPLETED      0:0 
2.batch           batch                  root          1  COMPLETED      0:0 
3            rebuild-s+    rebuild       root          1  COMPLETED      0:0 
3.batch           batch                  root          1  COMPLETED      0:0 

and Grafana is show slurm jobs 1,2,3.

So no idea what's happened there really TBH.

[edit]

OK after updating ansible/ci/retrieve_inventory.yml and running against the CI cluster, I was able to run ansible-playbook ansible/ci/check_grafana.yml with no errors. So, transient network issue??

[elelaysh]

After a version using python3.9 interpreter as much as possible to avoid python 3.6 platform-python on RL8, we decided to only use it where platform-python was causing an issue: installing an old pymysql in the mysql role.

[elelaysh]

build for d58bac2 https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/25317786937

[elelaysh]

Build for 06706f3: https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/25388565343