Slow `site.yml` on v2.18, appears to be due to `prometheus.prometheus` collection roles

[jcwomack]

On a large virtualised Ansible Slurm appliance (>300 compute instances), moving from an earlier version to v2.18, we saw a significant increase in time to complete running site.yml (doubled from ~30 min to ~1h).

From observation it seemed like a large amount of time was spent in prometheus.prometheus collection roles, which appear to have been introduced in v2.16 via #943. The PR mentions something related to this: #943 (comment).

Possibly this is a misconfiguration on our part and there is something missing from the image that results in more tasks needing to run during site.yml than is necessary.

[sjpb]

Interesting. I don't think the linked comment should affect it.

If you run ansible prometheus -a hostname what do you see? It should only report from the control node, I think.

If that looks ok, could we see what tasks actually run? If you ran ansible-playbook ansible/monitoring.yml --tags prometheus we could get that - I don't think we need any verboseness, just to see which tasks are running.

[jcwomack]

You are right that it only shows the control node

$ ansible prometheus -m ansible.builtin.command -a 'hostname'
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
[WARNING]: Found both group and host with same name: control
control | CHANGED | rc=0 >>
control

In the monitoring playbook we also have

- name: Deploy node_exporter
  hosts: node_exporter
  tags: node_exporter
  tasks:
    - ansible.builtin.import_role:
        name: prometheus.prometheus.node_exporter

which calls tasks from prometheus.prometheus

$ ansible node_exporter -m ansible.builtin.command -a 'hostname'

produces a list of all ~300 compute nodes + control + login.

Would running the playbook with --check be sufficient? This is a live system so do not want to be running a playbook that could make changes outside of a maintenance window.