default to unlock action and simplify things

Author: sjpb

.github/workflows/stackhpc.yml

@@ -166,7 +166,7 @@ jobs:
         run: |
           . venv/bin/activate
           . environments/.stackhpc/activate
-          ansible-playbook --limit login,control ansible/adhoc/lock-unlock-instances.yml -e "lock_unlock_action=unlock"
+          ansible-playbook ansible/adhoc/unlock.yml
           cd "$STACKHPC_TF_DIR"
           tofu init
           tofu apply -auto-approve -var-file="${{ env.CI_CLOUD }}.tfvars"
@@ -257,7 +257,7 @@ jobs:
         run: |
           . venv/bin/activate
           . environments/.stackhpc/activate
-          ansible-playbook ansible/adhoc/lock-unlock-instances.yml -e "lock_unlock_action=unlock"
+          ansible-playbook ansible/adhoc/unlock.yml
           cd "$STACKHPC_TF_DIR"
           tofu destroy -auto-approve -var-file="${{ env.CI_CLOUD }}.tfvars" || echo "tofu failed in $STACKHPC_TF_DIR"
         if: ${{ success() || cancelled() }}

README.md

@@ -142,7 +142,9 @@ To configure the appliance, ensure the venv and the environment are [activated](
 ansible-playbook ansible/site.yml
 ```
 
-To prevent the cluster instances from being changed or `tofu destroy` running, this playbook begins by locking the OpenStack instances. Any subsequent desired changes to the OpenTofu state require running an unlocking playbook as detailed in the adhoc command section of [docs/operations.md](docs/operations.md).
+To prevent nodes from being inadvertently changed or destroyed this playbook
+locks the OpenStack instances. Any subsequent changes to instances (e.g. via
+OpenTofu) require running an unlocking playbook as described in [docs/operations.md#ad-hoc-commands-and-playbooks](docs/operations.md#ad-hoc-commands-and-playbooks).
 
 Once `site.yml` completes you can log in to the cluster using:
 

ansible/adhoc/lock-unlock-instances.yml

@@ -8,17 +8,18 @@
 
 # See docs/slurm-controlled-rebuild.md.
 
-- name: Unlock compute instances for rebuild
+- name: Unlock compute instances
   vars:
-    lock_unlock_action: unlock
-    lock_unlock_hosts: compute
-  ansible.builtin.import_playbook: lock-unlock-instances.yml
+    unlock_hosts: compute
+  ansible.builtin.import_playbook: unlock.yml
 
 - hosts: login
   run_once: true
   gather_facts: false
   tasks:
-    - name: Run slurm-controlled rebuild
+    - name: Start slurm-controlled rebuild
       ansible.builtin.import_role:
         name: rebuild
         tasks_from: rebuild.yml
+
+# TODO: how do we lock the compute nodes again??

ansible/adhoc/rebuild-via-slurm.yml

@@ -1,19 +1,19 @@
 ---
 # Rebuild hosts with a specified image from OpenStack.
 #
+# NB: This is provided for development use only, and is not normally required
+# NB: Run the unlock playbook as shown below first
+#
 # Use ansible's -v output to see output.
 # Use --limit to control which hosts to rebuild (either specific hosts or the <cluster_name>_<partition_name> groups defining partitions).
 # Optionally, supply `-e rebuild_image=<image_name_or_id>` to define a specific image, otherwise the current image is reused.
 #
-# After running site.yml, all instances are locked, so to run the rebuild.yml, the unlock playbook must be run:
-#   ansible-playbook ansible/adhoc/lock-unlock-instances.yml -e "lock_unlock_action=unlock"
-# Similarly to rebuild, --limit can be used to control which hosts to unlock.
-#
 # NOTE: If a hostvar `instance_id` is defined this is used to select hosts.
 #       Otherwise the hostname is used and this must be unique, which may not be the case e.g. if using identically-named staging and production hosts.
 #
-# Example:
-#   ansible-playbook -v --limit ohpc_compute ansible/adhoc/rebuild.yml -e rebuild_image=openhpc_v2.3
+# Example of just rebuilding login nodes back to current image:
+#   ansible-playbook -v --limit login ansible/adhoc/unlock.yml
+#   ansible-playbook -v --limit login ansible/adhoc/rebuild.yml
 
 - hosts: cluster
   become: false

ansible/adhoc/rebuild.yml

@@ -0,0 +1,23 @@
+---
+# Lock or unlock cluster instances
+
+# Variable `unlock_action` may be 'unlock' (default) or 'lock'
+
+# A manual run is required before any action which will modify instances:
+# - ansible-playbook ansible/adhoc/rebuild.yml
+# - tofu apply
+# - tofu destroy
+# e.g.:
+#   ansible-playbook ansible/adhoc/lock.yml
+#
+# Instances are automatically locked by site.yml and unlocked by ansible/adhoc/rebuild-via-slurm.yml
+
+- hosts: "{{ unlock_hosts | default('cluster') }}"
+  gather_facts: false
+  become: false
+  tasks:
+    - name: "{{ unlock_action | default('unlock') | capitalize }} instances"
+      openstack.cloud.server_action:
+        action: "{{ unlock_action | default('unlock') }}"
+        server: "{{ instance_id }}"
+      delegate_to: localhost

ansible/adhoc/unlock.yml

@@ -2,8 +2,9 @@
 
 - ansible.builtin.import_playbook: check-production.yml
 
-- name: Lock cluster instances
-  ansible.builtin.import_playbook: adhoc/lock-unlock-instances.yml
+- ansible.builtin.import_playbook: adhoc/unlock.yml
+  vars:
+    unlock_action: lock
 
 - name: Run pre.yml hook
   vars:

ansible/site.yml

@@ -12,8 +12,8 @@ In summary, the way this functionality works is as follows:
 
 1. The image references(s) are manually updated in the OpenTofu configuration
    in the normal way.
-2. The adhoc playbook `lock-unlock-instances.yml` is run limited to control and login
-   nodes, with `lock_unlock_action=unlock` to allow the nodes to be rebuilt.
+2. The adhoc playbook `unlock.yml` is run to allow the login and control node
+   instances to be modified.
 3. `tofu apply` is run which rebuilds the login and control nodes to the new
    image(s). The new image reference for compute nodes is ignored, but is
    written into the hosts inventory file (and is therefore available as an
@@ -27,10 +27,11 @@ In summary, the way this functionality works is as follows:
    - Configures an application credential and helper programs on the control
      node, using the [rebuild](../../ansible/roles/rebuild/README.md) role.
 5. An admin submits Slurm jobs, one for each node, to a special "rebuild"
-   partition using the adhoc playbook `rebuild-via-slurm.yml`. Because this partition
-   has higher priority than the partitions normal users can use, these rebuild jobs
-   become the next job in the queue for every node (although any jobs currently running
-   will complete as normal).
+   partition using the adhoc playbook `rebuild-via-slurm.yml` which also unlocks
+   the compute instances. Because this partition has higher priority than the
+   partitions normal users can use, these rebuild jobs become the next job in
+   the queue for every node (although any jobs currently running will complete
+   as normal).
 6. Because these rebuild jobs have the `--reboot` flag set, before launching them
    the Slurm control node runs a [RebootProgram](https://slurm.schedmd.com/slurm.conf.html#OPT_RebootProgram)
    which compares the current image for the node to the one in the cluster
@@ -45,6 +46,10 @@ In summary, the way this functionality works is as follows:
    registers the node as having finished rebooting. It then launches the actual
    job, which does not do anything.
 
+TODO: need to relock the instances afterwards!
+TODO: mention the playbook to check rebuild state?
+TODO: maybe we should default to only unlocking the compute/login?
+
 ## Prerequsites
 
 To enable a compute node to rejoin the cluster after a rebuild, functionality

docs/experimental/slurm-controlled-rebuild.md

@@ -212,9 +212,9 @@ ansible-playbook ansible/adhoc/$PLAYBOOK
 Currently they include the following (see each playbook for links to documentation):
 
 - `hpctests.yml`: MPI-based cluster tests for latency, bandwidth and floating point performance.
-- `lock-unlock-instances.yml`: Lock cluster instances for preventing tofu changes, or unlock to allow changes.
+- `unlock.yml`: Unlock or lock nodes to allow/prevent changes to instances.
 - `rebuild.yml`: Rebuild nodes with existing or new images (NB: this is intended for development not for re-imaging nodes on an in-production cluster).
-  Requires `lock-unlock-instances.yml` be run first.
+  Requires `unlock.yml` be run first.
 - `restart-slurm.yml`: Restart all Slurm daemons in the correct order.
 - `update-packages.yml`: Update specified packages on cluster nodes (NB: not recommended for routine use).