Lustre mount via Ansible for SMHP Slurm LCS #682

amanshanbhag · 2025-05-15T22:47:32Z

Issue #, if available: #642

Description of changes: Using Ansible to mount FSxL file system -- simplifying the LCS by getting rid of all the redundancies. Simple ansible.posix.mount

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

amanshanbhag · 2025-05-15T22:47:56Z

Testing once before ready to merge

amanshanbhag · 2025-05-15T23:19:42Z

Seems like there's a difference in the way OpenZFS (NFS) and Lustre interact with the x-systemd.automount option. The FSxL file system isn't mounted on boot, only when accessed. Debating whether to remove that option

amanshanbhag · 2025-05-15T23:36:07Z

I see the entry in /etc/fstab, just not in df -Th...

amanshanbhag · 2025-05-15T23:49:08Z

Weird, I can mount it when run directly on the instance, but it doesn't seem to mount as a LifeCycleScript (with and without the automount option)

mhuguesaws · 2025-05-16T14:12:37Z

It's mounted on access since it is a systemd automount

amanshanbhag · 2025-05-16T14:41:11Z

Yeah, but when I try to mimic access in the LCS (egs. ls -la /fsx/ubuntu), I still don't see it mounted. I understand that an automount fs (and fstab entry) is created prior, and only mounted on access

mhuguesaws · 2025-05-16T16:19:12Z

Check the systemd daemon status
systemctl status fsx.automount

amanshanbhag · 2025-05-16T17:58:50Z

Fixed.

mhuguesaws

LGTM

…oon); Adding -y flag to ssh-keygen (Issue #676)

* Fixing Race Conditions reported in #674 * Fixing lifecycle_script.py by removing ansible call (PR #682 coming soon); Adding -y flag to ssh-keygen (Issue #676) * Converting 1/0 int --> str based on class def * Adding --allow-downgrades to install_docker.sh to account for pinned version (< what's available in AMI). NVIDIA/nvidia-container-toolkit#1093 * Reverting back -y for ssh-keygen, moving to gracious error handling + continue instead * Adding or to previous error handling

* Adding ssh keys to additional (OZFS at `/home`) file system (#700) * Testing ssh with ozfs LCS * Replacing ********* with localhost in OZFS mount script (#696) (#699) * Update gen-keypair-ubuntu.sh * Update nccl-tests.Dockerfile * [feat]: Add describe alarm permissions in the execution role for Rolling Update Autorollback. (#698) Co-authored-by: Vinay Devadiga <[email protected]> * fsdp k8s yaml to use c10d rdzv backend instead of etcd, updated readm… (#701) * fsdp k8s yaml to use c10d rdzv backend instead of etcd, updated readme, and cleaned yaml arguments * updating PR from feedback: using generate-sbatch-training-files, and ReadMe updated * small readme fix in envsubst line * model file name change, generate training file pytorchjob name update * Update 3.test_cases/pytorch/FSDP/kubernetes/README.md --------- Co-authored-by: mhuguesaws <[email protected]> * Fixing Race Conditions reported in #674 (#703) * Fixing Race Conditions reported in #674 * Fixing lifecycle_script.py by removing ansible call (PR #682 coming soon); Adding -y flag to ssh-keygen (Issue #676) * Converting 1/0 int --> str based on class def * Adding --allow-downgrades to install_docker.sh to account for pinned version (< what's available in AMI). NVIDIA/nvidia-container-toolkit#1093 * Reverting back -y for ssh-keygen, moving to gracious error handling + continue instead * Adding or to previous error handling --------- Co-authored-by: mhuguesaws <[email protected]> Co-authored-by: Vinay Devadiga <[email protected]> Co-authored-by: Vinay Devadiga <[email protected]> Co-authored-by: mvinci12 <[email protected]>

amanshanbhag · 2025-05-28T19:44:31Z

Tested with:

Cluster with Head Node + Worker Nodes + only FSxL
Cluster with Login Node + Worker Nodes + FSxL + FSxOpenZFS

Works as expected.

KeitaW

Generally looks good to me. Reviewed files under 1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-config. Suggested few changes for possible improvements.

1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-config/mount_fsx.sh

KeitaW · 2025-06-04T01:27:19Z

1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-config/mount_fsx_openzfs.sh

set -eux if we apply the aforementioned change.

1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-config/utils/fsx_ubuntu.sh

1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-config/utils/gen-keypair-ubuntu.sh

mhuguesaws

Please rebase the branch to only show the code changes. There are unrelated file change in this PR.

mhuguesaws · 2025-06-06T14:30:33Z

Still waiting on rebase to resolve conflicts.

…fig/mount_fsx.sh Co-authored-by: Keita Watanabe <[email protected]>

…fig/utils/fsx_ubuntu.sh Co-authored-by: Keita Watanabe <[email protected]>

…fig/utils/gen-keypair-ubuntu.sh Co-authored-by: Keita Watanabe <[email protected]>

amanshanbhag · 2025-06-09T20:32:22Z

Conflicts with main resolved

nghtm · 2025-05-28T20:28:48Z

1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-config/lifecycle_script.py

@@ -160,6 +160,8 @@ def main(args):
    params = ProvisioningParameters(args.provisioning_parameters)
    resource_config = ResourceConfig(args.resource_config)

+    ExecuteBashScript("./utils/install_ansible.sh").run()


Where is this getting installed? need architecture diagram and readme

Please clarify, does ansible get installed on every node?

Architecture diagram for what? Yes, ansible, the package, is installed on every node (hence localhost to mount in inventory)

nghtm · 2025-06-17T22:44:41Z

1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-config/mount_fsx_openzfs.sh

@@ -10,9 +10,6 @@ FSX_OPENZFS_DNS_NAME="$1"
 OPENZFS_MOUNT_POINT="$2"
 NFS_VERSION=4.2


Does this need to be hardcoded? Can we add a comment why Version 4.2 is used

We need to pin versions. For everything.

nghtm · 2025-06-17T22:45:54Z

1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-config/utils/install_ansible.sh

+{
+    apt-get update
+    # apt-get install -y ansible=$ANSIBLE_VERSION
+    apt-get install -y python3-pip


do we need retry on the network call, similar to what we have with other calls that was due to issues observed with u22.04 AMI. How many nodes has this script been tested on?

nghtm · 2025-06-17T22:46:40Z

1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-config/utils/install_docker.sh

@@ -27,6 +27,7 @@ echo \
 $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
 apt-get -y -o DPkg::Lock::Timeout=120 update


do we need to add backoff on these apt-get calls? how many nodes has this been tested on?

nghtm · 2025-06-17T22:47:19Z

Can you please clarify before merging, what advantages does moving to ansible bring over bash for HyperPod Lifecycle scripts?

Lustre mount via Ansible

8a73b80

amanshanbhag requested a review from mhuguesaws May 15, 2025 22:48

Fixed automount bug for fsxl + ansible

15cc3a8

Ansible for systemd calls

715c6e4

mhuguesaws approved these changes May 22, 2025

View reviewed changes

mhuguesaws self-assigned this May 22, 2025

amanshanbhag and others added 2 commits May 27, 2025 10:54

Resolving ozfs conflicts

b6ca6d5

Merge branch 'main' into lustre-ansible

51cd4e9

amanshanbhag added a commit that referenced this pull request May 27, 2025

Fixing lifecycle_script.py by removing ansible call (PR #682 coming s…

96d9046

…oon); Adding -y flag to ssh-keygen (Issue #676)

amanshanbhag and others added 5 commits May 28, 2025 14:02

Syncing with latest pushes to repo

4918505

Merging from main into local branch

97885b9

Merge changes from main

0d2efa4

small fix for #705

0a1aeab

amanshanbhag mentioned this pull request May 28, 2025

install_docker.sh errors out during LCS execution #705

Closed

amanshanbhag requested a review from KeitaW June 3, 2025 19:53

KeitaW approved these changes Jun 4, 2025

View reviewed changes

mhuguesaws requested changes Jun 4, 2025

View reviewed changes

Update 1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-con…

1003b6b

…fig/mount_fsx.sh Co-authored-by: Keita Watanabe <[email protected]>

amanshanbhag and others added 9 commits June 9, 2025 10:01

Update 1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-con…

357d413

…fig/mount_fsx.sh Co-authored-by: Keita Watanabe <[email protected]>

Update 1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-con…

a2f0a8b

…fig/utils/fsx_ubuntu.sh Co-authored-by: Keita Watanabe <[email protected]>

Update 1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-con…

00045a2

…fig/utils/fsx_ubuntu.sh Co-authored-by: Keita Watanabe <[email protected]>

Update 1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-con…

5217c0b

…fig/utils/gen-keypair-ubuntu.sh Co-authored-by: Keita Watanabe <[email protected]>

Update 1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-con…

73146bc

…fig/utils/gen-keypair-ubuntu.sh Co-authored-by: Keita Watanabe <[email protected]>

Update 1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-con…

269efa6

…fig/utils/gen-keypair-ubuntu.sh Co-authored-by: Keita Watanabe <[email protected]>

Update 1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-con…

72fad2b

…fig/utils/gen-keypair-ubuntu.sh Co-authored-by: Keita Watanabe <[email protected]>

Update 1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-con…

36498ea

…fig/utils/gen-keypair-ubuntu.sh Co-authored-by: Keita Watanabe <[email protected]>

Merge branch 'main' into lustre-ansible

ce3022b

mhuguesaws approved these changes Jun 11, 2025

View reviewed changes

nghtm reviewed Jun 17, 2025

View reviewed changes

		@@ -10,9 +10,6 @@ FSX_OPENZFS_DNS_NAME="$1"
		OPENZFS_MOUNT_POINT="$2"
		NFS_VERSION=4.2

		@@ -27,6 +27,7 @@ echo \
		$(lsb_release -cs) stable" \| tee /etc/apt/sources.list.d/docker.list > /dev/null
		apt-get -y -o DPkg::Lock::Timeout=120 update

Lustre mount via Ansible for SMHP Slurm LCS #682

Are you sure you want to change the base?

Lustre mount via Ansible for SMHP Slurm LCS #682

Uh oh!

Conversation

amanshanbhag commented May 15, 2025

Uh oh!

amanshanbhag commented May 15, 2025

Uh oh!

amanshanbhag commented May 15, 2025

Uh oh!

amanshanbhag commented May 15, 2025

Uh oh!

amanshanbhag commented May 15, 2025

Uh oh!

mhuguesaws commented May 16, 2025

Uh oh!

amanshanbhag commented May 16, 2025

Uh oh!

mhuguesaws commented May 16, 2025

Uh oh!

amanshanbhag commented May 16, 2025

Uh oh!

mhuguesaws left a comment

Choose a reason for hiding this comment

Uh oh!

amanshanbhag commented May 28, 2025

Uh oh!

KeitaW left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

KeitaW Jun 4, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mhuguesaws left a comment

Choose a reason for hiding this comment

Uh oh!

mhuguesaws commented Jun 6, 2025

Uh oh!

amanshanbhag commented Jun 9, 2025

Uh oh!

nghtm May 28, 2025

Choose a reason for hiding this comment

Uh oh!

nghtm Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

amanshanbhag Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nghtm Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

amanshanbhag Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

nghtm Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

nghtm Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

nghtm commented Jun 17, 2025

Uh oh!

Uh oh!

amanshanbhag Jun 17, 2025 •

edited

Loading