Add N/2 NCCL tests for K8s

[amanshanbhag]

Title says it all. To isolate a bad node, create a script to run N/2 tests (pairwise).

[KeitaW]

#531 for easier bad node isolation.

[mhuguesaws]

This is present in the workshop. What's new here https://catalog.workshops.aws/sagemaker-hyperpod/en-US/07-tips-and-tricks/06-nccl-tests?

[amanshanbhag]

EKS. Also, based on linked issue (#531) maybe we can put specific baseline numbers for instance types (instead of pass/fail tests)?

[mhuguesaws]

Usually, we check NVlink and network cards on a single instance. It can be achieved by having prolog for Slurm and init container for Kubernetes.