GPU driver issues on Vertex AI

[abhishek-ranjan-kindred]

I am running a simple Kedro pipeline on Vertex AI with a single node requiring a GPU. I modified the autogenerated config conf/base/vertexai.yml resources as follows:

  resources:

    # Default settings for the nodes
    __default__:
      nvidia.com/gpu: 1

But the run fails with this error: *** failed with error: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx

Questions:

1. Do these drivers need to be installed somewhere or does it come pre-installed in the "pod" that Vertex AI uses?
2. Is there a specific base container needed to run nodes with GPU? My container builds on FROM nvidia/cuda:11.1.1-cudnn8-devel-ubuntu18.04 .

[marrrcin]

Hi @abhishek-ranjan-kindred ,
we have already adressed this issue in #96 and it's pending release.
You can either build the package from develop branch on your own or wait until beginning of the next week when we will release it.
Thanks!

[abhishek-ranjan-kindred]

Thank you for the pointer! The develop branch with your PR works! There are some gotchas that might need fixing or documenting:

1. I needed to include node_selectors in the yml. Just with resources I got this error: "Empty acceleratorConfig.type. Both type and count in acceleratorConfig should be specified or none.". Details: "Empty acceleratorConfig.type. Both type and count in acceleratorConfig should be specified or none.">
2. Once I used node_selectors, using node tag worked smoothly without any issues.
3. When I used node name instead (my node name was segm_train) then it did not work because in Vertex AI the node name gets converted to segm-train (hyphen instead of underscore). I needed to change the node name in vertexai.yml accordingly and then it worked.

Here is my vertexai.yml snippet:

    # the actual node name in the python code is segm_train
    segm-train: 
      cpu: 8
      gpu: 1
      memory: 16Gi

    __default__:
      cpu: 8
      gpu: 1
      memory: 16Gi

  # without node_selectors section, it does not work
  node_selectors:

    segm-train:
      cloud.google.com/gke-accelerator: NVIDIA_TESLA_T4