GPU in Docker container becomes unavailable if not used for sometime

[matifali]

Using a Docker-based template that has access to GPU, If we leave the workspace in a running state and do not use the GPU actively. GPU becomes unavailable after some time.

(DL) atif@workspace:~$ nvidia-smi 
Failed to initialize NVML: Unknown Error

This might be because nvidia-container-runtime somehow stopped giving access to GPU to the running containers.

Similar issues

NVIDIA/nvidia-docker#1671
https://bbs.archlinux.org/viewtopic.php?id=266915

[mafredri]

@matifali since this isn't a Coder specific issue, I converted this into a discussion so the community can chip in with potential solutions for your issue.

[matifali]

ok. good for it.

[matifali]

removing runtime and using gpus in docker_container fixed it for me.

resource "docker_container" "workspace" {
  count = data.coder_workspace.me.start_count
  image = docker_image.aihwkit.image_id
  cpu_shares = var.cpu
  memory = "${var.ram*1024}"
  gpus = "all"
  # Uses lower() to avoid Docker restriction on container names.
  name = "coder-${data.coder_workspace.me.owner}-${lower(data.coder_workspace.me.name)}"
  # Hostname makes the shell more user friendly: coder@my-workspace:~$
  hostname = lower(data.coder_workspace.me.name)
  dns      = ["1.1.1.1"]
  # Use the docker gateway if the access URL is 127.0.0.1
  command = ["sh", "-c", replace(coder_agent.dev.init_script, "127.0.0.1", "host.docker.internal")]
  env     = ["CODER_AGENT_TOKEN=${coder_agent.dev.token}"]

  host {
    host = "host.docker.internal"
    ip   = "host-gateway"
  }
}

But after every coder update, we have to run systemctl damon-reload which somehow makes the GPU unavailble again and then a workspace stop and start are required to make GPU available again.

[matifali]

A sample template that used GPU can be accessed here. deeplearning coder template

[mafredri]

Thanks for sharing your discovery and templates @matifali! Does the GPU unavailability happen with a simple coder server restart as well, or only after systemctl daemon-reload? To my knowledge the server should not be playing a part in accessing the host GPU, so I wonder if there could be a docker/systemd bug triggered by daemon-reload. 🤔

[matifali]

I will test that report. Usually, a daemon-reload is required after every coder update, otherwise coder service fails to restart.

[matifali]

For record, I ended up moving Coder installation to another machine and connecting to the GPU machine via ssh in docker provider.

This decouples the Coder updates and resolves the issue of GPU being unavailable.