|
1 | 1 | <!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. |
2 | 2 | SPDX-License-Identifier: OpenMDW-1.1 --> |
3 | 3 |
|
4 | | -# Cosmos3 Certified NIM Helm deployment |
| 4 | +# Deploy the Cosmos3 Certified NIM with Helm |
5 | 5 |
|
6 | | -Helm deployment guidance is not included in this cookbook yet. Do not adapt a |
7 | | -chart or values file from another NIM. Use the documented |
8 | | -[Docker deployment](deployment.md) for the current Cosmos3 Certified NIM image. |
| 6 | +Use this page to deploy the Cosmos3 Certified NIM on Kubernetes with the shared |
| 7 | +`nim-wfm` Helm chart. Model selection, hardware floors, and verification are |
| 8 | +the same as in [Deployment](deployment.md). |
| 9 | + |
| 10 | +## Prerequisites |
| 11 | + |
| 12 | +- A Kubernetes cluster with GPU nodes that match a row in the |
| 13 | + [Support matrix](support-matrix.md), with the |
| 14 | + [NVIDIA GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/) |
| 15 | + installed. |
| 16 | +- Helm and `kubectl` access to the target namespace. The commands below use the |
| 17 | + default namespace. |
| 18 | +- A storage class for the persistent model cache. |
| 19 | +- `NGC_API_KEY` exported as described in |
| 20 | + [Network and NGC access](prerequisites.md#network-and-ngc-access). |
| 21 | + |
| 22 | +## Create secrets |
| 23 | + |
| 24 | +The chart needs an image pull secret and a generic secret with a key named |
| 25 | +`NGC_API_KEY`: |
| 26 | + |
| 27 | +```bash |
| 28 | +kubectl create secret docker-registry ngc-secret \ |
| 29 | + --docker-server=nvcr.io \ |
| 30 | + --docker-username='$oauthtoken' \ |
| 31 | + --docker-password="$NGC_API_KEY" |
| 32 | +kubectl create secret generic ngc-api \ |
| 33 | + --from-literal=NGC_API_KEY="$NGC_API_KEY" |
| 34 | +``` |
| 35 | + |
| 36 | +## Fetch the chart |
| 37 | + |
| 38 | +This page uses `nim-wfm` chart version 1.3.0: |
| 39 | + |
| 40 | +```bash |
| 41 | +export NIM_WFM_VERSION='1.3.0' |
| 42 | +helm fetch "https://helm.ngc.nvidia.com/nim/charts/nim-wfm-${NIM_WFM_VERSION}.tgz" \ |
| 43 | + --username='$oauthtoken' \ |
| 44 | + --password="$NGC_API_KEY" |
| 45 | +``` |
| 46 | + |
| 47 | +The chart README lists every value supported by that version: |
| 48 | + |
| 49 | +```bash |
| 50 | +helm show readme "nim-wfm-${NIM_WFM_VERSION}.tgz" |
| 51 | +helm show values "nim-wfm-${NIM_WFM_VERSION}.tgz" |
| 52 | +``` |
| 53 | + |
| 54 | +## Configure values |
| 55 | + |
| 56 | +Save the following as `cosmos3-values.yaml`. It deploys the Nano Generator, |
| 57 | +optimized for latency, on one GPU: |
| 58 | + |
| 59 | +```yaml |
| 60 | +image: |
| 61 | + repository: nvcr.io/nim/nvidia/cosmos3 |
| 62 | + tag: "2.0.0" |
| 63 | +imagePullSecrets: |
| 64 | + - name: ngc-secret |
| 65 | +model: |
| 66 | + ngcAPISecret: ngc-api |
| 67 | + nimCache: /opt/nim/.cache |
| 68 | +persistence: |
| 69 | + enabled: true |
| 70 | + size: 220Gi |
| 71 | +sharedMemory: |
| 72 | + sizeLimit: 16Gi |
| 73 | +resources: |
| 74 | + limits: |
| 75 | + nvidia.com/gpu: 1 |
| 76 | +env: |
| 77 | + - name: NIM_MODEL_TYPE |
| 78 | + value: generator |
| 79 | + - name: NIM_MODEL_VARIANT |
| 80 | + value: nano |
| 81 | + - name: NIM_PERF_PROFILE |
| 82 | + value: latency |
| 83 | +``` |
| 84 | +
|
| 85 | +Adjust it for the chosen configuration: |
| 86 | +
|
| 87 | +- **Model and GPUs:** Change `NIM_MODEL_VARIANT`, `NIM_PERF_PROFILE`, and |
| 88 | + `nvidia.com/gpu` together using the |
| 89 | + [Generator configurations](support-matrix.md#generator-configurations). The |
| 90 | + GPU count does not select a GPU type; use `nodeSelector` or `affinity` to |
| 91 | + target eligible GPU nodes, and `tolerations` if those nodes are tainted. On |
| 92 | + nodes with MIG enabled, request the MIG resource instead of |
| 93 | + `nvidia.com/gpu`, for example `nvidia.com/mig-4g.24gb: 2`. |
| 94 | +- **Cache size:** `220Gi` is a starting size for large model files. Set `persistence.storageClass` if the |
| 95 | + cluster has no default storage class. |
| 96 | +- **Memory:** A pod memory limit counts as system memory during profile |
| 97 | + selection. Keep any limit at or above the host RAM value of the selected row. |
| 98 | +- **Shared memory:** The chart mounts a memory-backed `emptyDir` at `/dev/shm` |
| 99 | + by default. `sharedMemory.sizeLimit: 16Gi` matches the 16 GiB that the Docker |
| 100 | + launch allocates. |
| 101 | +- **Pod user:** The chart runs the pod as UID and GID 1000 with |
| 102 | + `fsGroup: 1000` (`podSecurityContext`). The cache volume must be writable by |
| 103 | + that user; storage that ignores `fsGroup`, such as some NFS exports and |
| 104 | + `hostPath` directories, needs matching ownership. |
| 105 | + |
| 106 | +Set these through chart values rather than `env`: |
| 107 | + |
| 108 | +| Chart value | Replaces | |
| 109 | +| --- | --- | |
| 110 | +| `model.nimCache` | `NIM_CACHE_PATH` | |
| 111 | +| `model.ngcAPISecret` | `NGC_API_KEY` | |
| 112 | +| `model.apiPort` | `NIM_HTTP_API_PORT` | |
| 113 | +| `model.jsonLogging` | `NIM_LOGGING_JSONL` | |
| 114 | +| `model.logLevel` | `NIM_LOG_LEVEL` | |
| 115 | + |
| 116 | +Add other [Configuration](configuration.md) variables to `env`. |
| 117 | + |
| 118 | +To check profile compatibility before a cold download, run the Docker |
| 119 | +[pre-download profile preflight](deployment.md#run-the-pre-download-profile-preflight) |
| 120 | +on a host with the same GPUs and selectors. |
| 121 | + |
| 122 | +### Reasoner |
| 123 | + |
| 124 | +For Reasoner, replace `env` with the Reasoner selectors and choose the GPU |
| 125 | +count from the [Reasoner configurations](support-matrix.md#reasoner-configurations). |
| 126 | +Reasoner does not use `NIM_PERF_PROFILE`: |
| 127 | + |
| 128 | +```yaml |
| 129 | +env: |
| 130 | + - name: NIM_MODEL_TYPE |
| 131 | + value: reasoner |
| 132 | + - name: NIM_MODEL_VARIANT |
| 133 | + value: nano |
| 134 | +``` |
| 135 | + |
| 136 | +On DGX Spark/GB10 or Jetson AGX Thor nodes, also set |
| 137 | +`NIM_GPU_MEMORY_UTILIZATION` to `"0.80"` for image-only workloads or `"0.70"` |
| 138 | +for video or mixed-media workloads; see |
| 139 | +[Set the Reasoner memory share](deployment.md#set-the-reasoner-memory-share-on-unified-memory-systems). |
| 140 | + |
| 141 | +### OpenTelemetry |
| 142 | + |
| 143 | +To export traces and metrics over OTLP, add these entries to `env`. The |
| 144 | +endpoint assumes an OpenTelemetry Collector DaemonSet that listens on host |
| 145 | +port 4318; change `OTEL_EXPORTER_OTLP_ENDPOINT` for other collector layouts: |
| 146 | + |
| 147 | +```yaml |
| 148 | + - name: NIM_ENABLE_OTEL |
| 149 | + value: "1" |
| 150 | + - name: OTEL_SERVICE_NAME |
| 151 | + value: cosmos3 |
| 152 | + - name: OTEL_TRACES_EXPORTER |
| 153 | + value: otlp |
| 154 | + - name: OTEL_METRICS_EXPORTER |
| 155 | + value: otlp |
| 156 | + - name: HOST_IP |
| 157 | + valueFrom: |
| 158 | + fieldRef: |
| 159 | + fieldPath: status.hostIP |
| 160 | + - name: OTEL_EXPORTER_OTLP_ENDPOINT |
| 161 | + value: "http://$(HOST_IP):4318" |
| 162 | +``` |
| 163 | + |
| 164 | +`HOST_IP` must come before `OTEL_EXPORTER_OTLP_ENDPOINT` so that Kubernetes |
| 165 | +can expand it. |
| 166 | + |
| 167 | +### Other chart features |
| 168 | + |
| 169 | +The chart README documents these optional features; all are off by default: |
| 170 | + |
| 171 | +- **Prometheus metrics:** Set `metrics.serviceMonitor.enabled: true` to create |
| 172 | + a Prometheus Operator `ServiceMonitor` that scrapes `/v1/metrics` on the |
| 173 | + `http-api` port. See [Metrics](operations.md#metrics). |
| 174 | +- **Ingress:** `ingress.enabled`, `ingress.className`, `ingress.hosts`, and |
| 175 | + `ingress.tls`. The NIM does not authenticate requests, so protect any |
| 176 | + ingress outside the cluster. |
| 177 | +- **Autoscaling:** `autoscaling.enabled` with `minReplicas`, `maxReplicas`, and |
| 178 | + `metrics`. CPU and memory metrics are of limited use for scaling a NIM; use |
| 179 | + custom metrics, for example through `prometheus-adapter`. Each new replica |
| 180 | + must download or mount the model before it becomes ready; see |
| 181 | + [Storage and scaling](#storage-and-scaling). |
| 182 | + |
| 183 | +## Install |
| 184 | + |
| 185 | +Review the rendered manifests, then install the release: |
| 186 | + |
| 187 | +```bash |
| 188 | +helm template cosmos3 "nim-wfm-${NIM_WFM_VERSION}.tgz" -f cosmos3-values.yaml |
| 189 | +helm install cosmos3 "nim-wfm-${NIM_WFM_VERSION}.tgz" -f cosmos3-values.yaml |
| 190 | +kubectl get pods -w |
| 191 | +``` |
| 192 | + |
| 193 | +Cold download, materialization, and model load can take much longer than pod |
| 194 | +startup. Send requests only after the pod reports `Ready`. Kubernetes restarts |
| 195 | +the container if it is not ready within `startupProbe.initialDelaySeconds` plus |
| 196 | +`startupProbe.failureThreshold` × `startupProbe.periodSeconds`; raise |
| 197 | +`failureThreshold` for slow first downloads. |
| 198 | + |
| 199 | +## Verify |
| 200 | + |
| 201 | +The chart creates a `ClusterIP` Service with no ingress, and the NIM does not |
| 202 | +handle authentication. Port-forward the Service for testing: |
| 203 | + |
| 204 | +```bash |
| 205 | +kubectl port-forward service/cosmos3-nim-wfm 8000:http-api |
| 206 | +``` |
| 207 | + |
| 208 | +In another terminal: |
| 209 | + |
| 210 | +```bash |
| 211 | +export NIM_URL=http://localhost:8000 |
| 212 | +curl -fsS "$NIM_URL/v1/health/ready" |
| 213 | +curl -fsS "$NIM_URL/v1/metadata" | python3 -m json.tool |
| 214 | +``` |
| 215 | + |
| 216 | +Confirm that metadata reports `generator` and `/v1/infer`, or `reasoner` and |
| 217 | +`/v1/chat/completions`. Then run a first request from |
| 218 | +[Use an existing NIM endpoint](README.md#use-an-existing-nim-endpoint). |
| 219 | + |
| 220 | +## Storage and scaling |
| 221 | + |
| 222 | +The chart supports these cache options; use only one: |
| 223 | + |
| 224 | +| Option | Values | Notes | |
| 225 | +| --- | --- | --- | |
| 226 | +| StatefulSet volume claims | `persistence.enabled: true` with the default `statefulSet.enabled: true` | Default. Each replica gets its own claim and downloads the model before it becomes ready | |
| 227 | +| Shared volume claim | `persistence.enabled: true`, `statefulSet.enabled: false`, `persistence.accessMode: ReadWriteMany` | One cache shared by all replicas; needs a `ReadWriteMany` storage class | |
| 228 | +| Existing claim | `persistence.existingClaim: <claim-name>` | Reuse a pre-filled cache. Run one replica unless the claim is `ReadWriteMany` | |
| 229 | +| Direct NFS | `nfs.enabled: true`, `nfs.server`, `nfs.path` | Mount options cannot be set per pod; an NFS-backed claim is usually a better choice | |
| 230 | +| `hostPath` | `hostPath.enabled: true`, `hostPath.path` | Ties pods to one node and has security implications | |
| 231 | + |
| 232 | +Set `replicaCount` for a fixed number of replicas. With the default |
| 233 | +StatefulSet, you can pre-populate caches by scaling up to the maximum replica |
| 234 | +count once and then scaling down; the retained claims let later scale-ups |
| 235 | +start without a new download. |
| 236 | + |
| 237 | +With `statefulSet.enabled: false` and a `ReadWriteOnce` volume, scaling or |
| 238 | +rolling upgrades beyond one pod are likely to fail. Use a `ReadWriteMany` |
| 239 | +storage class, manually cloned `ReadOnlyMany` volumes of a pre-filled cache, |
| 240 | +or direct NFS. Without persistence, the cache is an `emptyDir` and every pod |
| 241 | +start downloads the model again. |
| 242 | + |
| 243 | +## Troubleshooting |
| 244 | + |
| 245 | +| Symptom | Check | |
| 246 | +| --- | --- | |
| 247 | +| Pod stays `Pending` | Run `kubectl describe pod <pod>` and check `Events` for insufficient GPUs, untolerated taints, or an unbound volume claim | |
| 248 | +| Pod restarts while preparing the model workspace | The startup probe ran out before the download finished; increase `startupProbe.failureThreshold` | |
| 249 | +| Scaling or upgrade fails with `statefulSet.enabled: false` | The cache volume is `ReadWriteOnce`; see [Storage and scaling](#storage-and-scaling) | |
| 250 | +| Profile selection fails | Compare GPU type, count, free memory, and pod memory limit with the [Support matrix](support-matrix.md); see [Troubleshooting](operations.md#troubleshooting) | |
| 251 | + |
| 252 | +## Remove the deployment |
| 253 | + |
| 254 | +```bash |
| 255 | +helm uninstall cosmos3 |
| 256 | +kubectl get pvc |
| 257 | +``` |
| 258 | + |
| 259 | +StatefulSet volume claims are retained by default and keep the model cache. |
| 260 | +Delete them and the `ngc-secret` and `ngc-api` secrets only when they are no |
| 261 | +longer needed. |
0 commit comments