trainer-operator

Standalone operator for Kubeflow Trainer v2. Reconciles the Trainer CR (components.platform.opendatahub.io/v1alpha1) and deploys upstream Kubeflow Trainer resources via kustomize rendering from opendatahub-io/trainer manifests.

Architecture

Manifest Pipeline

1. Build time: hack/get_trainer_manifests.sh fetches upstream trainer manifests into opt/manifests/
2. Dockerfile: copies upstream manifests into /opt/manifests-template/, runtimes into /opt/runtimes-template/, and imagestreams into /opt/imagestreams-template/
3. Runtime: copies templates to a writable work dir, applies RELATED_IMAGE env var overrides to params.env, renders the kustomize overlay, applies with Server-Side Apply

Reconcile Flow

• Managed: ensure finalizer → ensure namespace → render manifests → SSA apply → update status (Ready)
• Removed: ensure finalizer → GC cleanup (label-based discovery) → update status (NotReady)
• Deleted: GC cleanup → remove finalizer → CR deleted

Platform Utilities

The controller uses shared utilities from opendatahub-io/odh-platform-utilities:

• api/common — PlatformObject interface, ManagementSpec, Status, Condition types
• pkg/controller/conditions — Condition Manager with happiness recomputation
• pkg/controller/gc — GC Collector for label-based resource cleanup via discovery API
• pkg/render/kustomize — Kustomize manifest rendering
• pkg/resources — Server-Side Apply

Structure

api/v1alpha1/           CRD types implementing common.PlatformObject
internal/controller/    Reconciler, manifest rendering, params.env handling
cmd/main.go             Operator entrypoint
config/                 Kustomize manifests (CRDs, RBAC, manager deployment, samples)
hack/                   Manifest collection script (get_trainer_manifests.sh)
manifests/runtimes/     ClusterTrainingRuntime definitions (torch, training-hub)
manifests/imagestreams/ ImageStream definitions (CUDA, ROCm, CPU training images)
test/e2e/               End-to-end tests (Kind cluster)
test/e2e/ocp/           End-to-end tests (OCP/real cluster)
test/support/           Shared test client (Client wrapping kubernetes.Interface)
test/utils/             Shell command utilities (make, kind, cert-manager)

Development

Prerequisites

• Go 1.25+
• Podman (or Docker via CONTAINER_TOOL=docker)
• kubectl v1.28+
• Access to a Kubernetes v1.28+ cluster

Common Commands

make manifests          # Generate CRDs and RBAC from markers
make generate           # Generate DeepCopy methods
make fmt                # Format code
make vet                # Vet code
make lint               # Run linter
make lint-fix           # Run linter and auto-fix issues
make test               # Run unit tests (envtest)
make build              # Build the operator binary
make run                # Run operator locally against cluster
make docker-build       # Build container image (uses podman by default)
make install            # Install CRDs into cluster
make deploy IMG=<img>   # Deploy operator to cluster

To use Docker instead of Podman, set CONTAINER_TOOL=docker.

Running Tests

Unit tests (controller tests with envtest):

make test

E2E tests (creates a Kind cluster, deploys the operator, runs tests, tears down):

make test-e2e

The Kind cluster can also be managed separately:

make setup-test-e2e     # Create Kind cluster
make cleanup-test-e2e   # Tear down Kind cluster

OCP e2e tests (requires a cluster with the operator and JobSet controller deployed):

make test-e2e-ocp

Deploy to Cluster

make docker-build docker-push IMG=<some-registry>/odh-trainer-operator:tag
make install
make deploy IMG=<some-registry>/odh-trainer-operator:tag
kubectl apply -k config/samples/

Uninstall

kubectl delete -k config/samples/
make uninstall
make undeploy

Contributing

CRD Changes

1. Edit api/v1alpha1/trainer_types.go
2. Run make manifests generate to regenerate CRDs and DeepCopy methods
3. Update controller logic in internal/controller/trainer_controller.go

RBAC

RBAC rules are derived from // +kubebuilder:rbac: markers in the controller. After adding new markers, run make manifests to regenerate.

The controller SA must hold all permissions that upstream trainer ClusterRoles grant — Kubernetes RBAC escalation prevention blocks creating a ClusterRole with permissions the creator doesn't already have.

Before Committing

Run make lint after any code changes.

License

Copyright 2026.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.