RunAI Upgrade

1. Get Helm Files for Import

Run the following command to browse all available air-gapped packages:

curl -H "Authorization: Bearer <token>" "https://runai.jfrog.io/artifactory/api/storage/runai-airgapped-prod/?list"

Note

via their jfrog: https://runai.jfrog.io/ui/native/runai-airgapped-prod/runai-airgapped-package-2.XX.XX.tar.gz Get the auth from the authority/account manager.

Download the desired package:

curl -L -H "Authorization: Bearer <token>" -O "https://runai.jfrog.io/artifactory/runai-airgapped-prod/runai-airgapped-package-<VERSION>.tar.gz"

SSH Into a node with OC, Helm and Docker/Podman (Bastion in most cases)

Extract the downloaded package:

tar xvf runai-airgapped-package-<VERSION>.tar.gz

Push the Charts to Git (Optional):

curl --user $LDAP_USERNAME:$LDAP_PASSWORD -X POST --upload-file runai-$RUNAI_VERSION/charts/control-plane.tgz https://gitea.mlopslab.ncs/api/packages/mlopslab/helm/api/charts                   

curl --user $LDAP_USERNAME:$LDAP_PASSWORD -X POST --upload-file runai-$RUNAI_VERSION/charts/runai-cluster.tgz https://gitea.mlopslab.ncs/api/packages/mlopslab/helm/api/charts

Push to your image registry:

Note

No https:// and set the environment variables REGISTRY_USERNAME & REGISTRY_PASSWORD or use an REGISTRY_AUTH_FILE This step pushes to the image registry exported as an environment variable.

export REGISTRY_URL=<DOCKER REGISTRY ADDRESS>/runai
## NO "HTTPS://"
## Fallback: add in the REGISTRY_USERNAME and REGISTRY_PASSWORD env variables.
export REGISTRY_AUTH_FILE=$HOME/.docker/config.json
sudo ./setup.sh

2. Upgrade Instructions

Note

Backup current helm values before trying to run the helm upgrade command.

Backup Current Config:

helm get values runai-backend -n runai-backend > runai_control_plane_values.yaml

Refresh Repo Metadata

helm repo update
helm search repo runai --versions

Push Helm Upgrade to Control Plane

helm upgrade runai-backend control-plane.tgz -n runai-backend -f runai_control_plane_values.yaml --reset-then-reuse-values

Push upgrade to Cluster Get the installation instructions from Clusters > Select cluster > INSTALLATION INSTRUCTIONS > Run the helm upgrade commands.

helm repo add runai --force-update
helm repo update
helm upgrade -i runai-cluster <runai/runai-cluster> <runai-cluster.tgz> -n runai \
--set controlPlane.url=runai.apps.ocp.local \
--set controlPlane.clientSecret=<> \
--set cluster.uid=<> \
--set cluster.url=runai.apps.ocp.local --version="2.23.31" --create-namespace
--set global.customCA.enabled=true

3. Miscellaneous Patches

Enabling GPU Swap

oc patch -n runai runaiconfigs.run.ai/runai --type='merge' --patch '{"spec":{"global":{"core":{"swap":{"enabled": true}}}}}'

oc patch -n runai runaiconfigs.run.ai/runai --type='merge' --patch '{"spec":{"global":{"core":{"swap":{"biDirectional": {"enabled": true}}}}}}'

oc patch -n runai runaiconfigs.run.ai/runai --type='merge' --patch '{"spec":{"global":{"core":{"swap":{"mode": "mapped"}}}}}'

Node Level Scheduling

oc patch -n runai runaiconfigs.run.ai/runai --type='merge' --patch '{"spec":{"global":{"core":{"nodeScheduler":{"enabled": true}}}}}'

Select Resources → Node pools
Create a new node pool or edit an existing node pool
Under the Resource Utilization Optimization tab, change the number of workloads on each GPU to any value other than Not Enforced (i.e. 2, 3, 4, 5)

The Node Level Scheduler is now ready to be used on that node pool.

Note

In order for a workload to be considered by the Node Level Scheduler for rerouting, it must be submitted with a GPU Request and Limit where the Limit is larger than the Request.

apiVersion: v1
kind: Pod
metadata:
  annotations:
    user: test
    gpu-fraction: "0.5"
    gpu-fraction-num-devices: "2"
  labels:
    runai/queue: test
  name: multi-fractional-pod-job
  namespace: test
spec:
  containers:
  - image: gcr.io/run-ai-demo/quickstart-cuda
    imagePullPolicy: Always
    name: job
    env:
    - name: RUNAI_VERBOSE
      value: "1"
    - name: RUNAI_GPU_MEMORY_LIMIT
      value: "0.95"
    resources:
      limits:
        cpu: 200m
        memory: 200Mi
      requests:
        cpu: 100m
        memory: 100Mi
    securityContext:
      capabilities:
        drop: ["ALL"]
  schedulerName: runai-scheduler
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 5

Node Selector for Runai

global:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: node-role.kubernetes.io/runai-system
                operator: Exists

kubectl label nodes <node-name> node-role.kubernetes.io/runai-system=true
kubectl label nodes <node-name> node-role.kubernetes.io/runai-system=false

Flow of Helm, Gitea & Harbor

                   ┌──────────────────────────────────────────────┐
                   │              YOU (on bastion)                │
                   │  helm upgrade runai-cluster runai/runai-...   │
                   └──────────────────────────────────────────────┘
                                     │
                                     v
┌───────────────────────────────────────────────────────────────────────────┐
│ 1) HELM CLIENT (local)                                                    │
│    - Fetch chart (from Gitea Helm repo OR local .tgz)                     │
│    - Merge values.yaml + --set                                             │
│    - Render Kubernetes manifests (Deployments/Jobs/Services/CRDs/...)      │
│    NOTE: No images are pulled here                                         │
└───────────────────────────────────────────────────────────────────────────┘
                                     │
                                     v
┌───────────────────────────────────────────────────────────────────────────┐
│ 2) KUBERNETES API SERVER (via your kubeconfig context)                    │
│    - Helm sends rendered YAML to the cluster API                           │
│    - Cluster stores desired state (etcd)                                   │
│    - Controllers see new/changed workloads                                 │
└───────────────────────────────────────────────────────────────────────────┘
                                     │
                                     v
┌───────────────────────────────────────────────────────────────────────────┐
│ 3) CONTROLLERS + SCHEDULER                                                 │
│    - Create ReplicaSets/Pods from Deployments/StatefulSets                 │
│    - Scheduler assigns each Pod to a Node                                  │
└───────────────────────────────────────────────────────────────────────────┘
                                     │
                                     v
┌───────────────────────────────────────────────────────────────────────────┐
│ 4) NODE (kubelet + CRI-O)                                                  │
│    - kubelet sees Pod spec with image: <registry>/<repo>:<tag>             │
│    - CRI-O pulls image from registry referenced in Pod spec                │
│                                                                           │
│      IF image points to Harbor + exists + trusted TLS + pull secret OK     │
│          -> pull succeeds                                                  │
│          -> container starts                                               │
│          -> readiness probes pass                                          │
│      ELSE                                                                  │
│          -> ImagePullBackOff / ErrImagePull                                │
└───────────────────────────────────────────────────────────────────────────┘
                                     │
                                     v
┌───────────────────────────────────────────────────────────────────────────┐
│ 5) HELM (only if you used --wait)                                          │
│    - watches Kubernetes resources                                          │
│    - waits for Pods Ready / Jobs complete / hooks succeed                  │
│    - marks release DEPLOYED or FAILED                                      │
└───────────────────────────────────────────────────────────────────────────┘


Air-gapped “must-haves” before upgrade:
  [Gitea Helm repo]  -> chart reachable
  [Harbor registry]  -> ALL referenced images exist internally
  [values.yaml]      -> renders ONLY internal image references

Break glass reinstall commands

helm upgrade -i runai-backend runai/control-plane --version 2.22.74 -n runai-backend --set global.domain=runai.apps.ocp.mlopslab.ncs --set global.config.kubernetesDistribution=openshift --set global.customCA.enabled=true --set tenantsManager.config.adminUsername=test@run.ai --set tenantsManager.config.adminPassword="Abcd!234" -f custom-env.yaml

From Installation Instructions, append the following keys: 
--set global.customCA.enabled=true --set global.image.registry=repo.mlopslab.ncs/runai