AMD

• Join as per usual to cluster
• Enter iDRAC
• Graceful Shutdown
• Virtual Media > RHCOS_ > Connect Virtual Media > CD/DVD >RHCOS > Boot > Boot to virtual ISO > Reset System (Warm Boot)/(Cold Boot)
• Set the relevant Node network settings via nmtui and ensure that the network interfaces are bonded. Check that the Hostname is correct as per the authority's convention.
• Ignite the node:

  sudo coreos-installer install /dev/sdX --ignition-url http://<>/worker.ign --copy-network --insecure-ignition
  

• Create NFR

  apiVersion: nfd.openshift.io/v1alpha1
  kind: NodeFeatureRule
  metadata:
    name: amd-gpu-operator-nfdrule
    namespace: openshift-amd-gpu
  spec:
    rules:
      - name: amd-gpu
        labels:
          feature.node.kubernetes.io/amd-gpu: "true"
        matchAny:
          - matchFeatures:
              - feature: pci.device
                matchExpressions:
                  vendor: {op: In, value: ["1002"]}
                  device: {op: In, value: [
                    "75a3", # MI355X
                    "75a0", # MI350X
                    "74a5", # MI325X
                    "74a0", # MI300A
                    "74a1", # MI300X
                    "74a9", # MI300X-HF
                    "74bd", # MI300X-HF
                    "740f", # MI210
                    "7408", # MI250X
                    "740c", # MI250/MI250X
                    "738c", # MI100
                    "738e" # MI100
                  ]}
      - name: amd-vgpu
        labels:
          feature.node.kubernetes.io/amd-vgpu: "true"
        matchAny:
          - matchFeatures:
              - feature: pci.device
                matchExpressions:
                  vendor: {op: In, value: ["1002"]}
                  device: {op: In, value: [
                    "75b3", # MI355X VF
                    "75b0", # MI350X VF
                    "74b9", # MI325X VF
                    "74b5", # MI300X VF
                    "7410", # MI210 VF
                  ]}
  

• Amend the RunAIConfig via apiExplorer > runai
  • Add the nodeSelectorTerms
• Add this to a oc patch command for ease of editing.

  spec:   
    global:
       managedNodes:
         inclusionCriteria:
            nodeSelectorTerms:
            - matchExpressions:
              - key: nvidia.com/gpu.product  
                operator: Exists
  

oc patch runaiconfig runai-config -n runai --type='merge' -p '{
  "spec": {
    "global": {
      "managedNodes": {
        "inclusionCriteria": {
          "nodeSelectorTerms": [
            {
              "matchExpressions": [
                {
                  "key": "nvidia.com/gpu.product",
                  "operator": "Exists"
                }
              ]
            }
          ]
        }
      }
    }
  }
}'

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: amd-gpu
spec:
  machineConfigSelector:
    matchExpressions:
    - key: machineconfiguration.openshift.io/role
      operator: In
      values: ["worker","amd-gpu"]
  nodeSelector:
    matchExpressions:
    - key: node-role.kubernetes.io/amd-gpu
      operator: Exists
    - key: feature.node.kubernetes.io/amd-gpu
      operator: In
      values: ["true"]
    - key: node-role.kubernetes.io/worker
      operator: Exists

• This adds a blacklist to the modprobe so that the driver running on the OS level is managed by the AMD GPU operator instead.

  apiVersion: machineconfiguration.openshift.io/v1                                 
  kind: MachineConfig                                                              
  metadata:                                                                        
    name: amdgpu-module-blacklist                                                  
    labels:                                                                        
      machineconfiguration.openshift.io/role: amd-gpu  # Match MCP name here       
  spec:                                                                            
    config:                                                                        
      ignition:                                                                    
        version: 3.2.0                                                             
      storage:                                                                     
        files:                                                                     
          - path: "/etc/modprobe.d/amdgpu-blacklist.conf"                          
            mode: 420                                                              
            overwrite: true                                                        
            contents:                                                              
              source: "data:text/plain;base64,YmxhY2tsaXN0IGFtZGdwdQo="
  

• Create device config:

  apiVersion: amd.com/v1alpha1
  kind: DeviceConfig
  metadata:
    name: test-cr
    namespace: openshift-amd-gpu
  spec:
    driver:
      enable: true
      image: image-registry.openshift-image-registry.svc:5000/$MOD_NAMESPACE/amdgpu_kmod
      version: 6.2.2
    selector:
      "feature.node.kubernetes.io/amd-gpu": "true"
  

Prometheus Integration

in .spec,

metricsExporter:
  enable: true
  prometheus:
    serviceMonitor:
      enable: true
      interval: "60s" # Scrape frequency
      attachMetadata:
        node: true
      honorLabels: false
      honorTimestamps: true
      labels:
        release: prometheus-operator # Prometheus release label for target discovery

## With TLS
metricsExporter:
    prometheus:
        serviceMonitor:
            enable: true
            bearerTokenFile: "/var/run/secrets/kubernetes.io/serviceaccount/token" # Deprecated
            authorization:
                credentials:
                    name: metrics-token
                    key: token
            tlsConfig:
                insecureSkipVerify: false
                serverName: metrics-server.example.com
                ca:
                    configMap:
                        name: server-ca
                        key: ca.crt
                cert:
                    secret:
                        name: prometheus-client-cert
                        key: tls.crt
                keySecret:
                    name: prometheus-client-cert
                    key: tls.key

# KMM resources (Modules/Builds should appear)
oc get module -A
oc get pods -n openshift-kmm

# Operator components
oc get pods -n openshift-amd-gpu -o wide


# If stuck in Init, try: 
ssh core@<amd GPU>
dmesg

To test if metrics are scraping, go to Observability > Targets > filter by AMD. Alternatively, run the test json dashboard.

Troubleshooting

https://access.redhat.com/solutions/5315421
https://access.redhat.com/solutions/4970731