Skip to content

RunAI Log Collection 24 3 26

(1) Workload-exporter logs

kubectl logs -n runai -l app=workload-exporter --tail=500 > workload-exporter.log

(2) Pod specs — one working vs one broken workload

Please compare a working workload pod vs. a broken workload pod:

A workload that SHOWS the GPU memory limit_

kubectl get pod <working-pod> -n <namespace> -o yaml > working-pod.yaml

A workload that DOES NOT show the GPU memory limit_

kubectl get pod <broken-pod> -n <namespace> -o yaml > broken-pod.yaml

(3) Prometheus metric check If you have access to the cluster's Prometheus (e.g., via port-forward), please run these queries and share the results:

runai_scheduler_gpu_memory_mb_limit_per_pod

runai_gpu_memory_mb_limit_per_workload

runai_requested_gpu_memory_mb_per_workload

If Prometheus access is not feasible, the pod specs from (2) will be sufficient for now.

(4) HAR recording 

This is still needed to determine whether the issue is on the data collection side or the UI rendering side. Please reproduce the issue on both a working and a broken workload and capture the HAR file.

Please refer it : https://support.zendesk.com/hc/en-us/articles/4408828867098-Workflow-Generating-a-HAR-file-for-troubleshooting 1. Open Chrome and go to the page where the issue occurs. 2. Look for the ⋮ button and select More Tools > Developer Tools. 3. Select the Network tab from the panel. You must keep the menu open while you reproduce the issue. Optional: If Zendesk Customer Support requests a HAR file with WebSockets, select the Socket option in the Network tab. Reload the browser to start seeing the traffic over the WebSocket. The Socket option does not capture all logs when you download the HAR file, but for display filter purposes and download WebSocket logs only. 4. Look for a round record button in the upper-left corner of the tab and ensure it is red. If it is grey, click the button once to start recording. 5. Check the Preserve log box. 6. Click the grey crossed circle button to clear existing logs from the network tab. 7. Reproduce the issue while the network requests record. 8. Click the download button, Export HAR, to download and save the file to your computer: Save as HAR with Content. 9. Upload the HAR file to the ticket for further investigation. See Instructions for sending HAR files.

(5) Quick questions :

  • Were the affected workloads created before or after the v2.20 → v2.23.31 upgrade?
  • Are the affected workloads using whole GPUs or fractional GPUs?
  • Are they using MIG (Multi-Instance GPU)?
  • What GPU type is in use (check nvidia.com/gpu.product node label)?