Container Service for Kubernetes:Using AI Profiling via command line

Verify USDT availability

Python profiling depends on User Statically-Defined Tracing (USDT) in the Python interpreter. Run the following command inside your workload container:

python -c "import sysconfig; print(sysconfig.get_config_var('WITH_DTRACE'))"

The output must be 1 for Python profiling to work. If the output is 0 or empty, Python profiling is unavailable in that container.

Start profiling

1. Get the profiling pod name:

   kubectl get pod

   Expected output:

   NAME                                     READY   STATUS    RESTARTS   AGE
   ai-profiler-89bf5b305acf2ec-xxxxx        2/2     Running   0          1m

2. Open a shell in the profiling pod's debugger container:

   kubectl exec -ti ai-profiler-89bf5b305acf2ec-xxxxx -c debugger -- bash

3. List GPU processes and generate a profiling command template:

   If you plan to use Python-level profiling, set the environment variable shown at the end of the output before running the profiling command.

   llmtracker generateCommand

   Expected output:

   I0314 11:42:42.389890 2948136 generate.go:51] GPU PIDs in container:
   
   I0314 11:42:42.389997 2948136 generate.go:53] PID: xxxxx, Name: {"pid":xxxxx}
   I0314 11:42:42.390008 2948136 generate.go:69] The profiling command is:
   
   llmtracker profile\
   -p <ProcessID-To-Profiling>\
   -t <Profiling-Type(python,cuda,syscall,cpu or all)>\
   -o /tmp/data.json\
   -v 5\
   --cpu-buffer-size <CPU-Buffer-Size, recommand to 20>\
   --probe-file <Enable-CUDA-Lib-Profile-File>\
   -d <Duration-To-Profiling>\
   --delay <Delay-Time>\
   --enable-cuda-kernel <Enable-CUDA-Kenrel-Profile(true or none)>
   
   I0314 14:37:12.436071 3083714 generate.go:86] Profiling Python Path is: /usr/bin/python3.10. If you want to profiling Python, please ser the environment variable:
   export EBPF_USDT_PYTHON_PATH=/usr/bin/python3.10

4. Modify the generated command for your scenario and run it. The following parameters control profiling scope and duration: Difference between `-t cuda` and `--enable-cuda-kernel`: For all available parameters, run llmtracker profile -h.

   • -t cuda uses eBPF to record CUDA library symbol calls — the call time and parameters of each API function — to analyze the actual call behavior of the process.

   • --enable-cuda-kernel uses process injection to collect execution details of individual CUDA kernel functions, enabling inspection of task flow states on the GPU.

   Parameter	Default	Description
   -p	—	PID to profile. Can be specified multiple times to profile multiple processes.
   -t	—	Profiling type: python, cuda, syscall, cpu, or all.
   -o	/tmp/data.json	Output file path.
   -v	—	Log verbosity level.
   --cpu-buffer-size	20	CPU buffer size for eBPF data collection.
   --probe-file	—	JSON configuration file for CUDA library profiling. See CUDA library configuration file for the format or use the default template.
   -d	—	Profiling duration in seconds. We recommend keeping this value below 60s, as prolonged profiling durations may generate excessive data, which can lead to increased memory consumption and storage load.
   --delay	—	Delay before profiling starts, in seconds. Set this to at least 2 if CUDA kernel profiling is enabled.
   --enable-cuda-kernel	—	Set to true to enable CUDA kernel profiling.

5. Run the profiling command. The following example enables all profiling types (including CUDA kernel profiling), uses probe.json for CUDA library configuration, writes output to /tmp/data.json, delays 3 seconds before starting, and profiles for 5 seconds:

   export EBPF_USDT_PYTHON_PATH=/usr/bin/python3.10
   llmtracker profile -p xxxxx -t all -o /tmp/data.json -v 5 --enable-cuda-kernel true --cpu-buffer-size 20 --probe-file probe.json --delay 3 -d 5

Export results

Convert the result file to a standard format for timeline visualization:

llmtracker export -i /tmp/data.json -o /output/out.json --cupti-dir /tmp

Include --cupti-dir /tmp if the result file contains CUDA kernel profiling data.

Build a custom configuration

1. Use ldd to get the link dependencies of your target library and identify which library files to trace.

   

2. Extract symbol names from the target library. The following example uses libnccl.so:

   readelf -Ws libnccl.so.2 | grep pnccl

   Expected output:

   ...
      223: 00000000000557d0   650 FUNC    GLOBAL DEFAULT   11 pncclGroupStart
      224: 0000000000050200   243 FUNC    GLOBAL DEFAULT   11 pncclRedOpDestroy
      225: 0000000000062081   656 FUNC    GLOBAL DEFAULT   11 pncclCommAbort
      227: 000000000006320c   721 FUNC    GLOBAL DEFAULT   11 pncclCommUserRank
      228: 0000000000064ee0    20 FUNC    GLOBAL DEFAULT   11 pncclGetVersion
      231: 0000000000045f60  1778 FUNC    GLOBAL DEFAULT   11 pncclAllGather
      232: 00000000000604f8  1578 FUNC    GLOBAL DEFAULT   11 pncclCommInitAll
      233: 000000000004ff20   728 FUNC    GLOBAL DEFAULT   11 pncclRedOpCreatePreMulSum
      238: 0000000000074520   653 FUNC    GLOBAL DEFAULT   11 pncclCommDeregister
      240: 00000000000474b0    30 FUNC    GLOBAL DEFAULT   11 pncclBcast
      243: 000000000006173d   789 FUNC    GLOBAL DEFAULT   11 pncclCommFinalize
      244: 00000000000483d0  2019 FUNC    GLOBAL DEFAULT   11 pncclSend
   ...

3. Build the JSON configuration file. The file defines the information needed for Probe, including: the relative path of the target library file in the container for uprobes, the symbol of the method to monitor in the library for uprobes, and the symbol of the system method to monitor for kprobes.