Intelligent Computing LINGJUN:Deployment and release guide

1.1 Deliverables

Runtime Kit consists of two layers that connect through a unified gRPC interface:

The interface is the only boundary between the two layers and the foundation for backward compatibility. Alibaba Cloud Lingjun provides the resource-layer capabilities, and you adapt the container layer based on the open-source components. All version compatibility conclusions in this topic are based on this interface.

Issue ownership and support channels: The gRPC interface is the boundary. Issues in the resource-layer Runtime itself are Alibaba Cloud Lingjun product issues. Submit a ticket to Alibaba Cloud Lingjun technical support. If you use the official precompiled versions of the open-source components directly, you can submit feedback through channels such as issues in the open-source repository. Implementations that you customize or adapt are maintained by you, and Alibaba Cloud Lingjun provides troubleshooting assistance as needed.

1.2 Prerequisites

1. Lingjun bare metal node: The Runtime package comes preinstalled with the Lingjun bare metal OS image and is available only in the Lingjun bare metal environment. Before you deploy the open-source components, the Runtime service on the node should already be running. You can confirm this by checking the socket files in the /run/amp/eKitC/xdm/sockets/ directory.

2. Kubernetes cluster: Kubernetes 1.18 or later is required for Device Plugin and Scheduler Framework.

3. kubelet read-only port: The open-source components get information about the Pods on the node through the kubelet read-only port (10255). For more information, see Section 3.2.

4. Container runtime configuration: GPU and PPU device injection depends on the lj-runtime runtime handler and the corresponding RuntimeClass. For more information, see Section 3.2.

5. Optional dependency: If you use cross-node gang topology scheduling and your cluster does not provide gang scheduling capabilities, install JobSet. JobSet v0.8.0 or later is recommended. For more information, see Section 3.2.

1.3 Limits

1. To use Runtime Kit capabilities in Kubernetes, you must integrate them in the standard way provided by the open-source component suite. That is, call the underlying capabilities through the unified gRPC interface.

2. Time-division multiplexing (TDM) and Multi-Instance GPU (MIG) spatial partitioning cannot be enabled on the same GPU at the same time.

3. MIG is a hardware-level NVIDIA technology and is supported on all card models that provide MIG capability.

2.1 Open-source repository

The open-source component code is available in the GitHub repository aliyun/LJ-RuntimeKit. The main branch always contains the latest usable code. Stable releases are published as tags. To obtain a stable release, use a tag. To obtain the latest code, use the main branch.

2.2 Precompiled images

If you do not need to customize the components, you can use the precompiled component images directly without building them yourself. The precompiled components are released and maintained by Alibaba Cloud Lingjun, which also provides technical support. If you encounter issues, you can submit feedback through channels such as issues in the open-source repository.

The image registry is lj-runtimekit-registry.cn-hangzhou.cr.aliyuncs.com/lj-runtimekit/. The following table describes the version mapping of the precompiled images for each component:

Note: In the table, Controller Job Agent is a node-side execution image that is deployed together with the Controller (see 3.1 Components). You do not need to deploy it separately.

3.1 Components

Controller Job Agent is the node-side execution image that the Controller uses to enable or disable features on a node. It is referenced by the --job-image parameter and is deployed together with the Controller. You do not need to deploy or manage it separately. For more information about how it works, see 3.5 Enable and disable features.

3.2 Pre-deployment tasks

Perform the following tasks in order.

Step 1: Verify that the Kubernetes cluster is available (on any machine that has kubectl permissions):

kubectl cluster-info
kubectl get nodes

Expected result: The cluster runs normally.

Step 2: Verify the GPU node environment (on each GPU node):

# Verify that the accelerator driver is installed (choose by device type)
nvidia-smi    # NVIDIA GPU
ppu-smi       # PPU
rocm-smi      # AMD

# Verify that the Lingjun Runtime service is running (preinstalled by default on Lingjun bare metal)
ls /run/amp/eKitC/xdm/sockets/
# A list of socket files should be returned (such as config-daemon.sock)

Expected result: The driver is installed correctly, and you can query the device information.

The Lingjun container suite comes with a container runtime that is compatible with heterogeneous GPU, PPU, and AMD devices. The runtime is built in and requires no separate installation. You need to confirm only that the driver works and that the socket file exists. For the registration configuration between the runtime and containerd, see Step 5.

Step 3: Enable the kubelet read-only port (10255) (on each GPU node as root):

# Check whether the port is enabled
curl -s http://localhost:10255/pods | head -c 100

If the port is not enabled, edit the kubelet configuration file (such as /var/lib/kubelet/config.yaml; the path varies with the cluster deployment method) and set readOnlyPort: 10255. Then run systemctl restart kubelet and verify the port again. Port 10255 is read-only and supports queries only. If your cluster security policy does not allow this port, contact the cluster administrator to evaluate whether the port can be enabled.

Step 4: Label the nodes that use Runtime Kit capabilities:

kubectl label node <node-name> lj_runtimekit_enable=true
kubectl get nodes -l lj_runtimekit_enable=true   # Verify

Only nodes labeled lj_runtimekit_enable=true run Device Plugin and Exporter. Runtime Kit components are not deployed on unlabeled nodes, and the related capabilities are not enabled. This label is also the feature switch. For more information, see 3.5 Enable and disable features.

Step 5: Configure the container runtime (required for GPU and PPU device injection):

Runtime Kit automatically configures the lj-runtime runtime handler in the containerd config.toml file. You do not need to modify config.toml manually. The following configuration is added after the existing runtimes.nvidia section:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.lj-runtime]
  runtime_type = "io.containerd.runc.v2"
  container_annotations = ["xdm.amp.com*"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.lj-runtime.options]
  BinaryName = "/usr/local/harp/kitC/xpu-container-runtime"
  SystemdCgroup = true

Create the corresponding RuntimeClass resource in the cluster:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: lj-runtime
handler: lj-runtime

kubectl apply -f lj-runtime-class.yaml

Verification:

# Confirm on the node that containerd has loaded the handler
containerd config dump | grep -A 5 "lj-runtime"

# Confirm that the RuntimeClass is created
kubectl get runtimeclass lj-runtime

Subsequent workload Pods must specify runtimeClassName: lj-runtime. Otherwise, GPU and PPU devices cannot be correctly injected into the container.

Step 6 (optional): Deploy JobSet. If your cluster does not provide gang scheduling capabilities, and you need topology-aware scheduling for a large number of Pods that start and stop at the same time, you must deploy JobSet. For more information, see JobSet installation. JobSet v0.8.0 or later is recommended.

3.3 One-click deployment (recommended)

If you have no customization requirements, we recommend that you use the precompiled images directly and deploy them with the following script.

Replace <version> with the target image tag in the version mapping table:

export lj_runtimekit_version=<version>

# Device Plugin
crictl pull lj-runtimekit-registry.cn-hangzhou.cr.aliyuncs.com/lj-runtimekit/lj-runtimekit-device-plugin:${lj_runtimekit_version}
# Scheduler Framework
crictl pull lj-runtimekit-registry.cn-hangzhou.cr.aliyuncs.com/lj-runtimekit/lj-runtimekit-scheduler:${lj_runtimekit_version}
# Exporter
crictl pull lj-runtimekit-registry.cn-hangzhou.cr.aliyuncs.com/lj-runtimekit/lj-runtimekit-exporter:${lj_runtimekit_version}
# Controller
crictl pull lj-runtimekit-registry.cn-hangzhou.cr.aliyuncs.com/lj-runtimekit/lj-runtimekit-controller:${lj_runtimekit_version}
# Controller Job Agent
crictl pull lj-runtimekit-registry.cn-hangzhou.cr.aliyuncs.com/lj-runtimekit/lj-runtimekit-controller-job-agent:${lj_runtimekit_version}

./deploy/deploy-all.sh deploy      # Deploy all components in one step
#./deploy/deploy-all.sh undeploy    # Uninstall all components in one step

# Deploy or uninstall a single component
#./deploy/deploy-all.sh deploy --components=device-plugin
#./deploy/deploy-all.sh deploy --components=scheduler
#./deploy/deploy-all.sh deploy --components=exporter
#./deploy/deploy-all.sh deploy --components=controller

For simple customization, you can adjust the preceding script and the scripts that it calls, such as deploy-all.sh. For more information, see the comments in the scripts.

The deployment script can be run repeatedly. If a deployment fails, fix the environment as prompted and run the script again. The script outputs a complete operation log throughout execution to help you verify the deployment and troubleshoot issues.

3.4 Verify the deployment

After the deployment is complete, perform the following steps to verify each item.

Step 1: Check the running status of the components

Run the following command:

kubectl get pods -n kube-system | grep -E 'lj-runtimekit-device-plugin|lj-runtimekit-scheduler|lj-runtimekit-exporter|lj-runtimekit-controller'

Expected result:

Criteria:

• All Pods are in the Running state and the READY column shows 1/1. This indicates a successful deployment.

• If a Pod is in the CrashLoopBackOff state, view the logs to identify the cause: kubectl logs -n kube-system <pod-name>.

• If a Pod is in the Pending or Init state, check whether the node labels and resources meet the scheduling conditions: kubectl describe pod -n kube-system <pod-name>.

Step 2: Check the resources reported by the node

kubectl describe node <node-name> | grep -E 'aliyun.com|alibabacloud.com'

Expected result (whole-GPU scenario): Accelerator-related extended resources appear in the Capacity and Allocatable sections of the node. For example:

alibabacloud.com/ppu:     8
aliyun.com/gpu:           8
aliyun.com/gpu-core:      800
aliyun.com/gpu-memory:    1152

Criteria:

• The resource quantities in Capacity should match the actual number of devices on the node (for example, an 8-GPU node should report 8).

• For the extended resources reported by Device Plugin, Allocatable is normally equal to Capacity. To view the resource usage of Pods, run kubectl describe node and check the Allocated resources section. If Allocatable is 0 but Capacity is not 0, this is usually residue from a resource name change.

• If no extended resources appear, Device Plugin failed to report them. Check the Device Plugin Pod logs and the status of the Runtime service on the node, including whether the socket file exists.

Step 3: Verify that the Exporter exposes metrics (run on a node where the Exporter is deployed)

# The default port is 9501. If you have changed the port in the Exporter deployment configuration, replace it with the actual value.
curl -s localhost:9501/metrics | head -20

Expected result: Metrics in Prometheus format are returned (lines that start with # HELP and # TYPE).

Criteria:

• If the metrics text is returned, the Exporter is working properly.

• If the request fails with Connection refused, check whether the Exporter Pod is running properly and confirm that the ports match.

3.5 Enable and disable features

The Controller automatically enables or disables Runtime Kit features on a node based on the node label. When you add the lj_runtimekit_enable=true label to a node, the Controller automatically enables the features on that node. When you remove the label from the node, the features are automatically disabled. The mechanism works as follows: The Controller automatically creates a Kubernetes Job that runs the Controller Job Agent image on the target node to complete these operations. After the operations are complete, the Job is cleaned up automatically. Label new nodes when you add them. The following table describes the impact of enabling and disabling features:

We recommend that you perform component upgrades and feature switch operations during off-peak hours.

3.6 Advanced deployment and customization

Use this section if you need to deploy components individually, build images yourself, or customize the components based on the open-source code.

Before you deploy the components, build the component images and push them to an image registry that the cluster can access. You can build the image for each component from its source directory by running make docker. After you build the images, push them to an image registry accessible to the cluster nodes, and make sure that the image names and tags in the deployment manifests match the actual build artifacts. The deployment files of the components are as follows:

For example, the deployment file for Device Plugin is deploy/deviceplugin/03-daemonset.yaml, and the image configuration in it is as follows:

containers:
- name: lj-runtimekit-device-plugin
  image: "<your-registry>/lj-runtimekit-device-plugin:<tag>"

If you need to modify only configuration items such as extended resource names, node label keys, and annotation keys without changing the code logic, edit the configuration (ConfigMap) in the deployment manifest of each component and redeploy it. You do not need to rebuild the image. For more information, see 3. Deploy customized components.

4.1 Version number format

Runtime Kit uses a unified version numbering scheme in the format v{major}.{feature}.{iteration} (for example, v1.13.0).

Upgrade criteria: whether existing behavior changes after an upgrade when you do not modify any configuration or YAML file. An iteration version upgrade does not change user-visible behavior. A feature version upgrade may introduce new features or behavior changes. A major version upgrade may contain incompatible changes, and you must assess them before you upgrade.

4.2 Runtime versions and open-source component versions

Runtime version: Starting from version 1.13.0, the Runtime supports integration with the open-source components.

Open-source component version: The open-source component versions follow the Runtime versions.

5.1 Compatibility policy

The gRPC interface between the Runtime and the open-source components is additive only and backward compatible: the interface evolves incrementally (new interfaces and fields are added), and the semantics of existing interfaces and fields do not change.

If special circumstances later require an incompatible change, the change is always released as a major version upgrade and is prominently marked in the release notes. Conversely, a major version upgrade does not necessarily contain incompatible changes. Check the release notes for that version.

5.2 Compatibility conclusions

Based on this policy:

1. Within the same major version, starting from version 1.13.0, any version of the open-source components works with any version of the Runtime.

2. A Runtime upgrade does not break the deployed components. You do not need to perform any upgrade for compatibility reasons.

3. Implementations that you customize based on the proto of an earlier version continue to work after a Runtime upgrade.

4. When a major version upgrade occurs, review the release notes to check for incompatible changes and assess them before you upgrade.

5.3 Version dependencies between components

There is no strict version binding between the open-source components. The components connect downward to the Runtime through the unified gRPC interface and upward to the cluster through standard Kubernetes extension mechanisms (the device plugin framework, Scheduler Framework, and the Prometheus metrics protocol). Components of different versions can work together in principle.

Because a new feature usually requires the Runtime and multiple components to work together, each new feature has a unified minimum version requirement for the Runtime and the entire component suite. For more information, see Section 6.

7.1 Upgrade scenarios

Version compatibility does not require you to upgrade. Upgrades occur only in the following two scenarios:

1. Use a new feature: Upgrade the Runtime and the open-source components to or above the corresponding minimum versions as described in Section 6.

2. Fix a known issue: After the issue is fixed in a new version, upgrade to that version.

7.2 Runtime upgrades

Runtime upgrades are transparent to users: you do not need to know how the upgrade is performed. If required, contact PDSA or SRE at Alibaba Cloud Lingjun to initiate the upgrade. The upgrade follows an internal Alibaba Cloud Lingjun process.

Impact of the upgrade window: During a Runtime upgrade, the gRPC service on the node becomes temporarily unavailable. Operations that depend on this interface, such as new GPU resource allocation and monitoring data collection, cannot be performed during this period. Running jobs that have already been allocated resources are not affected. We recommend that you schedule upgrades during off-peak hours.

7.3 Open-source component upgrades

Each Kubernetes component (Device Plugin, Scheduler Framework, Exporter, and Controller) is deployed as a DaemonSet or Deployment and supports native Kubernetes rolling updates. A node-by-node upgrade does not affect existing Pods.

Upgrade steps (precompiled components):

• Confirm the image tag that corresponds to the target version.

• In the one-click deployment directory, change the image field in the component YAML file to the target version (for example, deploy/deviceplugin/02-daemonset.yaml or deploy/scheduler/03-deployment.yaml).

• When you upgrade the Controller, also update the Controller Job Agent image referenced by its --job-image parameter to the target version.

• Run the one-click deployment script to deploy a single component or all components.

• Verify that the Pods of the new version are running properly.

# 1. Change the image version in the YAML file of the component (Device Plugin is used as an example)
# Edit deploy/deviceplugin/03-daemonset.yaml and change the image field to the target version
# image: lj-runtimekit-registry.cn-hangzhou.cr.aliyuncs.com/lj-runtimekit/lj-runtimekit-device-plugin:<new-version>

# 2. Deploy a single component
./deploy/deploy-all.sh deploy --components=device-plugin

# Or deploy all components
./deploy/deploy-all.sh deploy

# 3. Verify the rolling update result
kubectl rollout status daemonset/lj-runtimekit-device-plugin -n kube-system
kubectl get pods -n kube-system -l app=lj-runtimekit-device-plugin -o wide

Rollback:

If the new version does not work as expected, change the image tag in the component YAML file back to the original version, and then re-run the one-click deployment script to roll back.

7.4 Upgrade responsibilities

The underlying node-side component package (RPM) is pushed centrally by Alibaba Cloud Lingjun. You do not need to perform any manual operations.

Q1: Resource allocation fails and a Pod stays in Pending with an Insufficient resource error

Possible causes: No node in the cluster meets the requirements, the node label is missing, or Device Plugin did not report the resources. Solution: Check whether the target node is labeled lj_runtimekit_enable=true. View the Device Plugin logs to confirm that the resources are reported.

Q2: The node label lj_runtimekit_enable=true is set, but the feature is not enabled

Troubleshooting steps:

1. Confirm that the Controller Pod is running properly.

2. Check the Controller logs to see whether it detected the label change.

3. Check whether the corresponding Job is created.

4. View the Job Agent logs (kubectl logs -n kube-system job/lj-agent-<node-name>-<enable|disable>. The namespace is kube-system by default.)

5. Confirm that the config-daemon socket exists on the node: ls /run/amp/eKitC/xdm/sockets/config-daemon.sock.

Note: After a Job completes successfully, the Controller deletes it automatically. If the corresponding Job no longer exists, the operation completed successfully. You can confirm the result through the lj-runtimekit-controller/last-action-status annotation on the node or the ActionSucceeded event in the Controller logs. Failed Jobs are retained. You can view the logs by using the command in Step 4.

Q3: Component logs repeatedly show EOF or connection reset

Possible causes: The core services on the node restart, the socket file is deleted by mistake, or the client does not implement reconnection. Solution: Check the core service processes and the socket status. The client should implement reconnection with exponential backoff (we recommend an initial interval of 1 second, a multiplier of 2, and a maximum of 30 seconds).

Log locations