Container Service for Kubernetes:Best practices for configuring AI model inference services in Knative

Accelerated model deployment

AI model files are large. Packaging them inside container images creates two problems: images become too large to pull quickly, and model versions get locked to image versions. Instead, store model files in Object Storage Service (OSS) or File Storage NAS (NAS), then mount the storage into your Knative Service pods using a persistent volume claim (PVC).

To further accelerate model loading, use Fluid—a distributed dataset orchestration and acceleration engine that can load large language models within seconds by caching data on local SSD.

apiVersion: v1
kind: Secret
metadata:
  name: access-key
stringData:
  fs.oss.accessKeyId: your_ak_id          # Replace with your AccessKey ID.
  fs.oss.accessKeySecret: your_ak_skrt    # Replace with your AccessKey secret.
---
apiVersion: data.fluid.io/v1alpha1
kind: Dataset
metadata:
  name: oss-data
spec:
  mounts:
  - mountPoint: "oss://{Bucket}/{path-to-model}"  # Replace with your OSS bucket and model path.
    name: xxx
    path: "{path-to-model}"                        # Replace with the model path to load.
    options:
      fs.oss.endpoint: "oss-cn-beijing.aliyuncs.com"  # Replace with your OSS bucket endpoint.
    encryptOptions:
      - name: fs.oss.accessKeyId
        valueFrom:
          secretKeyRef:
            name: access-key
            key: fs.oss.accessKeyId
      - name: fs.oss.accessKeySecret
        valueFrom:
          secretKeyRef:
            name: access-key
            key: fs.oss.accessKeySecret
  accessModes:
    - ReadOnlyMany
---
apiVersion: data.fluid.io/v1alpha1
kind: JindoRuntime
metadata:
  name: oss-data
spec:
  replicas: 2
  tieredstore:
    levels:
      - mediumtype: SSD
        volumeType: emptyDir
        path: /mnt/ssd0/cache
        quota: 100Gi
        high: "0.95"
        low: "0.7"
  fuse:
    properties:
      fs.jindofsx.data.cache.enable: "true"
    args:
      - -okernel_cache
      - -oro
      - -oattr_timeout=7200
      - -oentry_timeout=7200
      - -ometrics_port=9089
    cleanPolicy: OnDemand

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: sd-backend
spec:
  template:
    spec:
      containers:
        - image: <YOUR-IMAGE>    # Replace with your container image name.
          name: image-name
          ports:
            - containerPort: xxx
              protocol: TCP
          volumeMounts:
            - mountPath: /data/models  # The path where the model is loaded inside the container.
              name: data-volume
      volumes:
        - name: data-volume
          persistentVolumeClaim:
            claimName: oss-data    # Must match the Dataset name.

apiVersion: eci.alibabacloud.com/v1
kind: ImageCache
metadata:
  name: imagecache-ai-model
  annotations:
    k8s.aliyun.com/eci-image-cache: "true"  # Enable cache reuse.
spec:
  images:
  - <YOUR-IMAGE>
  imageCacheSize:
    25    # Cache size in GiB.
  retentionDays:
    7     # Number of days to retain the cache.

Graceful pod shutdown

If an application uses HTTP probing, after the application pods receive the SIGTERM signal, the pods enter the Not Ready state so no new requests are routed to them. However, Knative's queue-proxy may still forward in-flight requests for a short period after SIGTERM. Set timeoutSeconds to 1.2× your maximum request timeout to give those requests time to complete.

For example, if your longest request takes 10 seconds, set timeoutSeconds to 12:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: helloworld-go
  namespace: default
spec:
  template:
    spec:
      timeoutSeconds: 12

Configure concurrency

Concurrency settings determine how many requests a single pod handles at a time. Tuning them correctly improves throughput and prevents overload.

Hard concurrency limit (containerConcurrency): An enforced cap. Requests above the limit queue and wait for capacity.

Soft concurrency limit (autoscaling.knative.dev/target): A non-strict target. Traffic spikes may temporarily exceed it. This is the safer default for most inference workloads.

Target utilization (autoscaling.knative.dev/target-utilization-percentage): Controls how close to the limit Knative tries to stay. Set it close to 100% to maximize pod utilization before triggering scale-up.

For more information, see Configure the soft concurrency limit.

Auto scaling

Before tuning auto scaling, measure your pod startup time—it determines which parameters matter most.

For full parameter reference, see Enable auto scaling to withstand traffic fluctuations.

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: helloworld
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/window: "40s"

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: helloworld
  namespace: default
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/panic-window-percentage: "20.0"

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: helloworld
  namespace: default
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/panic-threshold-percentage: "150.0"

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-autoscaler
  namespace: knative-serving
data:
  max-scale-up-rate: "500.0"

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-autoscaler
  namespace: knative-serving
data:
  max-scale-down-rate: "4.0"

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: helloworld-go
  namespace: default
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/scale-down-delay: "2m"  # Keep pods for 2 minutes after scale-down decision.

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: helloworld
  namespace: default
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/min-scale: "0"

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: helloworld
  namespace: default
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/max-scale: "3"

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: helloworld-go
  namespace: default
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/initial-scale: "3"

GPU sharing

Container Service for Kubernetes (ACK) supports GPU sharing through cGPU, which provides GPU memory isolation between containers. To enable GPU sharing in a Knative Service, set aliyun.com/gpu-mem under resources.limits.

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: helloworld-go
  namespace: default
spec:
  template:
    spec:
      containerConcurrency: 1
      containers:
      - image: registry-vpc.cn-hangzhou.aliyuncs.com/demo-test/test:helloworld-go
        name: user-container
        ports:
        - containerPort: 6666
          name: http1
          protocol: TCP
        resources:
          limits:
            aliyun.com/gpu-mem: "3"  # GPU memory in GiB allocated to this container.

Probes

Knative probes services more frequently than standard Kubernetes to minimize cold start latency. Configure both liveness and readiness probes for your inference containers.

• Liveness probe: Monitors container health. If the probe fails, Kubernetes restarts the container.

• Readiness probe: Controls traffic routing. Only pods that pass the readiness probe receive requests.

TCP vs HTTP:

• TCP probe: does not open the listening port until all components are loaded and the containers are ready.

• HTTP probe: passes only when the endpoint returns a successful response. This is more accurate for inference services that have a warm-up phase.

Set periodSeconds to a value shorter than the default 10 seconds to reduce the delay between a pod becoming ready and traffic being routed to it.

For more information, see Configure port probing in Knative.

Scale to zero

When a Knative Service receives no traffic, Knative can scale its pods to zero, eliminating resource costs for idle models.

For workloads with many models, create a separate Knative Service for each model. This keeps the business logic in each Service simple and allows each model to scale independently.

When a client first accesses an idle model, Knative scales up from zero. To reduce the cold start impact, configure a reserved instance so at least one warm pod is always available.

For more information, see Configure a reserved instance.

Canary releases

Knative keeps the old revision running until the new one is ready, ensuring continuity during deployments.

Control the traffic split using the percent field. The valid range is 10%–100%. Set it to 100% to fully cut over to the new revision and retire the old one immediately.

When resources are constrained—for example, 500 pods running with only 300 more available—use a low traffic percentage to limit how fast the old revision scales down. Setting too high a percentage in this situation can cause the old revision to scale down faster than the new one scales up, leaving insufficient capacity.

For more information, see Perform a canary release based on traffic splitting for a Knative Service.

Colocate ECS instances and elastic container instances

Configure ResourcePolicies to mix Elastic Compute Service (ECS) instances with elastic container instances. ECS instances run during normal traffic; elastic container instances absorb traffic spikes.

• Scale up: Knative schedules pods to ECS instances first. When pods exceed the predefined limit or ECS capacity is exhausted, new pods go to elastic container instances.

• Scale down: Pods on elastic container instances are removed first.

For more information, see Colocate ECS instances and elastic container instances in Knative.