Container Service for Kubernetes:Build a Kubernetes Ops agent with Kagent, ACK Gateway, and ACK MCP Server

Kagent overview

It integrates AI agents with toolchains, allowing them to perform complex, multi-step tasks through natural language and translate AI insights into concrete actions.

Prerequisites

1. Create a namespace named kagent in your ACK cluster.

2. Install the kagent-crds and kagent applications in the kagent namespace from the ACK Marketplace or by navigating to Applications > Helm on the ACK console.

3. In the ACK console, on the Add-ons page, install the Gateway API component and enable Experimental Channel.

4. On the ACK console, go to the Add-ons page to install the Gateway with Inference Extension component.

5. Activate Alibaba Cloud Model Studio and get an API key.

Step 1: Deploy ACK MCP Server

1. Create a custom permission policy.

   The following code block shows the required read-only permissions for ACK MCP Server.

   {
     "Version": "1",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "cs:Check*",
           "cs:Describe*",
           "cs:Get*",
           "cs:List*",
           "cs:Query*",
           "cs:RunClusterCheck",
           "cs:RunClusterInspect"
         ],
         "Resource": "*"
       },
       {
         "Effect": "Allow",
         "Action": "arms:GetPrometheusInstance",
         "Resource": "*"
       },
       {
         "Effect": "Allow",
         "Action": [
           "log:Describe*",
           "log:Get*",
           "log:List*"
         ],
         "Resource": "*"
       }
     ]
   }

2. Install ACK MCP Server. For more information, see Deploy and run ack-mcp-server.

   After installation, run thekubectl get --raw "/api/v1/namespaces/kube-system/services/ack-mcp-server/proxy/sse" --v=10 command to verify the installation.

3. Declare ACK MCP Server in the cluster.

   kubectl apply -f - <<EOF
   apiVersion: kagent.dev/v1alpha2
   kind: RemoteMCPServer
   metadata:
     name: ack-mcp-tool-server
     namespace: kagent
   spec:
     description: Official ACK tool server
     protocol: SSE
     sseReadTimeout: 5m0s
     terminateOnClose: true
     timeout: 30s
     # ACK MCP Server is installed in the kube-system namespace by default. If you switch to a different namespace, change the URL here accordingly.
     url: http://ack-mcp-server.kube-system:8000/sse
   EOF

4. Check the status of the RemoteMCPServer resource to get the ACK MCP tools.

   kubectl describe RemoteMCPServer ack-mcp-tool-server -n kagent

   Expected output:

   ...
   status:
     conditions:
     - lastTransitionTime: "2025-XX-XXT11:35:29Z"
       message: ""
       observedGeneration: 2
       reason: Reconciled
       status: "True"
       type: Accepted
     discoveredTools:
     - description: Gets a list of all ACK clusters in all regions. By default, a maximum of 10 clusters are returned.
       name: list_clusters
     - description: Execute kubectl command with intelligent context management. Supports
         cluster_id for automatic context switching and creation.
       name: ack_kubectl
     - description: Queries the Alibaba Cloud Prometheus data of an ACK cluster.
       name: query_prometheus
     - description: Gets Prometheus metric definitions and best practices.
       name: query_prometheus_metric_guidance
     - description: "Diagnoses Kubernetes resources in an ACK cluster. Use this tool for in-depth diagnosis when you encounter problems that are difficult to locate. The following resources are supported for diagnosis: \n1. **node**: Kubernetes
         node\n2. **ingress**: Ingress\n3. **memory**: Node memory\n4. **pod**: Pod\n5. **service**: Service\n6.
         **network**: Network connectivity\n                        "
       name: diagnose_resource
     - description: Generates and queries the latest health inspection report for an ACK cluster.
       name: query_inspect_report
     - description: |-
         Query Kubernetes (k8s) audit logs.
             Function Description:
             - Supports multiple time formats (ISO 8601 and relative time).
             - Supports suffix wildcards for namespace, resource name, and user.
             - Supports multiple values for verbs and resource types.
             - Supports both full names and short names for resource types.
             - Allows specifying the cluster name to query audit logs from multiple clusters.
             - Provides detailed parameter validation and error messages.
             Usage Suggestions:
             - You can use the list_clusters() tool to view available clusters and their IDs.
             - By default, it queries the audit logs for the last 24 hours. The number of returned records is limited to 10 by default.
       name: query_audit_log
     - description: Gets the current time in ISO 8601 format and Unix timestamp format.
       name: get_current_time
     - description: Queries the logs of control plane components in an ACK cluster. This tool first queries the control plane log configuration to verify that the component is enabled, and then queries the corresponding SLS logs.
       name: query_controlplane_logs
     observedGeneration: 2

Step 2: Deploy the gateway and configure Model Studio

1. Create a gateway.

   kubectl apply -f- <<EOF
   apiVersion: gateway.networking.k8s.io/v1
   kind: Gateway
   metadata:
     name: model-gateway
     namespace: kagent
   spec:
     gatewayClassName: ack-gateway
     infrastructure:
       parametersRef:
         group: gateway.envoyproxy.io
         kind: EnvoyProxy
         name: custom-proxy-config
     listeners:
     - name: http-bailian
       protocol: HTTP
       port: 8080
   ---
   apiVersion: gateway.envoyproxy.io/v1alpha1
   kind: EnvoyProxy
   metadata:
     name: custom-proxy-config
     namespace: kagent
   spec:
     provider:
       type: Kubernetes
       kubernetes:
         envoyService:
           type: ClusterIP
   EOF

2. Create a backend for Model Studio.

   kubectl apply -f- <<EOF
   apiVersion: gateway.envoyproxy.io/v1alpha1
   kind: Backend
   metadata:
     name: bailian
     namespace: kagent
   spec:
     endpoints:
       - fqdn:
           hostname: dashscope-intl.aliyuncs.com
           port: 443
   ---
   apiVersion: gateway.networking.k8s.io/v1alpha3
   kind: BackendTLSPolicy
   metadata:
     name: bailian-tls
     namespace: kagent
   spec:
     targetRefs:
     - group: gateway.envoyproxy.io
       kind: Backend
       name: bailian
     validation:
       hostname: dashscope-intl.aliyuncs.com
       wellKnownCACertificates: System
   EOF

3. Create a routing rule to route specified requests to the Model Studio backend.

   kubectl apply -f- <<EOF
   apiVersion: gateway.networking.k8s.io/v1
   kind: HTTPRoute
   metadata:
     name: bailian-route
     namespace: kagent
   spec:
     parentRefs:
       - name: model-gateway
     rules:
       - backendRefs:
           - group: gateway.envoyproxy.io
             kind: Backend
             name: bailian
         filters:
           - type: URLRewrite
             urlRewrite:
               hostname: dashscope-intl.aliyuncs.com
               path:
                 type: ReplacePrefixMatch
                 replacePrefixMatch: /compatible-mode/v1
         matches:
           - path:
               type: PathPrefix
               value: /v1
         timeouts:
           backendRequest: 10m
           request: 10m
   EOF

Step 3: Manage the API key with ACK Gateway

When you access external Large Language Model (LLM) services, you usually need an API key for authorization. ACK Gateway supports dynamically injecting API keys into requests. This allows you to centrally manage the API keys for all model services, reducing maintenance complexity and enhancing cluster security.

1. Create a Secret to store the API key for the Model Studio service.

   export PROVIDER_API_KEY=${YOUR_MODEL_STUDIO_API_KEY}
   kubectl create secret generic bailian-credential -n kagent --from-literal credential="Bearer $PROVIDER_API_KEY"

2. Create an HTTPRouteFilter resource that references this Secret.

   kubectl apply -f- <<EOF
   apiVersion: gateway.envoyproxy.io/v1alpha1
   kind: HTTPRouteFilter
   metadata:
     name: credential-injection
     namespace: kagent
   spec:
     credentialInjection:
       overwrite: true
       credential:
         valueRef:
           name: bailian-credential
   EOF

3. Modify the HTTPRoute resource to enable automatic API key injection.

   kubectl apply -f- <<EOF
   apiVersion: gateway.networking.k8s.io/v1
   kind: HTTPRoute
   metadata:
     name: bailian-route
     namespace: kagent
   spec:
     parentRefs:
       - name: model-gateway
     rules:
       - backendRefs:
           - group: gateway.envoyproxy.io
             kind: Backend
             name: bailian
         filters:
           - type: URLRewrite
             urlRewrite:
               hostname: dashscope-intl.aliyuncs.com
               path:
                 type: ReplacePrefixMatch
                 replacePrefixMatch: /compatible-mode/v1
           # This is the main part that was added.
           - type: ExtensionRef
             extensionRef:
               group: gateway.envoyproxy.io
               kind: HTTPRouteFilter
               name: credential-injection
         timeouts:
           backendRequest: 10m
           request: 10m
         matches:
           - path:
               type: PathPrefix
               value: /v1
   EOF

Step 4: Configure ModelConfig for Model Studio

1. Get the gateway address.

   export GATEWAY_HOST=$(kubectl -n kagent get gateway/model-gateway -o jsonpath='{.status.addresses[0].value}')
   echo $GATEWAY_HOST

2. Create the following ModelConfig.

   kubectl apply -f - <<EOF
   apiVersion: kagent.dev/v1alpha2
   kind: ModelConfig
   metadata:
     name: my-provider-config
     namespace: kagent
   spec:
     model: qwen-plus
     openAI:
       baseUrl: http://$GATEWAY_HOST:8080/v1
     provider: OpenAI
   EOF

Step 5: Create the agent

1. Define the agent using the following YAML content.

   kubectl apply -f - <<EOF
   apiVersion: kagent.dev/v1alpha2
   kind: Agent
   metadata:
     name: my-ack-ops-agent
     namespace: kagent
   spec:
     declarative:
       deployment:
         env:
           - name: OPENAI_API_KEY
             value: placeholder
         replicas: 1
       modelConfig: my-provider-config
       stream: true
       systemMessage: |-
         # Role
         You are a professional ACK (Alibaba Cloud Container Service for Kubernetes) intelligent assistant. Your mission is to accurately understand user requests about clusters and select the most appropriate tools to perform queries, diagnostics, or analyses.
         # Core Instructions
         1.  **Confirm the Target, First Principle**:
             *   Before performing any operation, you must first confirm the user's target `cluster_id`.
             *   If the user's query does not provide it, you **must** first call the `list_clusters` tool and ask the user which cluster they want to operate on.
         2.  **Tool Selection Strategy (by priority)**:
             *   **Complex Fault Diagnosis**: When encountering complex issues such as pod abnormalities, network failures, or NotReady nodes, **prioritize using `diagnose_resource`**.
             *   **Performance Metric Queries**: When the issue involves "high/low CPU/memory", "fast/slow", or "how much usage", **prioritize using `query_prometheus`**.
             *   **Security and Change Audits**: When the issue is about "who did what and when", **prioritize using `query_audit_log`**.
             *   **Overall Cluster Health**: When the user wants to know "if the cluster is healthy" or requests an "inspection report", **use `query_inspect_report`**.
             *   **Control Plane Issues**: When you suspect a problem with Kubernetes system components such as the API Server or Scheduler, **use `query_controlplane_logs`**.
             *   **General Queries**: For all other standard, explicit Kubernetes resource queries such as `get pods`, `describe service`, and `logs `, **use `ack_kubectl` as the default tool**.
         3.  **Security Red Lines**:
             *   Your primary responsibility is to query and diagnose. For any operation performed through `ack_kubectl` that **may modify the cluster state**, such as `apply`, `delete`, or creating a temporary pod for diagnosis, you **must** first explain the command you will execute and its purpose, and only proceed after receiving **explicit authorization from the user**.
         4.  **Code of Conduct**:
             *   If a user's question is unclear, ask for clarification before acting.
             *   Respond in a friendly and enthusiastic manner.
             *   If you still cannot find the answer after using the tools, **never invent one**. Honestly reply: "Sorry, I cannot locate the problem with the available tools," and you can provide your findings.
         # Response Format
         *   **Always use Markdown format**.
         *   Your response must include a **summary of your actions** and an **analysis and recommendations** based on the results.
         ---
         ### Summary
         *(Summarize what you did and your key findings in one sentence.)*
       tools:
         - mcpServer:
             apiGroup: kagent.dev
             kind: RemoteMCPServer
             name: ack-mcp-tool-server
             toolNames:
               - list_clusters
               - ack_kubectl
               - query_prometheus
               - query_prometheus_metric_guidance
               - diagnose_resource
               - query_inspect_report
               - query_audit_log
               - get_current_time
               - query_controlplane_logs
           type: McpServer
     description: This agent can interact with ACK MCP Tools to get cluster information and operate on the cluster.
     type: Declarative
   EOF

2. Verify the agent's status.

   kubectl get pod -n kagent

   Expected output:

   NAME                                              READY   STATUS    RESTARTS   AGE
   my-ack-ops-agent-66b74675fc-rqwwx                 1/1     Running   0          2m6s
   ...

Step 6: Use the agent through the UI

Kagent provides a default web UI that you can use to directly interact with the agent.

1. Use port forwarding to expose the kagent-ui service to your local machine.

   kubectl port-forward -n kagent service/kagent-ui 8082:8080

2. In a browser, navigate to localhost:8082 to access the agent.

   1. Sample interaction: Use Prometheus to view metrics for pods in the kagent namespace of the cluster.

      When the Kagent agent receives a query, it sequentially calls the list_clusters, get_current_time, and query_prometheus_metric_guidance (twice) tools. All calls show a green Completed status. The Agent Details panel on the right shows that the current agent is kagent/my-ack-ops-agent (qwen-plus). The available tools include list_clusters, ack_kubectl, query_prometheus, query_prometheus_metric_guidance, diagnose_resource, query_inspect_report, query_audit_log, get_current_time, and query_controlplane_logs. This process shows that the agent can automatically orchestrate multiple tools to complete complex query tasks.

      The agent returns an analysis report. The report shows that the agent successfully queried the CPU, memory, and network metrics for all pods in the kagent namespace of the prod-env-2 cluster, but did not collect disk I/O data. The CPU usage for all pods is extremely low and poses no risk of throttling: my-ack-ops-agent-7fb6cd8c84-cwsh8 at about 0.003 core, kagent-ui-5996fbb6fd-dpld2 at about 0.007 core, ack-mcp-server-fcf79dcdb-7jfl5 at about 0.002 core, and kagent-controller-8685568659-gxzsj at about 0.001 core. For memory usage, my-ack-ops-agent uses the most, at about 722 MiB.