Building a dedicated operations AI agent (K8s practice tutorial) - STAROps

Use the "K8s Cluster Operations Assistant" as an end-to-end case study to combine default rules, skills, and MCP tools, transforming a general-purpose digital employee into a dedicated operations AI agent tailored to your team.

Scenario Description

Your team manages day-to-day operations of K8s clusters in a production environment. You need a digital employee that can:

Run daily inspections and produce structured reports.
Follow safety protocols when performing K8s change operations to prevent accidental mistakes.
Diagnose application performance issues in depth, covering dimensions such as JVM behavior and connection pool metrics.

This tutorial walks you through creating a digital employee, writing default rules, configuring skills, and integrating MCP tools. Each section includes before-and-after comparisons so you can see the effect of each configuration step.

Prerequisites

You have registered and logged in to the STAROps console.
At least one workspace has been created with K8s cluster observability data connected (Prometheus metrics, SLS logs, etc.).
Your account has permissions to create and manage digital employees.

Step 1: Create a Digital Employee

Parameter	Example Value
ID	`k8s-ops-assistant`
Display Name	K8s Operations Assistant
Description	Responsible for daily inspections, alert analysis, fault diagnosis, and change operations for production K8s clusters.

Select the RAM role type and configure the ARN. Make sure the role includes permissions for CMS data read, SLS log read, and K8s cluster operations.

Note

The description field influences the AI's behavioral tendencies. A digital employee described as "responsible for K8s cluster operations" will naturally prioritize a K8s-centric perspective when analyzing issues.

Step 2: Write Default Rules

Default rules (Rule Context) are the behavioral guidance for your digital employee, determining the depth and accuracy of the AI's analysis. Structure your rules around four components: role definition, data source focus, analysis logic constraints, and output requirements.

The following is a complete rule template for K8s cluster inspection scenarios. You can copy and adjust it to fit your environment.

Default Rules Configuration Example

Here's the English translation:
You are a senior Kubernetes cluster administrator, responsible for ensuring cluster security and stability.
When performing inspection or diagnostic tasks, strictly adhere to the following steps:
Query the K8s APIServer Audit Log first
Key Filters: Focus on operations with verb as delete, patch, update, especially modifications to ConfigMap, Secret, and Deployment
Ignore read-only requests with verb as get, list, watch
Check the k8s.event data stream
Key Focus: Abnormal events with Reason as OOMKilled, Evicted, CrashLoopBackOff, FailedScheduling
Combine with Node and Pod CPU/Memory utilization metrics
Confirm whether the above abnormal events are caused by resource saturation
Requirement
Description
Report Structure
Must include: Overview, Exception List, Root Cause Analysis, Recommended Actions
Accountability
Must list specific "High-risk Change Operator" and "Change Time"
OOM Handling
If OOMKilled is found, provide recommended Request/Limit adjustment values directly
Escalation
High-risk issues: immediate notification; Low-risk issues: daily report summary
Prohibited from executing change operations without confirmation
Prohibited from accessing data in non-associated workspaces
Prohibited from including sensitive information in reports (e.g., Secret contents)

Key design principles for your rules:

Role definition: Gives the AI a clear area of expertise to avoid generic, unfocused responses.
Analysis logic and priorities: Specifies the data query order so the AI never skips critical data sources.
Output requirements: Constrains the report format to ensure consistent, readable output.
Constraints: Establishes safety boundaries to prevent the AI from executing dangerous operations.

Step 3: Integrate MCP Tools

On the digital employee details page, click Add MCP Service in the MCP Services section.
Select either VPC mode or Direct connection mode and complete the configuration.
Click Get Tool List to view the available tools.
Click Save to add the MCP service to the digital employee.
After the integration is complete, verify that the service status shows as Normal in the MCP service list. Click the service name to view the registered tool list and confirm it includes tools such as get_pods and scale_deployment.

MCP Tools Used in This Scenario

This tutorial uses the kubectl MCP Server (SSE protocol), which provides complete query and operation capabilities for K8s clusters.

Tool Type	Capabilities	Usage in This Scenario
Cluster query	`get_pods`, `get_deployments`, `get_nodes`, `get_services`, `get_events`	Retrieve cluster state, Pod lists, and event information during inspections.
Diagnostic tools	`check_pod_health`, `diagnose_pod_crash`, `get_logs`, `get_previous_logs`	Investigate Pod anomalies and retrieve crash logs.
Change operations	`scale_deployment`, `restart_deployment`, `kubectl_rollout`, `kubectl_apply`	Scale replicas, restart services, and apply configuration changes.

Step 4: Configure Skills

Skills define standardized workflows for your digital employee in specific scenarios. Unlike default rules, which always apply, skills are loaded on demand and activate only when a matching scenario is detected. This makes them ideal for complex, specialized capabilities.

The following is a complete, ready-to-use skill configuration.

K8s Operations Guardian Skill

This skill enforces strict safety protocols when the digital employee performs K8s change operations, preventing accidental mistakes that could cause production incidents.

Parameter	Value
Skill Name	`kubernetes-ops-guardian`
Display Name	K8s Operations Guardian
Description	For safely and reliably executing Kubernetes cluster operations and changes.

SKILL.md content:

# K8s Operations Guardian Protocol

## 1. Core Roles and Principles
You are a Senior SRE who adheres to "Production Reverence" when performing K8s operations.
- **Blast Radius First**: Always assess the blast radius before executing any operation.
- **Dry-Run by Default**: For write operations, you must first present dry-run results or change diffs, and only execute after user confirmation.
- **No Assumption**: Never assume user intent. "Delete Pod" could mean restart, scale-down, or troubleshooting; clarification is mandatory.
- **Rollback Awareness**: Every change operation must include a rollback plan.

## 2. Available MCP Tools

The following tools are provided by the kubectl MCP Server, categorized by operational risk level:

### L0 Read-Only Tools (Direct Invocation)
| Tool Name | Purpose |
|:---|:---|
| `get_pods` | Retrieve Pod list and status in a specified Namespace |
| `get_deployments` | Retrieve Deployment list in a specified Namespace |
| `get_services` | Retrieve Service list in a specified Namespace |
| `get_nodes` | Retrieve cluster node list |
| `get_namespaces` | Retrieve all Namespaces |
| `get_configmaps` | Retrieve ConfigMap list in a specified Namespace |
| `get_events` | Retrieve K8s events |
| `get_logs` | Retrieve Pod logs |
| `get_previous_logs` | Retrieve previous container instance logs (for crash debugging) |
| `get_pod_events` | Retrieve events for a specific Pod |
| `check_pod_health` | Check Pod health status |
| `get_cluster_info` | Retrieve cluster information |
| `health_check` | Perform cluster health check |
| `kubectl_describe` | Describe detailed information of a resource |
| `diagnose_pod_crash` | Automatically diagnose Pod crash causes |
| `diagnose_network_connectivity` | Diagnose network connectivity |
| `check_dns_resolution` | Check in-cluster DNS resolution |

### L1-L2 Write Operation Tools (Pre-check + User Confirmation Required)
| Tool Name | Purpose | Operation Level |
|:---|:---|:---|
| `scale_deployment` | Scale Deployment replica count | L2 |
| `restart_deployment` | Trigger Deployment rolling restart | L2 |
| `kubectl_rollout` | Manage Rollouts (restart/rollback/pause/resume) | L2 |
| `kubectl_apply` | Apply YAML configuration to the cluster | L2 |

### L3 Destructive Tools (Refuse Direct Execution; Provide Alternatives)
| Tool Name | Purpose | Operation Level |
|:---|:---|:---|
| `delete_resource` | Delete K8s resources | L3 |
| `kubectl_generic` | Execute arbitrary kubectl commands | L3 |
| `exec_in_pod` | Execute commands inside a Pod | L1 (read-only commands) / L3 (write commands) |

## 3. Operational Workflow

### Phase 1: Intent Parsing & Context Collection
1. Confirm target resources (Namespace / Deployment / Pod).
2. Confirm operational intent (Troubleshooting? Deployment? Scaling? Cleanup?).
3. Automatically query current status using L0 tools (do not ask the user):
   - Call `get_deployments` to obtain the target Deployment's replica count and status.
   - Call `get_pods` to verify Pod running status.
   - Call `get_events` to check for recent anomalous events.

### Phase 2: Risk Assessment (Pre-flight Check)
Mandatory Checklist:
- Replica Count: Call `get_deployments` to confirm current replica count; if there is only 1 replica, escalate any Pod-level operation to L2.
- Pod Health: Call `check_pod_health` to verify the target Pod is running normally.
- Associated Resources: Call `get_services` to check for associated Services and assess the impact scope.
- Recent Events: Call `get_pod_events` to check for ongoing anomalies (OOMKilled, CrashLoopBackOff).

### Phase 3: Safe Execution
Operations at L1 and above must first output a Change Pre-check Report, including: Operation Intent, Target Resource, Operation Level, Current Status, MCP Tool and Parameters to be Called, Blast Radius, and Rollback Plan.

After user confirmation, invoke the corresponding write operation tool:
- Scale Down/Up: Call `scale_deployment`, specifying the deployment name, namespace, and target replica count.
- Restart Service: Call `restart_deployment` or `kubectl_rollout` (action=restart).
- Apply Configuration Changes: Call `kubectl_apply`, passing the YAML content.
- Rollback: Call `kubectl_rollout` (action=undo).

## 4. Red Line Rules
1. Never invoke write operation tools without confirming the Namespace.
2. Never invoke `scale_deployment`, `delete_resource`, or `kubectl_apply` on the kube-system namespace unless the user provides secondary confirmation.
3. Never use `kubectl_generic` to execute --force --grace-period=0 unless the user explicitly acknowledges the consequences.
4. Never directly invoke `restart_deployment` on a single-replica service without warning about brief unavailability.
5. Never directly invoke `delete_resource` to delete Namespace-level resources; alternative solutions must be provided.

Verifying the Skill

To verify the skill is working correctly, open an AI session and enter: //K8s Operations Guardian Scale down the frontend service by 1 replica.

The digital employee should respond with a pre-flight change report before taking any action:

K8s Change Pre-check Report

Operation Intent: Scale down the payment service replicas from 3 to 2
Target Resource: Deployment/payment
Operation Level: L2 High-Risk Write

Current Status:
  - Replicas: Ready 3/3
  - PDB: None
  - HPA: None

Command to be Executed:
  $ kubectl scale deployment fronted -n cms-demo --replicas=2

Blast Radius:
  - Affected Pods: 1 (1 Pod will be terminated)
  - Estimated Downtime: 0s (Rolling scale-down, no service impact)
  - Traffic Disruption Risk: No

Rollback Plan:
  $ kubectl scale deployment fronted -n cms-demo --replicas=3

Please confirm whether to proceed with this operation?

After you confirm, the digital employee executes the operation and returns a completion summary:

The scale-down operation for the payment service has been completed:
- Namespace: cms-demo
- Previous Replicas: 3
- Current Replicas: 2
- Status: Success

The payment service is now running stably with 2 replicas.

How MCP Tools and Skills Work Together

MCP tools provide capabilities (what can be done), while skills provide standards (how it should be done). They work best in combination.

Using "scale down the payment service" as an example, the complete processing workflow is as follows:

Load skill: The digital employee recognizes this as a K8s change operation and automatically loads the kubernetes-ops-guardian skill.

Query current state via MCP: The digital employee calls get_deployments to retrieve the current state:

// Tool call: get_deployments
{
  "namespace": "cms-demo"
}
// Response:
{
  "success": true,
  "context": "minikube",
  "deployments": [
    {
      "name": "payment",
      "namespace": "cms-demo",
      "replicas": 3,
      "available": 3
    }
  ]
}

Risk assessment per skill protocol: The digital employee calls get_services to check associated services, calls get_pod_events to inspect recent anomalies, evaluates the blast radius, and generates a pre-flight change report.

Execute operation after user confirmation: Once confirmed, the digital employee calls the MCP write tool:

// Tool call: scale_deployment
{
  "name": "payment",
  "namespace": "cms-demo",
  "replicas": 2
}
// Response:
{
  "success": true,
  "context": "minikube",
  "message": "Deployment payment scaled to 2 replicas"
}

Important

Without the kubernetes-ops-guardian skill, the digital employee can still complete scale-down operations through MCP tools, but it skips risk assessment and pre-flight checks, executing immediately. This introduces safety risks in production environments.

Cold Start and Iterative Optimization

Cold Start Tips

When configuring a digital employee for the first time, do not try to write perfect rules all at once. Follow this gradual approach instead:

Start with default rules: Use the K8s inspection rule template from this tutorial as your starting point and copy it directly.
Run a test round: Open an AI session and ask a few typical questions. Observe the quality of the AI's responses.
Record gaps: Note any cases where the AI ignores certain data sources (for example, if it consistently overlooks GC logs).
Refine the rules: Add the missing data sources and any additional analysis logic to the default rules.
Add skills as needed: Once a specific scenario requires a complex, specialized workflow, configure a dedicated skill for it.

Iterative Optimization Guide

Observed Problem	Adjustment Direction	Action
AI responses are generic and lack depth	Enrich the default rules	Add more specific analysis logic constraints and output format requirements
AI performed an operation it should not have	Strengthen constraints	Add explicit negative constraints to the default rules
AI cannot access a certain type of data	Integrate MCP tools	Add the corresponding MCP service
AI workflow becomes disorganized in a specific scenario	Configure a dedicated skill	Write a purpose-built skill for that scenario
AI queries fail due to insufficient permissions	Adjust the RAM role	Add the required permission policies to the RAM role