E-MapReduce:Manage Spark workspaces with EMR Agent Skills

Install the Alibaba Cloud CLI

This skill requires Alibaba Cloud CLI 3.3.1 or later.

# macOS
brew install aliyun-cli

# Or use the official script
curl -fsSL https://aliyuncli.alicdn.com/aliyun-cli-install.sh | bash

# Verify the version
aliyun version

Configure Alibaba Cloud credentials

# Configure interactively
aliyun configure
# Enter your AccessKey ID, AccessKey Secret, and default region

Or use environment variables:

export ALIBABACLOUD_ACCESS_KEY_ID="your-ak"
export ALIBABACLOUD_ACCESS_KEY_SECRET="your-sk"
export ALIBABACLOUD_REGION_ID="cn-hangzhou"

Ensure RAM permissions

Your RAM user or role must have EMR Serverless Spark permissions.

Method 1: Use the npx skills CLI

Install from the aliyun/alibabacloud-aiops-skills GitHub repository using npx skills:

# Install the Spark management skill
npx skills add aliyun/alibabacloud-aiops-skills --skill alibabacloud-emr-spark-manage

Method 2: Manual installation

Copy the skill directory to .claude/skills/ in your project root:

your-project/
└── .claude/
    └── skills/
        └── alibabacloud-emr-spark-manage/
            ├── SKILL.md
            └── references/

To make the skill available to all projects, install it in ~/.claude/skills/.

Use the skill

After installation, use natural language in Claude Code. The Agent automatically loads the skill. Examples:

• "Create a Spark workspace"

• "Submit a PySpark job to workspace ws-xxx"

• "Use Kyuubi to run the SQL query: SELECT count(*) FROM my_table"

• "Scale a resource queue"

• "View job logs"

• "List my Spark workspaces"

How it works: On startup, Claude Code scans .claude/skills/ and reads the name and description metadata from each SKILL.md. When a request matches a skill description, the Agent loads and executes it.

Advanced: Install alibabacloud-find-skills (Optional)

alibabacloud-find-skills is a Meta Skill that lets the Agent search for and install other Alibaba Cloud Skills:

npx skills add aliyun/alibabacloud-aiops-skills --skill alibabacloud-find-skills

Ask the Agent to find and install skills, such as "Find a Skill for managing ECS" or "What database-related Skills does Alibaba Cloud have?". Requires aliyun CLI >= 3.3.1 and the agentexplorer plugin.

Install OpenClaw

# macOS / Linux
curl -fsSL https://openclaw.ai/install.sh | bash

# Verify the installation (requires Node.js >= 22.0)
openclaw --version

Method 1: Use the npx skills CLI

Use npx skills to install the skill:

npx skills add aliyun/alibabacloud-aiops-skills --skill alibabacloud-emr-spark-manage

Choose project or global scope during installation. Verify with:

openclaw skills list

Method 2: Manual installation

Copy the skill directory to one of these locations (highest priority first):

Use the skill

Start OpenClaw and interact using natural language:

# Terminal mode
openclaw tui

# Or Web mode
openclaw dashboard

The Agent matches your requests to the appropriate skill:

• "Submit a Spark SQL job: SELECT count(*) FROM my_table"

• "Show the resource queues for Spark workspace ws-xxx"

• "Run a PySpark job"

Install the HITL plugin (recommended)

The HITL (Human-in-the-Loop) plugin intercepts high-risk cloud operations and requires manual confirmation before execution:

openclaw plugins install @alicloud/alibabacloud-hitl-claw-plugin

Once installed, operations that create or release cloud resources pause for your confirmation before execution.

Q1: Why does the skill fail after installation?

• Claude Code: Ensure that the skill directory is located in .claude/skills/ and contains a SKILL.md file.

• OpenClaw: Run openclaw skills list to confirm that the skill is loaded.

• Restart the Agent session and try again.

Q2: Why does the skill fail during execution?

• Confirm that the Alibaba Cloud CLI is installed and your credentials are configured (run aliyun configure list to check).

• Ensure that the RAM user has the necessary permissions to manage EMR Serverless Spark resources.

• Verify that the correct region is configured.

Q3: npx skills installs to the wrong directory?

By default, npx skills installs to ~/.agents/skills/, while Claude Code scans ~/.claude/skills/. Solution:

# Create a symbolic link
ln -s ~/.agents/skills/alibabacloud-emr-spark-manage ~/.claude/skills/

Or, configure the SessionStart Hook in ~/.claude/settings.json to synchronize automatically.

Q4: How do I update a skill?

# Check for updates
npx skills check

# Apply updates
npx skills update