Community

Community Blog From Manual Operations to Automated Orchestration and AI Empowerment: The Evolution of Efficiency in ECS Disk Management

From Manual Operations to Automated Orchestration and AI Empowerment: The Evolution of Efficiency in ECS Disk Management

Section 1: Why I Undertook This Validation

During a weekly meeting between the TAM team and the customer, we learned about an operational pain point on the customer's side. The customer has a large ECS cluster. Whenever they need to change the specifications of disks attached to ECS instances, for example, PL0>PL1, they have to work through the instances one by one. This is time-consuming, prone to operational errors, and far from user-friendly. To address this pain point, we passed the VOC feedback to the product team and began pushing for product improvements. However, product managers generally have many customer requirements to handle and need to schedule them according to priority, so quickly delivering a new feature to resolve a customer's pain point is often difficult. We therefore recommended automated operations solutions such as OOS to address this kind of scenario quickly. However, the customer had a limited understanding of how Alibaba Cloud's features work and said they were unsure exactly how to use this feature to implement a solution.

At this point, the best option was to provide a minimal test scenario that could be validated, helping the customer quickly become familiar with Alibaba Cloud's product capabilities through hands-on practice and enabling them to iterate on the solution themselves. Compared with the previous approach of building everything manually from scratch, the assistance of an AI Agent allows us to achieve these goals quickly and efficiently.

Next, I will briefly share my end-to-end experience of using an AI Agent to address this customer's pain point. I hope it will be helpful to everyone.

To simplify the process, we narrowed the customer's pain point down to two scenarios with clear acceptance criteria.

● In the first scenario, only the instance IDs of two ECS instances are provided, and the program should automatically select all disks attached to those two instances.

● In the second scenario, both instance IDs and disk IDs are provided, and the program must process only the explicitly specified disks that are actually attached to those instances.

● Both scenarios must begin with prechecks, require explicit confirmation before execution, and output JSON evidence for each disk.

The question this article seeks to answer is not whether an API can be called, but whether the following workflow can be completed from end to end:

Create a repeatable test environment
  → Resolve instances and disk ownership
  → Eligibility checks and ModifyDiskSpec DryRun
  → Manual confirmation with --execute
  → Batch execution through OOS
  → Per-disk task audit
  → Upgrade selected disks to PL1 and leave unselected disks unchanged
  → Retain JSON evidence and complete cleanup

Section 2: Solution Overview and Architecture

Disclaimer: This solution is intended for learning in a test environment. It is not guaranteed to meet production best practices without modification.

Both Terraform and Python run in a local terminal. There is no need to log in to the test ECS instances, and no Guest OS scripts are executed. Terraform creates and cleans up the VPC, VSwitch, security group, 2 ECS instances, 2 additional data disks, OOS service role, and disk-level permissions. Python handles disk discovery, prechecks, starting OOS, waiting, auditing, and evidence output. The actual disk specification changes are performed in the cloud by the OOS public template ACS-ECS-BulkyModifyDiskSpec, which calls ModifyDiskSpec.

Permissions are divided into two layers. The operator uses a trusted RAM administrator account to complete the experiment, while OOS assumes the service role created by Terraform during execution. This service role allows ecs:ModifyDiskSpec only on the 4 disk ARNs created by Terraform for this experiment. The operator's broader permissions therefore do not give OOS permission to modify disks throughout the account.

The AI Agent is introduced here to assist with breaking down requirements, generating IaC and Python code, designing offline regression tests, checking official documentation, diagnosing actual errors, and ultimately compiling the work into a technical article.

See the diagram below for an overview of the solution architecture.

Section 3: Prerequisites and Environment Preparation

3.1 Test Scope

This article uses the Japan (Tokyo) region on Alibaba Cloud's international site as an example. The region ID is ap-northeast-1.

The experiment covers only standard ECS instances, single-attach ESSDs, and PL0→PL1 upgrades. PL0 supports 1–65536 GiB, and PL1 supports 20–65536 GiB, so 20 GiB is the minimum capacity that satisfies both in this experiment.

The test creates pay-as-you-go ECS instances and ESSDs and upgrades the disks from PL0 to PL1. OOS orchestration itself is free, but ECS instances, cloud disks, and the cloud services called by OOS are billed as usual. Upgraded cloud disks are billed at their new performance level. Actual costs depend on the instance specifications, cloud disk prices, and runtime in the region at the time. This article does not provide a fixed amount that has not been verified through a pricing inquiry. Run the test in a separate test account, and first confirm the account balance, quotas, and your organization's security requirements.

3.2 RAM Users, Permissions, and AccessKeys

Following Alibaba Cloud best practices, this solution does not use a Root AccessKey. If a trusted RAM administrator account already exists, it can be reused. Otherwise, an existing administrator should go to Identities > Users > Create User in the RAM console and create a RAM user solely for this experiment, such as pl-upgrade-admin. Select Permanent AccessKey for programmatic access and confirm that an AccessKey must be created. The system generates a key pair when the user is created; save it securely at that time. After creation, go to the user's Permissions > Add Permissions to grant permissions.

To streamline the internal test process, this example uses the same RAM administrator account for Terraform, Python/OOS, and cleanup. The operator needs AliyunECSFullAccess, AliyunVPCFullAccess, AliyunRAMFullAccess, and AliyunOOSFullAccess, or equivalent custom permissions.

Terraform's automatic instance selection calls GetPayAsYouGoPrice in the BSS OpenAPI, whose corresponding RAM Action is bss:DescribePrice, to find the lowest-priced ECS instance that meets the requirements and create it for this test. The following custom policy, restricted to ECS product price queries, must therefore also be attached:

{
  "Version": "1",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "bss:DescribePrice",
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "bssapi:ProductCode": "ecs"
        }
      }
    }
  ]
}

If a key pair was generated and saved when the user was created, do not create another one. If an existing RAM user has no usable key pair, a RAM administrator should go to that user's Credential > AccessKey > Create AccessKey, read the usage and security notices, complete the security verification, and then create the key pair.

Reminder: The AccessKey Secret is displayed only once. Do not put it in code, articles, screenshots, Git, or chats.

Reference image 1: Creating a RAM account and obtaining an AccessKey

Reference image 2: Creating and granting permissions

3.3 Local Terraform and Python

Before starting, refer to Section 8 to set up the local working directory and the three files to be run. Then enter the working directory and follow the operations below in order.

The end-to-end test in this article uses a zsh terminal on macOS. On other operating systems, use equivalent operations to perform the checks.

Use the following code to check whether brew is installed.

brew --version

If Homebrew is not installed, follow the official Homebrew installation page to install it.

Use the following code to check whether Terraform is installed.

terraform version

If Terraform is not installed, use the following code to install it.

brew tap hashicorp/tap
brew install hashicorp/tap/terraform

Use the following code to check whether Python is installed.

python3 --version

If Python is not installed, follow the code below to install it.

brew install python@3.11

Add Python 3.11 to the current terminal's PATH.

export PATH="$(brew --prefix python@3.11)/bin:$PATH"
python3.11 --version

Set up the Python runtime environment and dependency libraries.

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python run_upgrade.py --help

Enter the key pair interactively in the same zsh terminal. The following method does not write the Secret directly into the command history, but it will still be present in the current process environment. You will need to enter the key pair again after closing and restarting the CLI.

set +x
read -r "ALICLOUD_ACCESS_KEY?RAM AccessKey ID: "
read -r -s "ALICLOUD_SECRET_KEY?RAM AccessKey Secret: "
printf '\n'
export ALICLOUD_ACCESS_KEY ALICLOUD_SECRET_KEY

export ALIBABA_CLOUD_ACCESS_KEY_ID="$ALICLOUD_ACCESS_KEY"
export ALIBABA_CLOUD_ACCESS_KEY_SECRET="$ALICLOUD_SECRET_KEY"
unset ALICLOUD_SECURITY_TOKEN ALIBABA_CLOUD_SECURITY_TOKEN

export TF_VAR_region_id="ap-northeast-1"
export TF_VAR_name_prefix="pl-upgrade-blog"

Section 4: Operational Steps and Acceptance Criteria

4.1 Create Two ECS Instances and Four PL0 Disks

terraform init
terraform validate
terraform plan -out=tfplan
terraform apply tfplan
terraform output -json test | tee terraform-output.json

The plan should show 2 PayAsYouGo, NoSpot ECS instances, 2 additional data disks, and the VPC, VSwitch, security group, OOS role, policy, and attachments. The instances have no outbound public bandwidth configured, and no inbound rules are added to the security group. The ECS instances are set to be automatically released after approximately 4 hours as a cost safeguard, but this does not replace terraform destroy.

Automatic selection queries small ECS instance types in each availability zone and selects the lowest-priced option based on catalog prices for creation. If NO_TYPE is returned or there is no stock when placing the order, obtain a currently available zone and instance type from the console, specify them explicitly, and run plan again.

A successful run produces output like the following. Make a note of it, as this information will be used later.

test = {
  "account_id" = "5851XXXXXXXXX374"
  "data_disk_ids" = [
    "d-6we74XXXXXXXXXiw",
    "d-6we1zXXXXXXXXXyhft",
  ]
  "instance_ids" = [
    "i-6weXXXXXXXXX8ltny",
    "i-6weXXXXXXXXXjdu5v",
  ]
  "instance_type" = "ecs.e-c1m1.large"
  "oos_role" = "pl-upgrade-blog-oos-role"
  "region_id" = "ap-northeast-1"
  "system_disk_ids" = [
    "d-6we1XXXXXXXXX1gywo",
    "d-6we5XXXXXXXXX3dandy",
  ]
  "zone_id" = "ap-northeast-1a"
}

Reference image 3: As shown below, two ECS instances have been created, each with one PL0 system disk and one PL0 data disk attached.

4.2 Experiment A: Provide Only Instance IDs and Upgrade All Attached Disks

First run the prechecks without --execute: (Replace the portions in bold.)

python run_upgrade.py --region "$REGION" \
  --instance-ids "$I1" "$I2" \
  --oos-role "$ROLE" \
  --output all-preview.json

The expected output is result=PRECHECK_ONLY and precheckPassed=true, with selectedDiskIds containing 4 disks. When DryRun passes, the API returns DryRunOperation as specified in the official contract, and the disks remain unchanged.

After confirming that the correct disks are selected, explicitly execute the operation: (Replace the portions in bold and add the execute parameter.)

python run_upgrade.py --region "$REGION" \
  --instance-ids "$I1" "$I2" \
  --oos-role "$ROLE" \
  --execute \
  --output all-execute.json

The program waits for OOS to reach a terminal state, audits each per-disk loop task, and then reads the 4 disks again. All of the following acceptance criteria must be satisfied:

result == SUCCESS
oosStatus == Success
taskAudit.passed == true
verification.passed == true

Reference image 4: As shown below, the system disks and data disks of both ECS instances have all been upgraded to PL1.

4.3 Destroy and Rebuild a New PL0 Environment

Pay-as-you-go ESSDs cannot be downgraded back to PL0. Before Experiment B, the environment from Experiment A must be cleaned up, and entirely new instances and disks must be created:

terraform plan -destroy -out=destroy-a
terraform apply destroy-a
terraform state list

Recreate the test environment for Experiment B.

terraform plan -out=tfplan-b
terraform apply tfplan-b
terraform output -json test | tee terraform-output-b.json

4.4 Experiment B: Upgrade Only the ECS1 System Disk and the ECS2 Data Disk

Prechecks and execution. (Replace the portions in bold.)

python run_upgrade.py --region "$REGION" \
  --instance-ids "$I1" "$I2" \
  --disk-ids "$S1" "$D2" \
  --oos-role "$ROLE" \
  --output selected-preview.json

python run_upgrade.py --region "$REGION" \
  --instance-ids "$I1" "$I2" \
  --disk-ids "$S1" "$D2" \
  --oos-role "$ROLE" \
  --execute \
  --output selected-execute.json

The expected result of my test is that the ECS1 system disk and the ECS2 data disk change to PL1, while the ECS1 data disk and the ECS2 system disk remain at PL0.

Reminder: Use a new --output path for every run. If the program finds that the file already exists, it refuses to start to avoid overwriting evidence.

Reference image 5: As shown below, the ECS1 system disk and the ECS2 data disk have both been upgraded to PL1.

Section 5: Cleaning Up the Test Environment and Credentials

After confirming that OOS has reached a terminal state, run the following in the original directory:

terraform plan -destroy -out=destroy-plan
terraform apply destroy-plan
terraform state list

After completion, check in the console that no ECS instances, data disks, VPC, VSwitch, security group, OOS service role, or custom policy remain. Also delete the RAM account, permission policies, or AccessKey information used for the experiment in the console as needed.

Reference image 6: As shown below, the test resources, including ECS instances, have been removed.

Finally, clear the variables in the current terminal, including AccessKey variables:

unset ALICLOUD_ACCESS_KEY ALICLOUD_SECRET_KEY ALICLOUD_SECURITY_TOKEN
unset ALIBABA_CLOUD_ACCESS_KEY_ID ALIBABA_CLOUD_ACCESS_KEY_SECRET
unset ALIBABA_CLOUD_SECURITY_TOKEN
unset TF_VAR_region_id TF_VAR_name_prefix TF_VAR_zone_id
unset TF_VAR_instance_type TF_VAR_image_id

Section 6: Results of This Attempt

The value of this practice was not simply finding a way to adjust ECS disk performance levels in batches. More importantly, it moved our approach to technical support a step forward. In the past, when faced with similar requirements, we usually introduced optional tools such as Terraform and OOS to the customer and left them to research, combine, and validate these tools themselves. This time, we went further by turning the customer's pain point into a complete end-to-end test solution: from building the test environment, preparing permissions, and deploying resources to pre-change checks, batch execution, result verification, troubleshooting, and environment cleanup. We also provided Terraform configurations, a Python execution program, and operational documentation that could be used directly as references.

This delivery approach shortened the distance between the customer's understanding of product capabilities and actual validation of a solution. Instead of researching the relationships between different products from scratch, the customer can follow a clear operational path to complete validation quickly and understand how the solution works, the boundaries of its applicability, and its risk control methods. Building on this foundation, the customer can further extend capabilities such as resource selection, approval workflows, concurrency control, monitoring and alerting, and audit trails to suit their own environment, making this example a starting point for building an automated operations framework.

The AI Agent played an important role throughout the process. It assisted with requirement breakdown, architecture design, generation of Terraform and Python reference code, test scenario design, error diagnosis, and organization of the technical article. Work that previously required repeatedly switching between multiple product documents, code versions, and validation results was integrated into a continuous delivery workflow, allowing engineers to devote more time to confirming facts, assessing risks, and communicating with customers.

This efficiency improvement does not mean skipping necessary human judgment. On the contrary, the AI Agent took on a large amount of repetitive searching, organization, and initial generation work, while critical permission boundaries, execution risks, cloud test results, and suitability for the customer were still reviewed and confirmed by the engineer (TAM). Collaboration between people and AI improved both delivery speed and content completeness, while also making it easier to turn a one-off technical consultation into a reusable practical asset.

This practice also delivered value beyond the technical work itself. After learning how the solution was developed, the customer became very interested in how Alibaba Cloud's AI Agent could be applied and wanted to arrange a dedicated discussion to learn more about usage details and related pricing. This means that a requirement for automated disk operations led to new AI Agent application scenarios and potential business opportunities.

For the internal teams, this result also strengthened trust between the TAM team and the business team. TAM not only solved the customer's immediate technical problem, but also helped the business team discover new customer needs through a solution that could be executed, validated, and extended. The business team, in turn, could draw on this actual practice to discuss more targeted follow-up cooperation with the customer.

Therefore, this attempt delivered multiple benefits: the customer received more efficient and actionable technical support, the TAM team developed a reusable solution asset, and the business team discovered new opportunities for cooperation. A seemingly specific operational requirement ultimately connected customer success, team collaboration, and business development, while demonstrating the full evolution of efficiency from manual operations to automated operations and AI empowerment.

Please refer Japanese version from here: 手動運用から自動オーケストレーション、そしてAI活用へ：ECSディスク管理における効率化の進化

Section 7: Complete Reference Code

The minimal reproduction directory contains only three files:

.
├── main.tf
├── requirements.txt
└── run_upgrade.py

● main.tf creates the test resources and the OOS role.

● requirements.txt pins the ECS/OOS SDK versions and sets version ranges for the other dependencies.

● run_upgrade.py performs prechecks, execution, auditing, and evidence output.

● Save the following three code blocks as their corresponding files in the execution directory.

7.1 main.tf

terraform {
  required_version = ">= 1.5.0, < 2.0.0"
  required_providers {
    alicloud = {
      source  = "aliyun/alicloud"
      version = "= 1.294.0"
    }
  }
}

# Credentials are supplied by the RAM user, never stored in this file.
provider "alicloud" {
  region = var.region_id
}

variable "region_id" {
  description = "Test region, for example ap-northeast-1. Do not use a zone ID such as ap-northeast-1a."
  type        = string
}
variable "name_prefix" {
  description = "Unique resource prefix; keep it within the administrator-approved RAM policy scope."
  type        = string
  default     = "pl-upgrade-blog"
  validation {
    condition     = can(regex("^pl-upgrade-blog(-[a-z0-9]+)?$", var.name_prefix)) && length(var.name_prefix) <= 32
    error_message = "Use pl-upgrade-blog or pl-upgrade-blog-<lowercase-alphanumeric-suffix>, up to 32 characters."
  }
}
variable "zone_id" {
  description = "Optional availability zone; required with an explicit instance type."
  type        = string
  default     = ""
}
variable "instance_type" {
  description = "Optional ESSD-compatible instance type; set zone_id together with this value."
  type        = string
  default     = ""
}
variable "image_id" {
  description = "Optional x86_64 Linux image ID; otherwise use the latest Alibaba Cloud Linux 3 image."
  type        = string
  default     = ""
}

data "alicloud_account" "current" {}
data "alicloud_zones" "eligible" {
  available_resource_creation = "VSwitch"
  available_disk_category     = "cloud_essd"
}
data "alicloud_images" "linux" {
  count       = var.image_id == "" ? 1 : 0
  owners      = "system"
  name_regex  = "^aliyun_3_x64_20G_alibase_.*"
  most_recent = true
}

# Compare catalog prices, not real-time inventory, across five small shapes.
locals {
  image = var.image_id != "" ? var.image_id : data.alicloud_images.linux[0].images[0].id
  zones = var.zone_id != "" ? [var.zone_id] : [for z in data.alicloud_zones.eligible.zones : z.id]
  queries = {
    for q in flatten([
      for z in local.zones : [
        for shape in [[1, 1], [1, 2], [2, 2], [2, 4], [2, 8]] : {
          key = "${z}-${shape[0]}-${shape[1]}", zone = z, cpu = shape[0], memory = shape[1]
        }
      ]
    ]) : q.key => q
  }
  tags = { ManagedBy = "Terraform", Purpose = "essd-pl0-to-pl1", TestGroup = var.name_prefix }
}
data "alicloud_instance_types" "small" {
  for_each             = var.instance_type == "" ? local.queries : {}
  availability_zone    = each.value.zone
  cpu_core_count       = each.value.cpu
  memory_size          = each.value.memory
  instance_charge_type = "PostPaid"
  system_disk_category = "cloud_essd"
  image_id             = local.image
  sorted_by            = "Price"
}
locals {
  prices = sort(flatten([
    for key, result in data.alicloud_instance_types.small : [
      for item in result.instance_types :
      format("%020.8f|%s|%s", tonumber(item.price), local.queries[key].zone, item.id)
    ]
  ]))
  cheapest = split("|", try(local.prices[0], "0|NO_ZONE|NO_TYPE"))
  zone     = var.instance_type != "" ? var.zone_id : local.cheapest[1]
  type     = var.instance_type != "" ? var.instance_type : local.cheapest[2]
}

# Exactly two PayAsYouGo instances; each starts with two 20-GiB PL0 disks.
resource "alicloud_vpc" "test" {
  vpc_name   = "${var.name_prefix}-vpc"
  cidr_block = "172.31.0.0/16"
  tags       = local.tags
}
resource "alicloud_vswitch" "test" {
  vswitch_name = "${var.name_prefix}-vsw"
  vpc_id       = alicloud_vpc.test.id
  zone_id      = local.zone
  cidr_block   = "172.31.1.0/24"
  tags         = local.tags
}
resource "alicloud_security_group" "test" {
  security_group_name = "${var.name_prefix}-sg"
  description         = "Isolated disk-upgrade test; no inbound rules"
  vpc_id              = alicloud_vpc.test.id
  security_group_type = "normal"
  tags                = local.tags
}
resource "alicloud_instance" "test" {
  count                         = 2
  instance_name                 = "${var.name_prefix}-ecs-${count.index + 1}"
  availability_zone             = local.zone
  image_id                      = local.image
  instance_type                 = local.type
  instance_charge_type          = "PostPaid"
  spot_strategy                 = "NoSpot"
  status                        = "Running"
  vswitch_id                    = alicloud_vswitch.test.id
  security_groups               = [alicloud_security_group.test.id]
  internet_max_bandwidth_out    = 0
  system_disk_category          = "cloud_essd"
  system_disk_performance_level = "PL0"
  system_disk_size              = 20
  auto_release_time             = timeadd(timestamp(), "4h")
  tags                          = local.tags
  volume_tags                   = local.tags
  lifecycle {
    # OOS owns the PL change; do not try to downgrade it on a later apply.
    ignore_changes = [auto_release_time, system_disk_performance_level]
    precondition {
      condition     = local.type != "NO_TYPE" && (var.instance_type == "" || var.zone_id != "")
      error_message = "No small ESSD-compatible candidate found, or zone_id is missing. Set zone_id and instance_type explicitly."
    }
  }
}
resource "alicloud_ecs_disk" "data" {
  count                = 2
  disk_name            = "${var.name_prefix}-data-${count.index + 1}"
  zone_id              = local.zone
  category             = "cloud_essd"
  performance_level    = "PL0"
  size                 = 20
  payment_type         = "PayAsYouGo"
  delete_with_instance = true
  multi_attach         = "Disabled"
  tags                 = local.tags
  lifecycle {
    ignore_changes = [performance_level]
  }
}
resource "alicloud_ecs_disk_attachment" "data" {
  count                = 2
  disk_id              = alicloud_ecs_disk.data[count.index].id
  instance_id          = alicloud_instance.test[count.index].id
  delete_with_instance = true
}

# OOS may assume this service role. The role may modify only the four test disks.
resource "alicloud_ram_role" "oos" {
  role_name = "${var.name_prefix}-oos-role"
  assume_role_policy_document = jsonencode({
    Version   = "1"
    Statement = [{ Effect = "Allow", Action = "sts:AssumeRole", Principal = { Service = ["oos.aliyuncs.com"] } }]
  })
  description = "Temporary service role for the ESSD PL test"
}
resource "alicloud_ram_policy" "oos" {
  policy_name = "${var.name_prefix}-oos-disk-upgrade"
  policy_document = jsonencode({
    Version = "1"
    Statement = [{
      Effect = "Allow", Action = ["ecs:ModifyDiskSpec"]
      Resource = [for id in concat(alicloud_instance.test[*].system_disk_id, alicloud_ecs_disk.data[*].id) :
      "acs:ecs:${var.region_id}:${data.alicloud_account.current.id}:disk/${id}"]
    }]
  })
  description = "ModifyDiskSpec on the four Terraform-managed disks only"
}
resource "alicloud_ram_role_policy_attachment" "oos" {
  role_name   = alicloud_ram_role.oos.role_name
  policy_name = alicloud_ram_policy.oos.policy_name
  policy_type = alicloud_ram_policy.oos.type
}

# One JSON output supplies every launcher argument without console lookups.
output "test" {
  value = {
    account_id      = data.alicloud_account.current.id
    region_id       = var.region_id
    zone_id         = local.zone
    instance_type   = local.type
    instance_ids    = alicloud_instance.test[*].id
    system_disk_ids = alicloud_instance.test[*].system_disk_id
    data_disk_ids   = alicloud_ecs_disk.data[*].id
    oos_role        = alicloud_ram_role.oos.role_name
  }
  depends_on = [alicloud_ecs_disk_attachment.data, alicloud_ram_role_policy_attachment.oos]
}

7.2 requirements.txt

alibabacloud-credentials>=1.0.2,<2.0.0
alibabacloud-ecs20140526==7.11.6
alibabacloud-oos20190601==3.7.0
alibabacloud-tea-openapi>=0.3.12,<1.0.0
alibabacloud-tea-util>=0.3.13,<1.0.0
urllib3>=1.26,<2.0

7.3 run_upgrade.py

#!/usr/bin/env python3
"""Batch-upgrade attached ESSD PL0 disks through OOS; preview is the default.

Use RAM credentials. Instance IDs select all attached disks unless --disk-ids
is supplied. No local configuration file is required. Every --execute run
waits, audits the modification tasks, and verifies selected/unselected disks.
"""
from __future__ import annotations

import argparse
import json
import sys
import time
import uuid
from collections import Counter
from datetime import datetime, timezone
from pathlib import Path

from Tea.exceptions import TeaException
from alibabacloud_credentials.client import Client as Credentials
from alibabacloud_ecs20140526.client import Client as ECS
from alibabacloud_ecs20140526 import models as em
from alibabacloud_oos20190601.client import Client as OOS
from alibabacloud_oos20190601 import models as om
from alibabacloud_tea_openapi import models as api
from alibabacloud_tea_util.models import RuntimeOptions

TEMPLATE = "ACS-ECS-BulkyModifyDiskSpec"
TERMINAL = {"Success", "Failed", "Cancelled", "Terminated", "TimedOut"}
FIELDS = ("disk_id", "instance_id", "type", "category", "performance_level",
          "size", "status", "multi_attach")


def client(cls, service, region):
    """Use the default Credentials chain, with bounded network timeouts."""
    cfg = api.Config(credential=Credentials(), connect_timeout=10000, read_timeout=30000)
    cfg.endpoint = f"{service}.{region}.aliyuncs.com"
    return cls(cfg)


def error(exc):
    data = getattr(exc, "data", None) or {}
    code = getattr(exc, "code", None) or getattr(exc, "error_code", None)
    if not code and isinstance(data, dict):
        code = data.get("Code") or data.get("code")
    return {"code": str(code or type(exc).__name__), "message": str(getattr(exc, "message", None) or exc)}


def facts(disk):
    return {key: getattr(disk, key, None) for key in FIELDS}


def attached_disks(ecs, region, instances):
    """Read every disk page; truncation would violate all-attached selection."""
    result = {}
    for instance in instances:
        page = 1
        while True:
            req = em.DescribeDisksRequest(region_id=region, instance_id=instance, page_number=page, page_size=100)
            body = ecs.describe_disks_with_options(req, RuntimeOptions()).body
            rows = getattr(getattr(body, "disks", None), "disk", None) or []
            for row in rows:
                if row.disk_id in result:
                    raise RuntimeError("Duplicate disk in discovery; selection is ambiguous")
                result[row.disk_id] = facts(row)
            if len(rows) < 100:
                break
            page += 1
    return result


def select(disks, requested):
    """Fail closed: explicit disks must belong to one of the supplied instances."""
    missing = sorted(set(requested) - set(disks))
    if missing:
        raise ValueError(f"Specified disks are not attached to the supplied instances: {missing}")
    selected = list(dict.fromkeys(requested)) if requested else sorted(disks)
    if not selected:
        raise ValueError("No attached disks found")
    return selected


def eligibility(disk):
    checks = {
        "type must be system or data": disk["type"] in ("system", "data"),
        "category must be cloud_essd": disk["category"] == "cloud_essd",
        "performance level must be PL0": disk["performance_level"] == "PL0",
        "capacity must be at least 20 GiB": isinstance(disk["size"], (int, float)) and disk["size"] >= 20,
        "disk must be In_use": disk["status"] == "In_use",
        "multi-attach must be disabled": disk["multi_attach"] in ("Disabled", "disabled", False),
    }
    return [message for message, passed in checks.items() if not passed]


def dry_run(ecs, disk_id):
    """DryRunOperation is the expected success exception, not a normal 2xx."""
    req = em.ModifyDiskSpecRequest(disk_id=disk_id, disk_category="cloud_essd", performance_level="PL1", dry_run=True)
    try:
        ecs.modify_disk_spec_with_options(req, RuntimeOptions())
        return {"passed": False, "code": "UnexpectedSuccess"}
    except TeaException as exc:
        detail = error(exc)
        return {"passed": detail["code"] == "DryRunOperation", **detail}
    except Exception as exc:
        return {"passed": False, **error(exc)}


def loop_item(value):
    if value is None or not str(value).strip():
        return None
    text = str(value).strip()
    try:
        decoded = json.loads(text)
        return decoded if isinstance(decoded, str) else text
    except json.JSONDecodeError:
        return text


def task_rows(oos, region, execution):
    """Retain child tasks and follow every token without accepting token cycles."""
    rows, token, seen = [], None, set()
    while True:
        req = om.ListTaskExecutionsRequest(region_id=region, execution_id=execution,
                                          include_child_task_execution=True, max_results=100, next_token=token)
        body = oos.list_task_executions(req).body
        rows.extend({"taskName": row.task_name, "taskAction": row.task_action,
                     "loopItem": loop_item(row.loop_item), "status": row.status,
                     "statusMessage": row.status_message}
                    for row in (body.task_executions or []))
        token = body.next_token
        if not token:
            return rows
        if token in seen:
            raise RuntimeError("Repeated OOS task pagination token")
        seen.add(token)


def audit(rows, selected):
    """The public template has one modifyDiskSpec/ACS::ExecuteApi loop per disk."""
    iterations = [r for r in rows if r["loopItem"] and r["taskName"] == "modifyDiskSpec"
                  and r["taskAction"] == "ACS::ExecuteApi"]
    counts = Counter(r["loopItem"] for r in iterations)
    expected = set(selected)
    missing, unexpected = sorted(expected - set(counts)), sorted(set(counts) - expected)
    duplicates = {key: count for key, count in counts.items() if count != 1}
    failed = [r for r in iterations if r["status"] != "Success"]
    return {"passed": not (missing or unexpected or duplicates or failed), "expectedDiskIds": selected,
            "missingDiskIds": missing, "unexpectedDiskIds": unexpected, "duplicates": duplicates,
            "failedTasks": failed, "observedLoopItems": [r["loopItem"] for r in iterations]}


def poll(fetch, ready, timeout, interval):
    """Bounded polling; incomplete API visibility never counts as success."""
    deadline = time.monotonic() + timeout
    while True:
        value = fetch()
        if ready(value):
            return value
        remaining = deadline - time.monotonic()
        if remaining <= 0:
            return value
        time.sleep(min(interval, remaining))


def verification(current, baseline, selected):
    """Check PL1 and unchanged identity/category/size/attachment for every disk."""
    target, unselected = [], []
    selected = set(selected)
    for disk_id, before in baseline.items():
        after = current.get(disk_id)
        desired = "PL1" if disk_id in selected else before["performance_level"]
        stable = bool(after and all(after[k] == before[k] for k in FIELDS if k != "performance_level"))
        row = {"diskId": disk_id, "expectedLevel": desired, "before": before, "after": after,
               "passed": bool(stable and after["performance_level"] == desired)}
        (target if disk_id in selected else unselected).append(row)
    unexpected = sorted(set(current) - set(baseline))
    missing = sorted(set(baseline) - set(current))
    return {"selected": target, "unselected": unselected,
            "unexpectedDiskIds": unexpected, "missingDiskIds": missing,
            "passed": not (unexpected or missing) and all(r["passed"] for r in target + unselected)}


def save(report, path):
    if path:
        path.parent.mkdir(parents=True, exist_ok=True)
        path.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8")


def run(args, report):
    # Validate instances and capture the complete baseline before any mutation.
    instances = list(dict.fromkeys(args.instance_ids))
    ecs = client(ECS, "ecs", args.region)
    req = em.DescribeInstancesRequest(region_id=args.region, instance_ids=json.dumps(instances), page_size=100)
    body = ecs.describe_instances_with_options(req, RuntimeOptions()).body
    found = {r.instance_id: r.status for r in (getattr(getattr(body, "instances", None), "instance", None) or [])}
    if set(found) != set(instances) or any(s not in ("Running", "Stopped") for s in found.values()):
        raise ValueError("An instance is missing, unauthorized, or not Running/Stopped")
    baseline = attached_disks(ecs, args.region, instances)
    selected = select(baseline, args.disk_ids or [])
    report.update(instanceIds=instances, selectedDiskIds=selected, baseline=baseline,
                  selectionMode="explicit-disks" if args.disk_ids else "all-attached-disks")
    checks = []
    for disk_id in selected:
        failures = eligibility(baseline[disk_id])
        check = {"diskId": disk_id, "eligibilityFailures": failures,
                 "dryRun": None if failures else dry_run(ecs, disk_id)}
        checks.append(check)
    report["prechecks"] = checks
    report["precheckPassed"] = all(not c["eligibilityFailures"] and c["dryRun"]["passed"] for c in checks)
    if not report["precheckPassed"]:
        report["result"] = "BLOCKED_BY_PRECHECK"
        return 2
    if not args.execute:
        report["result"] = "PRECHECK_ONLY"
        return 0

    # Protect the precheck/submit gap and record the idempotency token before start.
    if attached_disks(ecs, args.region, instances) != baseline:
        raise RuntimeError("Disk inventory changed after prechecks; execution blocked")
    oos = client(OOS, "oos", args.region)
    token = str(uuid.uuid4())
    parameters = {"regionId": args.region, "diskIds": selected, "performanceLevel": "PL1",
                  "diskCategory": "cloud_essd", "OOSAssumeRole": args.oos_role,
                  "rateControl": {"Mode": "Concurrency", "MaxErrors": 0,
                                  "Concurrency": min(args.concurrency, len(selected))}}
    report.update(stage="START_OOS", clientToken=token, parameters=parameters, result="START_PENDING")
    save(report, args.output)
    req = om.StartExecutionRequest(region_id=args.region, template_name=TEMPLATE, mode="Automatic",
                                   loop_mode="Automatic", safety_check="Skip", client_token=token,
                                   parameters=json.dumps(parameters))
    started = oos.start_execution(req).body.execution
    execution_id = getattr(started, "execution_id", None)
    if not execution_id:
        raise RuntimeError("StartExecution returned no execution ID; inspect OOS before retrying")
    report.update(executionId=execution_id, stage="WAIT_OOS", result="OOS_STARTED")
    save(report, args.output)

    def status():
        # ListExecutions requires MaxResults between 10 and 100, even with an execution ID filter.
        request = om.ListExecutionsRequest(region_id=args.region, execution_id=execution_id, max_results=10)
        rows = oos.list_executions(request).body.executions or []
        return {"status": rows[0].status, "message": rows[0].status_message} if rows else {"status": "NotVisible"}

    state = poll(status, lambda s: s["status"] in TERMINAL, args.timeout, args.interval)
    report.update(oosStatus=state["status"], oosMessage=state.get("message"), stage="AUDIT_AND_VERIFY")
    # Allow completed task evidence to become visible, but keep strict auditing.
    rows = poll(lambda: task_rows(oos, args.region, execution_id),
                lambda r: audit(r, selected)["passed"], min(60, args.timeout), args.interval)
    report.update(taskExecutions=rows, taskAudit=audit(rows, selected))
    final = poll(lambda: verification(attached_disks(ecs, args.region, instances), baseline, selected),
                 lambda v: v["passed"], args.timeout if state["status"] == "Success" else 0, args.interval)
    report["verification"] = final
    if state["status"] != "Success":
        report["result"] = "OOS_FAILED" if state["status"] in TERMINAL else "OOS_WAIT_TIMEOUT"
        return 3
    if not report["taskAudit"]["passed"]:
        report["result"] = "TASK_AUDIT_FAILED"
        return 4
    if not final["passed"]:
        report["result"] = "DISK_VERIFICATION_FAILED"
        return 5
    report.update(result="SUCCESS", stage="COMPLETED")
    return 0


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--region", required=True)
    parser.add_argument("--instance-ids", nargs="+", required=True)
    parser.add_argument("--disk-ids", nargs="+", help="Only these attached disks; otherwise all attached disks")
    parser.add_argument("--oos-role", required=True, help="Terraform-created OOS service role name")
    parser.add_argument("--execute", action="store_true", help="Modify, wait, audit, and verify")
    parser.add_argument("--concurrency", type=int, default=2)
    parser.add_argument("--timeout", type=int, default=1800)
    parser.add_argument("--interval", type=int, default=10)
    parser.add_argument("--output", type=Path, help="Optional JSON evidence path")
    args = parser.parse_args()
    if min(args.concurrency, args.timeout, args.interval) < 1 or len(set(args.instance_ids)) > 100:
        parser.error("Positive concurrency/timeout/interval and at most 100 unique instance IDs are required")
    if args.output and args.output.exists():
        parser.error("Evidence file already exists; choose a new --output path")
    report = {"generatedAt": datetime.now(timezone.utc).isoformat(), "regionId": args.region,
              "templateName": TEMPLATE, "executeRequested": args.execute, "stage": "PRECHECK"}
    try:
        code = run(args, report)
    except Exception as exc:
        report.update(result="ERROR", error=error(exc))
        code = 1
    try:
        save(report, args.output)
    except Exception as exc:
        # Preserve stdout evidence even when the destination is not writable.
        report.update(result="ERROR", evidenceSaveError=error(exc))
        code = 1
    print(json.dumps(report, indent=2))
    return code


if __name__ == "__main__":
    sys.exit(main())

Terraform OOS AI Agent

0 1 0

Share on

Read previous post:

「洗車問題」から複数モデルの合意に基づく回答システムへ：Model Studio連携のエンドツーエンド実践