Offload the status of large Argo workflows to ApsaraDB RDS for MySQL - Container Service for Kubernetes

Workflows are stored as Kubernetes resources in workflow clusters, with a size limit of 1 MB per resource. Task statuses accumulate in the /status/nodes field — when a workflow has a large number of tasks, the resource can exceed this limit and cause exceptions during status updates. To resolve this issue, you can offload the status information to an external SQL database. This topic describes how to configure ApsaraDB RDS for MySQL as the offload target to support large workflows.

Prerequisites

Before you begin, make sure you have:

A workflow created. For more information, see Create a workflow.
A kubectl client connected to the workflow cluster. For more information, see Connect to a cluster using kubectl.
An ApsaraDB RDS for MySQL instance created. For more information, see Create an ApsaraDB RDS for MySQL instance.

Important
The RDS instance must be deployed in the same virtual private cloud (VPC) as the workflow cluster, and the VPC CIDR block must be added to the whitelist of the RDS instance.
A database and a database account created. For more information, see Create a database and account. The account must have read and write permissions on the database. Record the database name, username, password, and the RDS instance endpoint. To view the endpoint, see View and manage instance endpoints and ports.

Configure workflow offloading

Step 1: Create a Secret for database credentials

Create a file named secret.yaml with the following content:

apiVersion: v1
stringData:
  username: <database-username>  # Replace with the database username.
  password: <database-password>  # Replace with the database password.
kind: Secret
metadata:
  name: argo-mysql-config
  namespace: default
type: Opaque

Apply the Secret to your cluster:

kubectl apply -f secret.yaml

Step 2: Configure persistence in workflow-controller-configmap

Modify the workflow-controller-configmap to add the persistence section with nodeStatusOffLoad: true. For details on how to edit the ConfigMap, see Modify the configuration of a workflow cluster.

  persistence: |
    connectionPool:
      maxIdleConns: 100
      maxOpenConns: 0
      connMaxLifetime: 0s  # 0 means connections don't have a max lifetime.
    archiveTTL: 30d
    nodeStatusOffLoad: true
    mysql:
      host: rm-****.mysql.cn-beijing.rds.aliyuncs.com
      port: 3306
      database: argo-workflow  # The database you created in the prerequisites.
      tableName: argo_workflows
      userNameSecret:
        name: argo-mysql-config
        key: username
      passwordSecret:
        name: argo-mysql-config
        key: password

Parameter reference

Parameter	Description
`nodeStatusOffLoad`	Set to `true` to enable offloading of workflow node status to an external database.
`connectionPool.maxIdleConns`	Maximum number of idle connections in the connection pool.
`connectionPool.maxOpenConns`	Maximum number of open connections. `0` means no limit.
`connectionPool.connMaxLifetime`	Maximum lifetime of a connection. `0s` means connections do not expire.
`archiveTTL`	Retention period for archived workflows. Example: `30d`.
`mysql.host`	Endpoint of the ApsaraDB RDS for MySQL instance.
`mysql.port`	Port of the RDS instance. The default MySQL port is `3306`.
`mysql.database`	Name of the database.
`mysql.tableName`	Name of the table used to store workflow data.
`mysql.userNameSecret`	Reference to the Kubernetes Secret key that stores the database username.
`mysql.passwordSecret`	Reference to the Kubernetes Secret key that stores the database password.

Step 3: Restart the Argo Server pod

After saving the ConfigMap, restart the Argo Server pod:

kubectl delete pod argo-server-xxx -n <cluster-id>

Replace argo-server-xxx with the actual pod name and <cluster-id> with your cluster ID.

Verify that the pod restarts and reaches the Running state before proceeding.

Troubleshooting

The Argo Server pod keeps restarting after the configuration change.

This usually means the workflow cluster cannot connect to the RDS instance. Check the following:

The RDS instance and the workflow cluster are in the same VPC.
The VPC CIDR block is added to the whitelist of the RDS instance.
The endpoint, port, database name, and credentials in the ConfigMap are correct.

What's next

To view workflow status and understand billing, see Create a workflow.
To learn more about the offloading mechanism, see Offloading Large Workflows in the Argo Workflows documentation.