Event-driven autoscaling based on ApsaraMQ for RocketMQ message accumulation metrics (KEDA) - Container Service for Kubernetes

Background

ApsaraMQ for RocketMQ is a high-performance, highly reliable, and highly scalable distributed messaging middleware that is widely used in enterprise applications. However, message accumulation can easily occur when you use ApsaraMQ for RocketMQ, especially under high load. Message accumulation can drive system load too high and even cause outages.

In this situation, you can use the Kubernetes event-driven autoscaling tool KEDA to trigger Horizontal Pod Autoscaler for your applications based on custom ApsaraMQ for RocketMQ message accumulation metrics. This solution helps you scale your applications automatically and efficiently, which improves system reliability and stability. If you use open source Apache RocketMQ, you can also achieve similar capabilities by exposing message data through the JMX Prometheus Exporter. For more information, see the Apache RocketMQ community.

This topic uses Managed Service for Prometheus as the data source and describes how to configure the scaling object for ApsaraMQ for RocketMQ message-based scaling.

Prerequisites

The ack-keda add-on is deployed. For more information, see Event-driven autoscaling.
An ApsaraMQ for RocketMQ 5.x series instance is created. For more information, see Create an instance.

A 5.x Serverless instance can quickly scale resources based on business load, supports allocating resources and calculating fees based on actual usage, and can effectively reduce costs. For more information, see 5.x Serverless Series.
The ApsaraMQ for RocketMQ (5.0) service is integrated in Application Real-Time Monitoring Service (ARMS).
The Go runtime environment is installed.

Step 1: Deploy a workload

This example creates a sample NGINX application named sample-app.

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Workloads > Deployments.

On the Deployments page, click Create from YAML. As prompted on the page, set Sample Template to Custom. Use the following sample code to create an NGINX application named sample-app.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
  namespace: default
  labels:
    app: sample-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      containers:
      - name: sample-app
        # Replace this with the image of your actual ApsaraMQ for RocketMQ consumer.
        image: alibaba-cloud-linux-3-registry.cn-hangzhou.cr.aliyuncs.com/alinux3/nginx_optimized:20240221-1.20.1-2.3.0  
        resources:
          limits:
            cpu: "500m"

Step 2: Configure scaling policies with a ScaledObject

You can configure KEDA scaling policies in a ScaledObject YAML file, including the scaling target, the maximum and minimum number of replicas, and the scaling threshold (message accumulation threshold). Before you configure the ScaledObject, you must get information such as the Prometheus URL of the ApsaraMQ for RocketMQ instance metrics.

1. Obtain ApsaraMQ for RocketMQ instance information in the ApsaraMQ for RocketMQ console

Log on to the ApsaraMQ for RocketMQ console. In the left-side navigation pane, click Instances.
In the top navigation bar, select a region, such as China (Hangzhou). On the Instances page, click the name of the instance that you want to manage.
In the left-side navigation pane, click Topic Management, and then view and record the value of Name and the Instance ID in the upper-right corner, for example, keda and mq-cn-uax33****.

2. Obtain the Prometheus data source of the ApsaraMQ for RocketMQ instance in the Managed Service for Prometheus console

Log on to the ARMS console.
In the left navigation pane, choose Managed Service for Prometheus > Instances.
For the target instance, click Cloud Services-{{RegionId}}. In the left-side navigation pane, click Settings, and then record the HTTP API Address (Grafana Read Address).

Select the URL of the Public Endpoint.

3. Create the ScaledObject YAML file

Use the following content to create ScaledObject.yaml and configure the scaling policy.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: prometheus-scaledobject
  namespace: default
spec:
  scaleTargetRef:
    name: sample-app
  maxReplicaCount: 10
  minReplicaCount: 2
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://cn-beijing.arms.aliyuncs.com:9090/api/v1/prometheus/8cba801fff65546a3012e9a684****/****538168824185/cloud-product-rocketmq/cn-beijing
      metricName: rocketmq_consumer_inflight_messages
      query: sum({__name__=~"rocketmq_consumer_ready_messages|rocketmq_consumer_inflight_messages",instance_id="rmq-cn-uax3xxxxxx",topic=~"keda"}) by (consumer_group)
      threshold: '30'

The following table describes the parameters.

Parameter	Description
`scaleTargetRef.name`	The scaling target. In this example, set this parameter to the application sample-app that you created in Step 1: Deploy a workload.
`maxReplicaCount`	The maximum number of replicas to scale out to.
`minReplicaCount`	The minimum number of replicas to scale in to.
`serverAddress`	The Prometheus URL that stores ApsaraMQ for RocketMQ metrics, which is the HTTP API Address (Grafana Read Address) that you recorded earlier.
`metricName`	The data that the PromQL query requests.
`query`	Aggregates the data that the PromQL query in metricName requests. In this example, the aggregation is the PromQL for the message accumulation.
`threshold`	The scaling threshold. This example uses a message accumulation of 30 as the threshold. A scale-out is triggered when the accumulation exceeds 30.

Run the following commands to deploy the file and view the created resources.

# Apply the scaling configuration.
kubectl apply -f ScaledObject.yaml
scaledobject.keda.sh/prometheus-scaledobject created
# Get the status of the scaling configuration.
kubectl get ScaledObject
NAME                      SCALETARGETKIND      SCALETARGETNAME   MIN   MAX   TRIGGERS     AUTHENTICATION   READY   ACTIVE   FALLBACK   AGE
prometheus-scaledobject   apps/v1.Deployment   sample-app        2     10    prometheus                    True    False    False      105s
# Check whether the HPA is generated.
kubectl get hpa
NAME                               REFERENCE               TARGETS      MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-prometheus-scaledobject   Deployment/sample-app   0/30 (avg)   2         10        2          28m

(Optional) Use a Prometheus token to improve the security of data reads, and configure verification of the Prometheus token.

Expand to view the detailed steps

Generate a Prometheus token as prompted on the page.

After the token is generated, the page displays the HTTP API Address (Grafana Read Address) information. The information includes the generated token value, which you can get in full by using the view and copy icons. It also lists the Prometheus access URLs for the Public Endpoint and Internal Endpoint network types. Record the URL and token value that you need to configure the ScaledObject YAML file later.

Create a Secret in which the values of the customAuthHeader: "Authorization" and customAuthValue fields must be Base64-encoded.

apiVersion: v1
kind: Secret
metadata:
  name: keda-prom-secret
  namespace: default
data:
  customAuthHeader: "QXV0Xxxxxxxlvbg=="
  customAuthValue: "kR2tpT2lJeFpXSmxaVFV6WlMTxxxxxxxxRMVFE0TUdRdE9USXpaQzFqWkRZd09EZ3dOVFV5WWpZaWZRLjlDaFBYU0Q2dEhWc1dQaFlyMGh3ZU5FQjZQZWVETXFjTlYydVNqOU82TTQ="

Use the following sample code to create a credential for KEDA to request data, and then deploy the credential to the cluster.

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: keda-prom-creds
  namespace: default
spec:
  secretTargetRef:
    - parameter: customAuthHeader
      name: keda-prom-secret
      key: customAuthHeader
    - parameter: customAuthValue
      name: keda-prom-secret
      key: customAuthValue

In the ScaledObject YAML file, configure the authenticationRef field and specify the name of the credential that you created.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: prometheus-scaledobject
  namespace: default
spec:
  scaleTargetRef:
    name: sample-app
  maxReplicaCount: 10
  minReplicaCount: 2
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://cn-beijing.arms.aliyuncs.com:9090/api/v1/prometheus/8cba801fff65546a3012e9a684****/****538168824185/cloud-product-rocketmq/cn-beijing
      metricName: rocketmq_consumer_inflight_messages
      query: sum({__name__=~"rocketmq_consumer_ready_messages|rocketmq_consumer_inflight_messages",instance_id="rmq-cn-uax3xxxxxx",topic=~"keda"}) by (consumer_group)
      threshold: '30'
      authModes: "custom"
    authenticationRef: # Configure this field.
        name: keda-prom-creds # Enter the name of the credential.

Note

This example uses the Custom authentication type. To use other authentication methods, see the KEDA community documentation.

Step 3: Produce and consume data

This example uses the rocketmq-keda-sample project to produce and consume data. In the project code, configure the endpoint, username, and password of the ApsaraMQ for RocketMQ instance that you recorded in Step 2.

Step 4: Trigger scale-out and scale-in by producing and consuming data

Log on to the ApsaraMQ for RocketMQ console. In the left-side navigation pane, click Instances.
In the top menu bar, select a region, such as China (Hangzhou). Then, in the instance list, click the name of the target instance to view and record the Endpoint and Network Information.
In the left-side navigation pane, click Access Control. Then, click the Intelligent Identity Recognition tab to view and record the instance username and instance password.
Run the Producer program to produce data, and then run the following command to view the HPA scaling status.
```
kubectl get hpa
```
Expected output:
```
NAME                               REFERENCE               TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-prometheus-scaledobject   Deployment/sample-app   32700m/30 (avg)   2         10        10         47m
```
The expected output indicates that the sample-app application has scaled out to the maximum number of replicas configured for KEDA.

Stop the Producer program, run the Consumer program, and then run the following command to view the HPA scaling status.

kubectl get hpa -w

Expected output:

NAME                               REFERENCE               TARGETS            MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-prometheus-scaledobject   Deployment/sample-app   222500m/30 (avg)   2         10        10         50m
keda-hpa-prometheus-scaledobject   Deployment/sample-app   232400m/30 (avg)   2         10        10         51m
keda-hpa-prometheus-scaledobject   Deployment/sample-app   0/30 (avg)         2         10        10         52m
keda-hpa-prometheus-scaledobject   Deployment/sample-app   0/30 (avg)         2         10        2          57m

The expected output indicates that, a period of time after data consumption ends, the sample-app application scales in to the minimum number of replicas configured for KEDA.

References

You can also use KEDA based on RabbitMQ metrics to monitor the queue length and message rate metrics. For more information, see Autoscale pods based on RabbitMQ metrics.