Event-driven autoscaling (KEDA) based on ApsaraMQ for RocketMQ message accumulation metrics - Container Service for Kubernetes

When you use ApsaraMQ for RocketMQ, message accumulation can cause high system loads. To prevent service crashes and improve system reliability and stability, you can use Kubernetes Event-driven Autoscaling (KEDA) as an autoscaling solution. KEDA lets you implement automated and efficient horizontal pod autoscaling (HPA) based on ApsaraMQ for RocketMQ message accumulation metrics.

Background

ApsaraMQ for RocketMQ is a high-performance, highly reliable, and highly scalable distributed messaging middleware that is widely used in enterprise applications. However, message accumulation can occur, especially under high load. This can lead to excessive system load and even cause service crashes.

In this scenario, you can use the Kubernetes event-driven autoscaling tool KEDA to trigger horizontal pod autoscaling for your application based on custom ApsaraMQ for RocketMQ message accumulation metrics. This solution provides automated and efficient application scaling to improve system reliability and stability. If you use open-source Apache RocketMQ, you can achieve similar capabilities by exposing message data through a Java Management Extensions (JMX) Prometheus Exporter. For more information, visit the Apache RocketMQ community.

This topic describes how to configure a KEDA ScaledObject to work with ApsaraMQ for RocketMQ, using Managed Service for Prometheus as the data source.

Prerequisites

The ack-keda add-on is deployed. For more information, see event-driven autoscaling.
An ApsaraMQ for RocketMQ 5.x instance is created. For more information, see Create an instance.
A 5.x serverless instance rapidly scales resources based on business workloads. It allocates resources and calculated fees based on actual usage, saving costs. For more information, see Overview of 5.x serverless instances.
The ApsaraMQ for RocketMQ (5.0) service is integrated in the ARMS console.
The Go runtime environment is installed.

Step 1: Deploy a workload

This example uses a sample NGINX application named sample-app.

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Workloads > Deployments.

On the Deployments page, click Create from YAML. Set Sample Template to Custom, and use the following sample code to create an NGINX application named sample-app.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
  namespace: default
  labels:
    app: sample-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      containers:
      - name: sample-app
        # Replace this with the image of your actual ApsaraMQ for RocketMQ consumer.
        image: alibaba-cloud-linux-3-registry.cn-hangzhou.cr.aliyuncs.com/alinux3/nginx_optimized:20240221-1.20.1-2.3.0  
        resources:
          limits:
            cpu: "500m"

Step 2: Configure a ScaledObject

Configure KEDA scaling policies in a ScaledObject YAML file. These policies include the target to scale, the minimum and maximum number of replicas, and the scaling threshold, such as the message accumulation threshold. Before you configure the ScaledObject, you must obtain the Prometheus URL for your ApsaraMQ for RocketMQ instance metrics.

1. Obtain instance information in the ApsaraMQ for RocketMQ console

Log on to the ApsaraMQ for RocketMQ console. In the left-side navigation pane, click Instances.
In the top navigation bar, select a region, such as China (Hangzhou). On the Instances page, click the name of the instance that you want to manage.
In the left-side navigation pane, click Topics. View and record the topic Name and the Instance ID in the upper-right corner. For example, the topic name is keda and the instance ID is mq-cn-uax33****.

2. Obtain the Prometheus data source of the ApsaraMQ for RocketMQ instance in the Prometheus console

Log on to the ARMS console.
In the left navigation pane, choose Managed Service for Prometheus > Instances.
Click Cloud Services-{{RegionId}} for the target instance. In the left-side navigation pane, click Settings and record the HTTP API Address (Grafana Read Address).
You must use the URL for the Public Endpoint.

3. Create the ScaledObject YAML file

Use the following content to create a file named ScaledObject.yaml to configure the scaling policy.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: prometheus-scaledobject
  namespace: default
spec:
  scaleTargetRef:
    name: sample-app
  maxReplicaCount: 10
  minReplicaCount: 2
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://cn-beijing.arms.aliyuncs.com:9090/api/v1/prometheus/8cba801fff65546a3012e9a684****/****538168824185/cloud-product-rocketmq/cn-beijing
      metricName: rocketmq_consumer_inflight_messages
      query: sum({__name__=~"rocketmq_consumer_ready_messages|rocketmq_consumer_inflight_messages",instance_id="rmq-cn-uax3xxxxxx",topic=~"keda"}) by (consumer_group)
      threshold: '30'

The following table describes the parameters.

Parameter	Description
`scaleTargetRef.name`	The workload to scale. In this example, set to `sample-app`, the application created in Step 1: Deploy a workload.
`maxReplicaCount`	The maximum number of replicas to scale out to.
`minReplicaCount`	The minimum number of replicas to scale in to.
`serverAddress`	The URL of the Prometheus service for the ApsaraMQ for RocketMQ metrics. Use the HTTP API Address (Grafana Read Address) that you recorded earlier.
`metricName`	The data that the PromQL query requests.
`query`	Aggregates the data that the PromQL in `metricName` requests. In this example, the aggregation is the PromQL for message accumulation.
`threshold`	The threshold for scaling. This example uses a message accumulation of 30 as the threshold. A scale-out is triggered when the accumulation exceeds 30.

Run the following commands to deploy the file and view the created resources.

# Apply the scaling configuration.
kubectl apply -f ScaledObject.yaml
scaledobject.keda.sh/prometheus-scaledobject created
# Get the status of the scaling configuration.
kubectl get ScaledObject
NAME                      SCALETARGETKIND      SCALETARGETNAME   MIN   MAX   TRIGGERS     AUTHENTICATION   READY   ACTIVE   FALLBACK   AGE
prometheus-scaledobject   apps/v1.Deployment   sample-app        2     10    prometheus                    True    False    False      105s
# Check the status of the generated HPA.
kubectl get hpa
NAME                               REFERENCE               TARGETS      MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-prometheus-scaledobject   Deployment/sample-app   0/30 (avg)   2         10        2          28m

(Optional) Use a Prometheus token for authentication for more secure data reads.

Detailed steps

Generate a Prometheus token as prompted on the page.
After the token is generated, the page displays the HTTP API Address (Grafana Read Address) information. This information includes the generated token value, whose full value you obtain from the view and copy icons, and the Prometheus access URLs for the Public Endpoint and Internal Endpoint network types. Record the URL and the token value that you need for the ScaledObject YAML file configuration.

Create a Secret. The values for the customAuthHeader: "Authorization" and customAuthValue fields must be Base64-encoded.

apiVersion: v1
kind: Secret
metadata:
  name: keda-prom-secret
  namespace: default
data:
  customAuthHeader: "QXV0Xxxxxxxlvbg=="
  customAuthValue: "kR2tpT2lJeFpXSmxaVFV6WlMTxxxxxxxxRMVFE0TUdRdE9USXpaQzFqWkRZd09EZ3dOVFV5WWpZaWZRLjlDaFBYU0Q2dEhWc1dQaFlyMGh3ZU5FQjZQZWVETXFjTlYydVNqOU82TTQ="

Use the following sample code to create a KEDA TriggerAuthentication resource and deploy it to the cluster.

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: keda-prom-creds
  namespace: default
spec:
  secretTargetRef:
    - parameter: customAuthHeader
      name: keda-prom-secret
      key: customAuthHeader
    - parameter: customAuthValue
      name: keda-prom-secret
      key: customAuthValue

In the ScaledObject YAML file, configure the authenticationRef field with the name of the credential you created.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: prometheus-scaledobject
  namespace: default
spec:
  scaleTargetRef:
    name: sample-app
  maxReplicaCount: 10
  minReplicaCount: 2
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://cn-beijing.arms.aliyuncs.com:9090/api/v1/prometheus/8cba801fff65546a3012e9a684****/****538168824185/cloud-product-rocketmq/cn-beijing
      metricName: rocketmq_consumer_inflight_messages
      query: sum({__name__=~"rocketmq_consumer_ready_messages|rocketmq_consumer_inflight_messages",instance_id="rmq-cn-uax3xxxxxx",topic=~"keda"}) by (consumer_group)
      threshold: '30'
      authModes: "custom"
    authenticationRef: # The authentication reference.
        name: keda-prom-creds # The name of the TriggerAuthentication resource.

Note

This example uses the Custom authentication type. For other authentication methods, see the KEDA community documentation.

Step 3: Produce and consume data

This example uses the rocketmq-keda-sample project to produce and consume data. In the project's code, configure the endpoint, username, and password for the ApsaraMQ for RocketMQ instance with the values that you recorded in Step 2.

Step 4: Use produced and consumed data to trigger autoscaling

Log on to the ApsaraMQ for RocketMQ console. In the left-side navigation pane, click Instances.
In the top menu bar, select a region, such as China (Hangzhou). In the instance list, click the name of your target instance to view and record the Endpoint and Network Information.
In the left-side navigation pane, click Access Control. Then, click the Intelligent Identity Recognition tab to view and record the instance username and password.
Run the producer program to generate data. Then, run the following command to check the HPA status.
```
kubectl get hpa
```
Expected output:
```
NAME                               REFERENCE               TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-prometheus-scaledobject   Deployment/sample-app   32700m/30 (avg)   2         10        10         47m
```
The output indicates that the sample-app application has scaled out to the maximum number of replicas specified in the KEDA configuration.

Stop the producer program and run the consumer program. Then, run the following command to watch the HPA status.

kubectl get hpa -w

Expected output:

NAME                               REFERENCE               TARGETS            MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-prometheus-scaledobject   Deployment/sample-app   222500m/30 (avg)   2         10        10         50m
keda-hpa-prometheus-scaledobject   Deployment/sample-app   232400m/30 (avg)   2         10        10         51m
keda-hpa-prometheus-scaledobject   Deployment/sample-app   0/30 (avg)         2         10        10         52m
keda-hpa-prometheus-scaledobject   Deployment/sample-app   0/30 (avg)         2         10        2          57m

The output shows that after data consumption is complete and a cool-down period passes, the sample-app application scales in to the minimum number of replicas specified in the KEDA configuration.

Container Service for Kubernetes:Event-driven autoscaling (KEDA) based on ApsaraMQ for RocketMQ metrics