All Products
Search
Document Center

Function Compute (2.0):Use Function Compute to perform message cleansing

Last Updated:Nov 06, 2023

Common message processing templates, such as the message splitting, dynamic routing, and message enrichment templates, are provided to perform message cleansing. You can directly use a template to process messages or modify the code of the template based on your business requirements. This topic describes the types of templates used to perform message cleansing in ApsaraMQ for RocketMQ and how to use the templates.

Background information

Message cleansing tasks provide basic operator capabilities, and Function Compute provides the underlying logic. After you create a message cleansing task in ApsaraMQ for RocketMQ, you can log on to the Function Compute console to customize the code and modify function configurations.

Operator

Description

Message filtering

Matches the message content based on regular expressions and sends the matched messages to specified destinations. For more information, see Event patterns.

Message conversion

Converts message content based on string match conditions and sends the converted message to specified destinations. For example, character case conversion can be performed. For more information, see Event transformation.

Content segmentation

Splits message content based on regular expressions and sends the split messages to specified destinations.

Dynamic routes

Matches the message content based on regular expressions and sends the matched messages to specified destinations and the unmatched messages to default destinations.

Content enrichment

Enriches message content based on enrichment sources. If the original content of a message contains AccountID, AccountID is used to query the database and obtain the customer region. The system inserts the customer region into the source message body and sends the message body to specified destination services.

Content mapping

Maps message content based on regular expressions. For example, the system can mask sensitive fields in a message or reduce the message size to the minimum size.

Content splitting

Examples

The following message contains a list of students.

message:
[Jack, Male, 17, Class 4; Alice, Female, 17, Class 3; John, Male, 17, Class 4]

You need to split the message into the following messages, each of which contains information about a single student, before you send the messages to specified destination services. Example:

message:
    [Jack, Male, 17, Class 4]
message:
    [Alice, Female, 17, Class 3]
message:
    [John, Male, Class 17, Class 4]
dataclean_split

Procedure

  1. Log on to the ApsaraMQ for RocketMQ console.

  2. In the left-side navigation pane, choose Message Integration > Message Outflow. In the top navigation bar, select a region.

  3. On the Message Outflow page, click Create Task.

  4. In the Create Message Outflow Task panel, configure the following parameters and click Confirm.

    Configure the following parameters as described. Retain the default values for other parameters.

    • Basic Information

      Parameter

      Description

      Task Name.

      Enter a name for the task.

      Message Outflow Task Type

      In this example, select ApsaraMQ for RocketMQ. The following message services are supported: ApsaraMQ for RocketMQ, ApsaraMQ for RabbitMQ, Message Service, and ApsaraMQ for Kafka.

    • Resource Configuration

      Parameter

      Description

      Source

      Region

      In this example, China (Hangzhou) is selected.

      Version

      Select the version of ApsaraMQ for RocketMQ. In this example, RocketMQ 5.x is selected.

      Instance

      The ApsaraMQ for RocketMQ instance that produces messages.

      Topic

      Select the topic of the source instance.

      Target

      Version

      Select the version of ApsaraMQ for RocketMQ that receives messages. In this example, RocketMQ 5.x is selected.

      Instance ID

      The ID of the ApsaraMQ for RocketMQ instance that receives messages.

      Topic

      The topic that you want to use to receive the messages.

    • Data Processing

      • Message Filtering: Select None.

      • Message Conversion: Select Custom Configuration. Set Message Body to Data Cleansing and select Create Function Template. Set Function Template to Content Splitting transform_split and modify the function code based on your business requirements.

    After the creation, you can log on to the Function Compute console to view the created service and function,

Dynamic routing

Examples

The following message contains a list of toothpastes.

message:
[BrandA, toothpaste, $12.98, 100g
 BrandB, toothpaste, $7.99, 80g
 BrandC, toothpaste, $1.99, 100g]

The list needs to be sent to specified destination topics based on the custom dynamic rules. The following items describe the rules:

  • Send the messages that start with BrandA to the BrandA-item-topic and BrandA-discount-topic topics.

  • Send the messages that start with BrandB to the BrandB-item-topic and BrandB-discount-topic topics.

  • Send other messages to the Unknown-brand-topic topic.

The following sample code shows the JSON format of the rules:

{
  "defaultTopic": "Unknown-brand-topic",
  "rules": [
    {
      "regex": "^BrandA",
      "targetTopics": [
        "BrandA-item-topic",
        "BrandA-discount-topic"
      ]
    },
    {
      "regex": "^BrandB",
      "targetTopics": [
        "BrandB-item-topic",
        "BrandB-discount-topic"
      ]
    }
  ]
}
dataclean_dynamicroute

Procedure

For the detailed procedure, see Content splitting. Set Function Template to Dynamic Routing dynamic_routing.

Content enrichment

Examples

In this example, IP address segments are enriched. The following code snippet shows the access logs of a service:

{
  "accountID": "164901546557****",
  "hostIP": "192.168.XX.XX"
}

The source of the IP address needs to be queried, and the mapping relationship needs to be stored in a MySQL database.

CREATE TABLE `tb_ip` (
    ->      `IP` VARCHAR(256) NOT NULL,
    ->     `Region` VARCHAR(256) NOT NULL,
    ->      `ISP` VARCHAR(256) NOT NULL,
    ->      PRIMARY KEY (`IP`)
    -> );

The following code snippet shows the processed message.

{
  "accountID": "164901546557****",
  "hostIP": "192.168.XX.XX",
  "region": "beijing"
}
dataclean_enrich

Procedure

For the detailed procedure, see Content splitting. Set Function Template to Content Enrichment transform_enrichment.

Content mapping

Examples

The following message contains the registration information about the employees of a company. The information includes confidential content, such as employee IDs and mobile phone numbers.

Jack, Employee ID 1, 131 1111 1111
Alice, Employee ID 2, 132 2222 2222
John, Employee ID 3, 133 3333 3333 

The confidential content in the preceding message must be masked before the message is sent to the destination services. Example:

Ja**, Employee ID *, ***********
Alic*, Employee ID *, ***********
Joh*, Employee ID *, *********** 
dataclean_projection

Procedure

For the detailed procedure, see Content splitting. Set Function Template to Content Mapping transform_projection.