Reduce Consumer Load with RocketMQ Broker-Side Message Filtering - ApsaraMQ for RocketMQ

By default, ApsaraMQ for RocketMQ delivers every message in a topic to all subscribers. To receive only a subset, assign a filter expression to the consumer's subscription. The broker evaluates the expression server-side and delivers only matching messages, which reduces network traffic and consumer processing overhead.

Use message filtering for fine-grained categorization within a single business domain. For separate business domains, use different topics instead.

How it works

Producer attaches a tag or custom attributes to each message.
Consumer registers a filter expression with the broker when subscribing to a topic.
Broker evaluates each incoming message against the filter expression and delivers only matches.

Choose a filtering method

ApsaraMQ for RocketMQ provides two filtering methods: tag-based filtering and attribute-based SQL filtering.

	Tag-based filtering	Attribute-based SQL filtering
Filter target	Message tags	Message attributes (custom and system attributes). Tags are a system attribute.
Match type	Exact string match	SQL-92 expression match
Best for	Simple routing with a small number of categories	Complex filtering logic across multiple attributes

Because tags are a system attribute, tag-based filtering is a subset of SQL filtering. In SQL expressions, reference a message's tag through the TAGS keyword.

Tag-based filtering

Tag-based filtering performs exact string matching on the tag attached to each message.

Tag rules

Each message can have only one tag.
A tag is a case-sensitive string. For example, TagA and tagA are different tags.
Keep tags under 128 characters.

Filter expression syntax

Expression	Behavior	Example
Single tag	Delivers messages with that exact tag	`"TagA"`
Multiple tags	Delivers messages matching any listed tag. Separate tags with two vertical bars (`\|\|`).	`"TagA\|\|TagB\|\|TagC"`
Wildcard	Delivers all messages in the topic	`"*"`

Use case: e-commerce transaction routing

An e-commerce platform sends order, payment, and logistics messages to a single topic Trade_Topic. Different downstream systems subscribe with different tag filters:

System	Filter expression	Messages received
Payment system	`"Payment"`	Payment messages only
Logistics system	`"Logistics"`	Logistics messages only
Transaction analysis	`"Order\|\|Payment"`	Order and payment messages
Real-time computing	`"*"`	All messages

Sample code

Set a tag on the producer side:

Message message = messageBuilder.setTopic("topic")
                .setKeys("messageKey")
                .setTag("TagA")
                .setBody("messageBody".getBytes())
                .build();

Subscribe to messages with a single tag:

String topic = "Your Topic";
FilterExpression filterExpression = new FilterExpression("TagA", FilterExpressionType.TAG);
pushConsumer.subscribe(topic, filterExpression);

Subscribe to messages matching any of multiple tags:

String topic = "Your Topic";
FilterExpression filterExpression = new FilterExpression("TagA||TagB||TagC", FilterExpressionType.TAG);
pushConsumer.subscribe(topic, filterExpression);

Subscribe to all messages in the topic:

String topic = "Your Topic";
FilterExpression filterExpression = new FilterExpression("*", FilterExpressionType.TAG);
pushConsumer.subscribe(topic, filterExpression);

Attribute-based SQL filtering

Attribute-based SQL filtering evaluates SQL-92 expressions against message attributes. Producers attach custom key-value attributes to each message, and consumers write SQL expressions to select the messages they need.

Attribute rules

Each message can have multiple custom attributes. Each attribute is a key-value pair.
Attribute keys must start with a letter or underscore (_) and can contain only letters, digits, and underscores.

SQL-92 syntax reference

Write filter expressions using the following SQL-92 operators:

Operator	Description	Example
`IS NULL`	Attribute does not exist	`a IS NULL`
`IS NOT NULL`	Attribute exists	`a IS NOT NULL`
`>`, `>=`, `<`, `<=`	Numeric comparison. Cannot compare strings.	`a IS NOT NULL AND a > 100`
`BETWEEN x AND y`	Numeric range (inclusive). Equivalent to `>= x AND <= y`.	`a IS NOT NULL AND (a BETWEEN 10 AND 100)`
`NOT BETWEEN x AND y`	Outside numeric range. Equivalent to `< x OR > y`.	`a IS NOT NULL AND (a NOT BETWEEN 10 AND 100)`
`IN (x, y)`	Value is in a set. Set elements must be strings.	`a IS NOT NULL AND (a IN ('abc', 'def'))`
`=`, `<>`	Equal / not equal. Works for both numbers and strings.	`a IS NOT NULL AND (a = 'abc' OR a <> 'def')`
`AND`, `OR`	Logical operators. Enclose each condition in parentheses.	`a IS NOT NULL AND (a > 100) OR (b IS NULL)`

Numeric value constraints:

Numeric values are 32-bit integers, ranging from -2,147,483,648 to 2,147,483,647.
Strings that can be converted to numbers are treated as numeric values.

How the broker handles edge cases

If a filter expression cannot be evaluated cleanly, the broker drops the message instead of delivering it. This applies to three situations:

Situation	Example	Broker behavior
Evaluation exception	Comparing a numeric value with a non-numeric string: `a > 'abc'`	Message filtered out
Null or non-Boolean result	Consumer filters on an attribute the producer did not set	Message filtered out
Type mismatch	Producer sets a floating-point attribute value, but the filter expression uses integer comparison	Message filtered out

Note

The broker silently drops messages that trigger these edge cases. To avoid unexpected message loss, make sure that producers consistently set all attributes that consumers reference in filter expressions and that attribute types match the comparison operators used.

Use case: filtering logistics messages by region

An e-commerce platform sends order and logistics messages to Trade_Topic. Logistics messages carry a Region attribute with values such as Hangzhou or Shanghai.

System	Filter expression	Messages received
Logistics system 1	`"TAGS = 'Logistics' AND Region IS NOT NULL AND Region = 'Hangzhou'"`	Logistics messages for Hangzhou only
Logistics system 2	`"TAGS = 'Logistics'"`	All logistics messages
Order tracking	`"TAGS = 'Order'"`	Order messages only
Real-time computing	`"True"`	All messages

Sample code

Set a tag and custom attributes on the producer side:

Message message = messageBuilder.setTopic("topic")
                .setKeys("messageKey")
                .setTag("messageTag")
                // Custom attribute: region
                .addProperty("Region", "Hangzhou")
                .setBody("messageBody".getBytes())
                .build();

Subscribe to messages matching a single attribute:

String topic = "topic";
FilterExpression filterExpression = new FilterExpression(
    "Region IS NOT NULL AND Region='Hangzhou'", FilterExpressionType.SQL92);
simpleConsumer.subscribe(topic, filterExpression);

Subscribe to messages matching multiple attributes:

String topic = "topic";
FilterExpression filterExpression = new FilterExpression(
    "Region IS NOT NULL AND price IS NOT NULL AND Region = 'Hangzhou' AND price > 30",
    FilterExpressionType.SQL92);
simpleConsumer.subscribe(topic, filterExpression);

Subscribe to all messages in the topic:

String topic = "topic";
FilterExpression filterExpression = new FilterExpression("True", FilterExpressionType.SQL92);
simpleConsumer.subscribe(topic, filterExpression);

Subscription consistency

All consumers in the same consumer group must use identical filter expressions for a given topic. Mismatched expressions cause subscription inconsistency, which leads to missed messages.

For details, see Subscriptions.

Design topics and tags

Dimension	Guideline
Message type	Route different message types (ordered messages, normal messages) to separate topics. Do not use tags to separate message types.
Business domain	Use separate topics for different business domains. For example, use one topic for payment messages and another for logistics messages.
Volume and priority	Separate high-volume or high-priority messages into their own topics to avoid contention.
Within a domain	Use tags to subdivide messages within the same business domain. For example, tag logistics messages as `Ordinary` or `Urgent`.

Accumulated message calculations

When filtering is active, not all messages in a topic are delivered. This affects how accumulated (backlog) messages are counted:

SQL filtering: Accumulated messages = Ready messages + Inflight messages - Messages that do not match the filter
Tag-based filtering: Accumulated messages = (Ready messages + Inflight messages) x Tag match ratio

Tag match ratio = Messages matching the tag in the sample / Total sampled messages.

FAQ

Why do consumers in the same group lose messages when subscribing to different tags?

All consumers in a consumer group must use the same filter expression. If consumer A subscribes to TagA while consumer B subscribes to TagB, subscription inconsistency occurs and messages get lost. Either align the filter expressions across all consumers in the group, or split them into separate consumer groups.

How is the consumed message count calculated when filtering is active?

Only messages that pass the filter expression count toward consumption. Filtered-out messages are excluded from the count.

Why do messages accumulate even when consumers show zero consumption?

Messages that do not match the active filter still exist in the topic and count toward the backlog. This is expected. See Accumulated message calculations for the exact formulas.

What's next

For complete SDK code examples covering message filtering, see SDK reference overview.