A message can still be queried on the ApsaraMQ for Kafka broker after its retention period expires. This happens because ApsaraMQ for Kafka deletes messages at the segment level, not at the individual message level.
How segment-based deletion works
ApsaraMQ for Kafka stores messages in segments to prevent storage fragmentation. Each segment holds up to 1 GB of data by default. When the system runs a deletion cycle, it deletes entire segments rather than individual messages.
A segment goes through the following lifecycle:
Open: The segment accepts new messages. It stays open until it reaches the 1 GB size limit.
Closed: The segment has reached 1 GB and no longer accepts messages.
Eligible for deletion: A closed segment whose messages have exceeded the configured Message Retention Period becomes eligible for deletion.
Because ApsaraMQ for Kafka can only delete closed segments, any expired messages inside an open segment are retained until the segment closes and is subsequently deleted.
Causes
An expired message is retained when either of the following conditions is true:
The segment has not reached 1 GB. The segment containing the expired message is still open and accepting new messages. Low-throughput topics produce segments that take longer to reach the 1 GB limit, which delays deletion.
Disk usage is below 75%. When disk usage stays below 75%, ApsaraMQ for Kafka deletes expired messages on a fixed schedule: once per day at 04:00. Expired messages persist until the next scheduled deletion runs.
Disk-based deletion thresholds
ApsaraMQ for Kafka uses two disk usage thresholds to manage message retention:
| Disk usage | Behavior |
|---|---|
| Below 75% | Expired messages are deleted once per day at 04:00. |
| 85% or above | Messages are deleted immediately, starting from the oldest, regardless of the Message Retention Period setting. |
When disk usage reaches 85%, the system prioritizes service availability over retention settings. Messages are deleted from the earliest stored ones even if they have not reached the configured Message Retention Period.
Message Retention Period
The Message Retention Period parameter specifies the maximum duration that messages are retained when disk capacity is sufficient.
| Property | Value |
|---|---|
| Valid range | 24 to 480 hours |
| Default | 72 hours |
This setting defines an upper bound, not a guarantee. Actual retention depends on segment status and disk usage as described in the previous sections.
Troubleshooting
If expired messages persist longer than expected, check the following:
Verify segment utilization. Low-throughput topics produce segments that take longer to reach the 1 GB limit, which delays deletion. To determine whether this is the cause, check the data volume of the topic. If the topic receives a low volume of messages, the active segment may remain open for an extended period. Increasing the message volume or waiting for the segment to fill and close naturally resolves this issue.
Check disk usage. If disk usage is below 75%, expired messages are only deleted at the next scheduled run (04:00 daily). Monitor disk usage through the ApsaraMQ for Kafka console to confirm.
Review the retention period setting. Confirm that the Message Retention Period is configured to the intended value for the topic in the ApsaraMQ for Kafka console. The valid range is 24 to 480 hours, with a default of 72 hours.