Simple Message Queue (formerly MNS):Migrate data from AWS SQS/SNS to Simple Message Queue (SMQ)

Queue-based messaging model

Topic-based messaging model

Queue-based messaging model limits

Topic-based messaging model limits

Key limit differences and workarounds

Pay attention to these key limit differences:

• Maximum message payload (64 KB vs. 256 KB): SMQ defaults to 64 KB, compared to 256 KB for Amazon SQS and Amazon SNS. To request a higher limit, submit a ticket. Alternatively, store large message bodies in OSS and pass the object reference in the message. Transmit oversized messages.

• Maximum subscriptions per topic (100 vs. 12,500,000): SMQ allows 100 subscriptions per topic by default, compared to 12,500,000 for Amazon SNS. To increase this limit, submit a ticket.

• Message retention period (7 days vs. 14 days): SMQ retains messages for up to 7 days, compared to 14 days for Amazon SQS. Ensure your consumers process messages within the 7-day window.

• Data filtering: SMQ uses tag-based filtering for topic subscriptions; Amazon SNS uses JSON policy-based filtering. Redesign your filter logic during migration.

Stage 1: Baseline and resource preparation

Goal: Prepare equivalent SMQ resources.

Actions:

1. Inventory all Amazon SQS queues and Amazon SNS topics. Record their configurations: retention periods, visibility timeouts, delay settings, and subscriptions.

2. Create corresponding queues and topics in SMQ with equivalent configurations. Account for limit differences: message payload size, subscription limits, and retention periods.

3. Synchronize subscription metadata. Convert Amazon SNS JSON filter policies to SMQ tag-based filters where applicable.

4. Set up monitoring for both AWS and SMQ to compare message flow during migration.

Exit criteria: SMQ queues and topics are created and configured. Monitoring is in place for both environments.

Stage 2: Enable dual-read on consumers

Goal: Enable dual-read so that consumers process messages from both services.

Actions:

1. Update consumer applications to read from both Amazon SQS and SMQ queues (dual-read), ensuring they handle SMQ messages before producers switch over.

2. Validate that SMQ messages are consumed correctly. Verify message format compatibility, acknowledgment behavior, and error handling.

3. Monitor consumer lag and error rates on both sides to confirm that SMQ consumption is stable.

Exit criteria: Consumers successfully process messages from SMQ with no errors for a sustained period.

Stage 3: Switch producers to SMQ

Goal: Switch producers to write to SMQ while maintaining dual-read on the consumer side.

Actions:

1. Update producer applications to publish messages to SMQ instead of, or in addition to, Amazon SQS.

2. If running dual-write, monitor both message flows to confirm messages arrive correctly in SMQ.

3. Once producers are fully switched to SMQ, you can stop dual-write. Consumers should continue dual-read to drain any remaining messages from Amazon SQS.

Exit criteria: All producers are successfully publishing to SMQ. Message delivery is confirmed through monitoring.

Stage 4: Decommission AWS resources

Goal: Decommission Amazon SQS/Amazon SNS resources.

Actions:

1. Wait for all remaining messages in Amazon SQS queues to drain. Monitor queue depth until it reaches zero.

2. Remove dual-read logic from consumer applications so they only read from SMQ.

3. Decommission the Amazon SQS queues and Amazon SNS topics in AWS.

4. Update your documentation and runbooks to reflect the new SMQ-based architecture.

Exit criteria: All Amazon SQS/Amazon SNS resources are decommissioned. All producers and consumers operate exclusively through SMQ.