Cloud Storage Gateway (CSG) instances run in on-premises data centers and in Alibaba Cloud, but the CSG console cannot reach them directly over private networks. ApsaraMQ for MQTT bridges this gap by providing secure and compliant penetration between public and private networks, enabling a bidirectional messaging channel between the console and every gateway instance, regardless of where the instance is deployed.
How it works
CSG supports two deployment modes:
| Deployment mode | Description |
|---|---|
| On-premises | CSG runs as a virtual machine in your local data center, connected only to your intranet. |
| Cloud-based | CSG runs inside an Elastic Compute Service (ECS) instance within an Alibaba Cloud virtual network environment. |
In both modes, CSG instances use private IP addresses. The CSG console cannot access these addresses directly. Exposing cloud-deployed gateways through public IP addresses introduces security risks and attack surfaces.
ApsaraMQ for MQTT solves this by acting as a relay between the CSG console and all gateway instances. The solution uses message pushing and active pulling to display statuses and issue commands. Two primary communication patterns are used:
Console-to-gateway (point-to-point downlink): The CSG console publishes commands to specific gateway instances through ApsaraMQ for MQTT.
Gateway-to-console (fan-in uplink): Each CSG instance reports its status back to the console by publishing messages or responding to status queries.
The CSG console also calls backend APIs to manage individual gateway instances. ApsaraMQ for MQTT handles all real-time messaging, while ApsaraMQ for RocketMQ can be integrated for server-side message processing.
Benefits
| Benefit | Details |
|---|---|
| Elastic scalability | Message transmission capability is infinitely scalable. Adding more CSG instances does not degrade performance. |
| Low-latency delivery | Commands reach over a million devices in milliseconds without queuing congestion. |
| End-to-end encryption | TLS/SSL encryption is supported to protect data in transit. |
| High availability | All ApsaraMQ for MQTT service nodes are highly available and stable. |
| Multi-language SDKs | SDKs are available in multiple programming languages for both MQTT brokers and client applications. |
| Flexible networking | Both internal and public network access are supported for different deployment topologies. |
Design the MQTT messaging layer
To build the management channel, design your client IDs, topic structure, and message parameters as described in the following sections.
For SDK downloads and API references, see ApsaraMQ for MQTT SDK and ApsaraMQ for RocketMQ SDK.
Design client IDs
Every MQTT client must have a globally unique client ID. In ApsaraMQ for MQTT, a client ID consists of two parts joined by the @@@ separator:
<Group ID>@@@<Device ID>The total length must not exceed 64 characters.
| Component | Description | Best practice |
|---|---|---|
| Group ID (prefix) | A Group ID registered in the ApsaraMQ for MQTT console. | Create separate Group IDs for each Alibaba Cloud region where the CSG console runs (for example, one for China (Hangzhou) and another for China (Shanghai)). This makes it easier to isolate issues by region. |
| Device ID (suffix) | An application-generated identifier mapped one-to-one to a CSG instance ID. | Map each Device ID directly to a CSG instance ID to guarantee global uniqueness. |
Example client IDs:
GID_csg_hangzhou@@@csg-instance-a1b2c3d4
GID_csg_shanghai@@@csg-instance-e5f6g7h8For more details on client ID structure, see Terms.
Design the topic structure
MQTT uses a publish/subscribe model. Topics follow a hierarchical directory structure with parent topics and subtopics. The total topic length (parent + subtopics) must not exceed 64 characters.
| Topic level | Description | Registration |
|---|---|---|
| Parent topic | The first level in the topic hierarchy. Equivalent to a namespace. | Must be registered in the ApsaraMQ for MQTT console before use. |
| Subtopic | Any level below the parent topic. | No registration required. Define subtopics freely based on your needs. |
For background on the MQTT publish/subscribe model, see the MQTT v3.1.1 specification and What is ApsaraMQ for MQTT?.
Topic design principles
Follow these rules when defining topics for CSG management:
Separate uplink and downlink traffic into different parent topics. This isolates message flows and simplifies troubleshooting.
Uplink: messages from CSG instances to the management service.
Downlink: messages from the management service to CSG instances.
Use different parent topics for messages with different priorities or payload sizes. This prevents large, low-priority transfers from delaying small, time-sensitive control commands.
Example topic hierarchy:
# Downlink: console-to-gateway commands (registered parent topic: csg_cmd)
csg_cmd/hangzhou/csg-instance-a1b2c3d4/restart
csg_cmd/hangzhou/csg-instance-a1b2c3d4/config
# Uplink: gateway-to-console status reports (registered parent topic: csg_status)
csg_status/hangzhou/csg-instance-a1b2c3d4/heartbeat
csg_status/hangzhou/csg-instance-a1b2c3d4/metrics
# Uplink: large file transfers (separate parent topic: csg_data)
csg_data/hangzhou/csg-instance-a1b2c3d4/logsAvoid leading forward slashes in topic names (for example, use csg_cmd/hangzhou instead of /csg_cmd/hangzhou). A leading slash introduces an unnecessary empty level in the topic hierarchy.
Recommended messaging pattern
Use P2P messaging for the interaction process described above. P2P messages do not require the receiver to subscribe to a topic -- the sender specifies the target client directly. This simplifies command-and-control scenarios because the console sends commands to specific gateway instances without managing subscriptions.
For details on topic structure, see Terms.
Configure message parameters
The CSG console uses MQTT to simulate RPC calls. Each command is sent with noreply -- the console does not wait for a return value, and previous messages do not need to be received again. If an error code is received, the console prompts the operator to issue the command again.
Set the following parameter on every MQTT client connection:
| Parameter | Value | Reason |
|---|---|---|
| cleanSession | true | The broker does not queue messages for disconnected clients. Since commands are fire-and-forget, previous messages do not need to be replayed after reconnection. |
For more details on cleanSession behavior, see Terms.
Terms
| Term | Definition |
|---|---|
| MQTT | Message Queuing Telemetry Transport -- an industry-standard IoT and mobile messaging protocol. ApsaraMQ for MQTT supports MQTT by default. |
| CSG | Cloud Storage Gateway -- a hybrid storage service that uses Object Storage Service (OSS) buckets as backend storage. CSG exposes file storage over Network File System (NFS) and Common Internet File System (CIFS), and block storage over Internet Small Computer Systems Interface (iSCSI). For details, see What is Cloud Storage Gateway? |
| MQTT broker | A server-side node in ApsaraMQ for MQTT that receives messages from publishers and forwards them to subscribers. |
| MQTT client | A device or application that connects to an MQTT broker. In this use case, each CSG instance acts as an MQTT client. |
| ApsaraDB RDS | A managed relational database service on Alibaba Cloud. |
| Simple Log Service (SLS) | A log analytics service on Alibaba Cloud, used here for auditing and tracing. |