IoT devices - Apsara 2022

Date: Oct 1, 2022

Traditional message queues ((Kafka, RocketMQ, etc.) have been polished for many years, and they have done extremely well in many aspects such as high performance, massive accumulation, and message reliability. However, in IoT scenarios, it is often necessary to face massive message delivery. , the performance of traditional message queue is "powerless" Apsara conference 2022.

Apsara conference 2022 the IoT field, from the application server to the embedded chip, it is necessary to transmit event messages, such as the opening of the cabinet of the shared charging treasure, the light-on command sent from the server to the device, the high-frequency message flow of the industrial gateway, etc. In the process of transmitting these information , the biggest significance of the queue is to turn the entire message event into a smoothly running system under uncontrollable environmental factors, because IoT devices will occasionally cause a large number of message floods due to failure or network jitter Apsara conference 2022.

Apsara conference 2022 as a leader and innovator in the field of Internet of Things, Alibaba Cloud AIoT has continuously cultivated and accumulated in the field of message queues. In order to allow IoT practitioners to better understand IoT scene queues, Alibaba Cloud technical expert Lv Jianwen has compiled a list of IoT queues. Dry goods knowledge, discuss with you a message queue suitable for the Internet of Things system Apsara conference 2022.

1. Differences between IoT queues and ordinary queues Apsara conference 2022

1. Separation of upstream and downstream Apsara conference 2022

In the IoT scenario, we divide the need queue into two scenarios, one is the upstream queue and the other is the downstream queue. After the split, you can isolate the uplink and downlink, and control a device. For example, if the payment is successful, the cabinet must be issued to open the cabinet. If there is any problem in the uplink, it must not affect the downlink business. In addition, the characteristics of the uplink and downlink links are very different. Device uplink messages have a very high concurrency, but in many scenarios, the reliability and latency requirements are low, while device downlink messages have low concurrency, but downlink messages (usually control device commands) require high arrival power Apsara conference 2022.

2. Supports massive device-level topics Apsara conference 2022

Apsara conference 2022 core requirement of traditional queues is that no matter how much they accumulate, their performance will not be affected. When there are more topics in Kafka, the original advantage of writing files in order of messages will cause a broker to degenerate to random writing and lose its advantage. In addition, there are limitations in requiring zookeeper to coordinate so many topics, so these queues themselves provide a plug-in proxy bridge to the outside world. The entrances are multiple device topics, which are then bridged and mapped to a small number of actual kafka topics. This solution is feasible to a certain extent, but it cannot achieve the isolation effect, and it will not cure the symptoms Apsara conference 2022.

Through the comparison between Figure 1 and Figure 2, it is obvious that congestion in one queue minimizes the impact on other devices. What we need is "to isolate a large number of topics from each other as much as possible without affecting the overall performance", and try to ensure that the messages of device A accumulate topics without affecting device B Apsara conference 2022.

3. Real-time message generation and priority sending Apsara conference 2022
Let's take an example first. The queues of a courier locker business are piled up, and then the user next to the locker "at this moment" desperately uses his mobile phone to open the locker and can't open it (the back-end system is restored at this time), the problem That is, there are still hundreds of thousands of messages in the queue, and new messages need to be queued, waiting for the previous messages to be consumed, regardless of whether these messages are still useful. Therefore, messages generated in real-time are sent preferentially, and messages that are piled up enter degraded mode.

Second, a thought - how to solve massive topic problems Apsara conference 2022?

First of all, in the face of "a large number" of problems, we generally consider partitioning, unitization, grouping and other isolation and splitting. Here we discuss how to achieve as many topics as possible in a single instance mode, and any number of topics can be 100. % No problem is certainly not realistic Apsara conference 2022.

Since the broker and the storage have been isolated, the broker has nothing to do with the topic, or any topic data generation, what the broker does is write and distribute.

• Massive topics, each topic has a limited number of subscriptions: Redis cache or local cache is used for the relationship between topics and subscribers. There is a topic tree tree algorithm for matching mqtt topics. Hivemq has an implementation version.

• Mass subscription of a single topic: This scenario is actually multicast and broadcast. We will not consider doing this on the queue itself, but encapsulate the broadcast component in the upper layer to coordinate tasks and batch sending.

3. Alibaba Cloud AIoT message queue

Currently, Alibaba Cloud AIoT queues are also called server-side subscriptions, which means that users use the server-side to subscribe to their device messages. In order to reduce the access cost, users can use the AMQP1.0 protocol to access, which is in line with the open source ecosystem. At the same time, it is compatible with traditional queues and new queues, and it is left to the user to choose according to the scene. Users can choose to use kafka, mq, iot queue, or even a combination mode, such as configuring the flow queue according to the message feature rules.

Alibaba Cloud AIoT's scene queue practice, in addition to the existing mq queue and kafka queue integration, adds its own real-time priority queue implementation, at the same time, joins the queue gateway agent, which allows users to choose ordinary message queues, or Choose a lightweight IoT message queue.