How to initiate performance testing of MQTT billion level connections and millions of message throughput

What is the MQTT protocol?

MQTT is a Asynchronous communication message protocol based on TCP/IP protocol stack. It is a lightweight publish and subscribe information transmission protocol. It can be extended in unreliable network environments and is suitable for scenarios where device hardware storage space or network bandwidth is limited. Using the MQTT protocol, message senders and receivers are not limited by time and space.

For IoT protocols, it is necessary to address several key issues related to IoT device communication: complex and unreliable network environment, small memory and flash memory capacity, and limited processor capacity. The MQTT protocol occupies half of the IoT protocol market due to its simple implementation, QoS support, and small packet size.

MQTT's publish subscribe mode

The publish subscribe model is different from the traditional client server model in that it separates the sending client (publisher) from the receiving client (subscriber), and the publisher and subscriber do not need to establish direct contact. We can have multiple publishers publish messages to a single subscriber, or have multiple subscribers simultaneously receive messages from one publisher. The essence of this is that an intermediate role called a broker (also known as MQTT Broker) is responsible for all message routing and distribution work. The traditional client server model can achieve similar effects, but it cannot be as concise and elegant as the publish subscribe model.

The advantage of the publish subscribe model lies in the decoupling between publishers and subscribers, which is manifested in the following two aspects:

Space decoupling: There is no need to establish a direct connection between subscribers and publishers, and new subscribers do not need to modify the publisher's behavior when they want to join the network.

Time decoupling: Subscribers and publishers do not need to be online at the same time, and even if there are no subscribers, it does not affect the publisher's ability to publish messages.

Why do we need to conduct MQTT performance testing

MQTT performance testing mainly helps us achieve the following:

1. Understand the core indicators of MQTT

End-to-end latency of messages in different network environments

• Maximum number of connections simultaneously maintained by MQTT Broker

TPS for MQTT sending and receiving messages

2. Auxiliary MQTT Broker Selection

There are countless MQTT Brokers available in the IoT industry. In addition to classic Mosquito, AWS, Azure, Baidu Cloud, Alibaba Cloud, IBM, and other products that provide IoT MQTT access services, there are also multiple MQTT Brokers available for commercial production.

However, the system performance and application scenarios of each MQTT Broker are different. For example, EMQ has high single machine performance, with single machines supporting millions of levels of concurrency and clusters supporting tens of millions of levels of concurrency. Its advantages lie in its high concurrency connection and high throughput message service capabilities; HiveMQ single machine performance is relatively poor, with a certain degree of high concurrency connections and high throughput message service capabilities.

Through MQTT performance testing, we can more accurately select suitable MQTT brokers.

How does the industry generally conduct MQTT performance testing?

The difficulty of MQTT performance testing lies in being able to initiate and maintain massive connections simultaneously.

The most common MQTT performance testing method currently is implemented based on JMeter's MQTT Xmeter plugin. The underlying principle of this plugin is that each thread simulates an MQTT Client and sends connection requests to the MQTT Broker. However, JMeter's threading model (one concurrency is a thread) determines that in order to initiate a massive number of connection requests, the pressure machine must create an equal number of threads, which not only causes serious resource waste, but also limits the total number of connection requests that a single pressure machine can initiate. Alibaba Cloud's performance testing PTS allows a single thread to initiate and maintain a maximum of 100, a maximum of 50000 per pressure machine, and a maximum of 100 million MQTT connections per pressure test.

Using PTS can help users easily initiate millions or even billions of MQTT connections, saving a lot of time and machine costs.

How to use PTS to initiate MQTT billion level connections and test the throughput of millions of messages

Usage steps

1. Create a scene. We create MQTT pressure testing scenarios in the PTS console's [Pressure Testing Center] ->[Create Scenario];

2. Scenario configuration. PTS supports four test nodes: establishing connections, publishing messages, subscribing to messages, and closing connections. If users only need to test the maximum number of connections supported by the broker, they can only create connection nodes.

For Alibaba Cloud's MQTT instance, users only need to turn on the signature authentication switch, fill in the AccessKey ID, AccessKey Secret, and instance ID, and PTS will dynamically generate a username and password. PTS supports specifying the number of single concurrent connections, as well as three flow models: uniform increment, stepped increment, and fixed pressure. Users can control the growth rate of MQTT connections by adjusting the concurrent number in pressure testing.

Scenario configuration example: 10000 MQTT connections are used as pub clients to send messages, and each pub client sends 1 QoS 0 message per second; 990000 MQTT connections serve as sub clients to receive messages, with each sub client receiving 1 QoS 0 message per second. Therefore, the total message publishing throughput is 10000 per second, and the total message receiving throughput reaches 1 million per second.

Summary

This article elaborates on:

1. What is the MQTT protocol

2. Necessity of MQTT performance testing

3. Advantages of using PTS for MQTT performance testing

4. How to use PTS to quickly initiate MQTT performance testing