The Apache RocketMQ community revealed the product capabilities of Message Queue for Apache RocketMQ 5.0 at the RocketMQ Summit 2022 in July. The community also explained the technical positioning and development direction of Message Queue for Apache RocketMQ 5.0 for developers.
With rich experience in cloud computing, we have witnessed the continuous development of Message Queue for Apache RocketMQ over the past seven years. Now, Message Queue for Apache RocketMQ 5.0, the latest version of Message Queue for Apache RocketMQ, has officially been launched.
According to the community, as cloud-native and cloud adoption have become inevitable trends, Apache RocketMQ has updated its architecture and adapted some of its capabilities to meet the requirements of business developers. How can Message Queue for Apache RocketMQ 5.0 be implemented in production? This article focuses on the cloud-native transformation of messaging architectures and product capabilities. It also describes how Alibaba Cloud has adapted its products and technologies around Message Queue for Apache RocketMQ 5.0 and how they are upgrading products to better suit the technical and capability demands of the enterprise customers.
First, let's take a look at how cloud-native messaging services will evolve.
As we venture into the future, cloud-native messaging products must make breakthroughs in the following aspects:
In the following sections, let me briefly introduce how Message Queue for Apache RocketMQ 5.0 implements the preceding four aspects in practice.
Messaging services are typically integrated in core systems and widely used in scenarios that feature large traffic fluctuations (such as transactions and payments). We usually see huge amounts of business traffic during special occasions (such as big sales promotions and flash sales).
In such cases, Message Queue for Apache RocketMQ 5.0 is an ideal choice. It can automatically adapt to business requirements by scaling resources. On the one hand, Message Queue for Apache RocketMQ 5.0 reserves resources for enterprises to deal with their basic business needs based on the cost optimization policy. On the other hand, it supports auto scaling to ensure business stability when traffic spikes occasionally. In this case, you are charged for additional resources that you use. This allows you to reserve minimal resources to ensure reliable services while ensuring business continuity during unexpected traffic spikes.
In addition to scalability, the messaging system is a stateful system that stores a large amount of valuable business data. When the number of API calls fluctuates, the storage also needs to scale up or down. In this case, you have to prevent data loss while saving storage costs. The scaling of conventional disk-based architectures can be problematic. For one, the capacity of local disks is limited. You can only add nodes to local disks if you want to scale up. This results in a waste of computing resources. For two, local disks cannot be dynamically scaled down. You can only reduce storage costs by isolating the business traffic. However, this operation is very complicated.
By comparison, Message Queue for Apache RocketMQ 5.0 stores messages in Serverless mode. You are charged for storage space based on actual usage. You only need to set a proper value for time-to-live (TTL) to ensure data integrity for long-term storage.
Ease of use is a requirement for system design. When we develop messaging services, we must optimize the API design, development process, configurations, and O&M to reduce the burden on users and help them avoid mistakes. For example, when consumers consume messages in Message Queue for Apache RocketMQ 4.X or Message Queue for Apache Kafka, they are often troubled by the load balancing policy. They have to pay attention to the number of queues and the number of consumers in the current topic because consumers implement load balancing and allocate tasks based on queues. If the capabilities for processing messages are different among consumers or the number of messages assigned to each consumer is different, some consumers will have long queues.
However, in a typical business integration scenario, the consumer client only needs to consume messages in a stateless messaging model, and consumers only need to focus on whether the messages are processed instead of the storage model and policy.
Based on this requirement, Message Queue for Apache RocketMQ 5.0 provides a new consumer model called simple consumers. This model supports the consumption, retry, and commitment of a single message.
Personnel with experience in message queue O&M know that the messaging system decouples upstream production and downstream consumption. As a result, when a fault occurs, it is difficult to tell whether the fault is due to the messaging service or business logic.
The observability feature of Message Queue for Apache RocketMQ 5.0 provides solutions to the preceding problem using events, traces, and metrics. The following describes the content of the events, traces, and metrics in Message Queue for Apache RocketMQ 5.0:
Message Queue for Apache RocketMQ has achieved a lot in terms of observability. It is the first of its kind to support comprehensive query of message traces. The latest version (5.0) supports reporting traces and metrics in the client and broker to a third party for storage using the standard OpenTelemetry protocol. Message Queue for Apache RocketMQ 5.0 can display and analyze data in a standard way with help from open-source technology (like Prometheus and Grafana).
Stability is a key capability of the production system, and messaging services are typically integrated in core transaction systems. Therefore, the stability of the messaging system directly impacts the stability of a business. We have to optimize O&M management and the design of the system architecture to ensure stability. We must quantize the boundaries and metrics of services. Only when we specify the metrics that are available and reliable for services can we design a troubleshooting mechanism and redundancy security mechanism.
Conventional O&M-based stability solutions are only applicable for basic scaling and metric monitoring. The conventional O&M-based stability guarantee cannot provide good quantitative services for various scenarios with complicated boundaries in messaging services (such as message accumulation, cold data reading, and broadcasting). Once these scenarios are triggered, the system fails.
By comparison, Message Queue for Apache RocketMQ 5.0 provides systematic stability through system design. This ensures quantitative services for scenarios (such as message accumulation and cold data reading) and helps determine message sending RT, end-to-end delay, and messaging throughput (TPS). Once these scenarios are triggered, Message Queue for Apache RocketMQ 5.0 can protect the system from failures in certain cases.
This article introduces the evolution of Message Queue for Apache RocketMQ 5.0 from the perspectives of large scalability, ease of use, observability, and SLA-based stability. This article also talks about how Message Queue for Apache RocketMQ 5.0 implements these four aspects in practice.
Currently, Message Queue for Apache RocketMQ 5.0 is available for commercial use. We have enhanced its features, scalability, ease of use, and O&M capabilities. Meanwhile, the prices of instances are up to 50% lower than those of the previous generation. We aim to help enterprises lower their costs and improve their efficiency in business development and integration. The Message Queue for Apache RocketMQ 5.0 instances support free scaling at a scale of 0 to 1 million TPS. They also support burst scaling and Serverless storage. In terms of observability, the Message Queue for Apache RocketMQ 5.0 instances support the integration of traces and metrics. In terms of ease of use, the Message Queue for Apache RocketMQ 5.0 instances support the next-gen lightweight native SDKs for multiple languages, making them more stable and easier to use.
Cloud-Native Observability Suite: Building Ubiquitous Observability Infrastructure
376 posts | 41 followersFollow
Alibaba Cloud Native Community - January 5, 2023
Alibaba Cloud Native Community - March 20, 2023
Alibaba Cloud Native Community - December 16, 2022
Alibaba Cloud Native Community - November 23, 2022
Alibaba Cloud Community - December 21, 2021
Alibaba Cloud Native Community - December 16, 2022
376 posts | 41 followersFollow
AlibabaMQ for Apache RocketMQ is a distributed message queue service that supports reliable message-based asynchronous communication among microservices, distributed systems, and serverless applications.Learn More
Alibaba Cloud Function Compute is a fully-managed event-driven compute service. It allows you to focus on writing and uploading code without the need to manage infrastructure such as servers.Learn More
A message service designed for IoT and mobile Internet (MI).Learn More
High Performance Computing (HPC) and AI technology helps scientific research institutions to perform viral gene sequencing, conduct new drug research and development, and shorten the research and development cycle.Learn More
More Posts by Alibaba Cloud Native Community