By Wei Li, Apache RocketMQ Committer, RocketMQ Python Client Project Owner, Apache Doris Contributor, and TencentDB Development Engineer
Recently, it occurred to me that lots of software and hardware are not what they are by design. There is a root reason for their designs: to meet the demand at that time and in that scene. Once I realize that, I feel that I'm talking to the designers. I understand their ideas and learn their methods, and I resonate with them: live and learn.
Bearing the preceding questions in mind, let's review the distribution of commit logs.
We can find the following commit log files under the root directory in which the configuration files of the broker are stored.
Real distribution of data files of the broker
It can be seen that there are multiple real stored files, each of which is named after a string resembling a number and 1 GB in size.
It can be learned from the source code that the actual abstract model is as follows:
Abstract model of commit log file distribution
As shown in the preceding figure:
• Commit log is the name of a type of file. There are multiple commit log files, each of which is called a commit log file.
As shown in the preceding figure, there are a total of T commit log files. These files are stored by creation time.
• Each commit log file stores messages, and messages are stored in the order in which they are written. Messages are always written in the latest file that is created and in one thread at a time.
As shown in the preceding figure, the number 1 indicates the first message in the first file. Such is the case for numbers 2, 3, 4, and others. It can be seen that the 1234th message is the last in the first commit log file, and the 1235th message is the first in the second commit log file.
Note 1: The actual storage space occupied by all messages in each commit log file is at most 1 GB. Think about the reason for it.
Note 2: Each time you write a commit log, RocketMQ locks the log. See https://github.com/apache/rocketmq/blob/7676cd9366a3297925deabcf27bb590e34648645/store/src/main/java/org/apache/rocketmq/store/CommitLog.java#L676-L722 for the code snippets.
Append a lock
In the commit log file, many messages are stored according to the established protocol. What is the specific protocol and how do you know that?
Let’s take a look at the following source code: https://github.com/apache/rocketmq/blob/rocketmq-all-4.9.3/store/src/main/java/org/apache/rocketmq/store/CommitLog.java#L1547-L1587
Commit log storage protocol
The protocol fields are put together in the following figure.
Commit log storage protocol from what I understand
Note: The message protocol number in the preceding figure is different from that in the code. The latter only indicates the order of the protocol, while the storage protocol in physical files will be more detailed.
Here are a few questions that need to be clarified.
• The binary protocol uses byte orders, which are often referred to as big-endian and little-endian orders. These orders will not be described in detail here. If you are interested, google it or ask ChatGPT about it. They will provide more detailed answers.
• In Java, each data type occupies a certain number of bytes. A byte occupies 1 byte, an int 4, a short 2, and a long 8.
• The host does not directly convert the IP:Port value as a string to a byte array. Instead, it encodes each number as a byte. This will be explained in the following Golang code section.
• In the encoding of extended information, two invisible characters are used as segmentation. Therefore, the extended fields as key-value pairs cannot contain these invisible characters. What are these two characters?
You have learned about the protocol, but how to prove that your physical files are written according to this protocol?
RocketMQ is written in Java. According to the storage protocol described in the preceding section, I built a tool in Golang to decode commit logs and consumer queues.
Code repository: https://github.com/rmq-plus-plus/rocketmq-decoder
Currently, this tool supports the following two features.
• Specify the commit log offset, directly parse the messages in the commit logs, and print the results.
• Specify the consumer offset, parse the consumer queues to obtain the commit log offset, parse the commit log based on the commit log offset, and print the results.
In Golang, no code relies on RocketMQ to decode. Instead, it relies purely on the protocol.
Import in Golang
Here is an example of parsing a commit log offset in Golang. In Java, a commit log offset is a long data type and occupies 8 bytes.
In Golang, a commit log offset is obtained by reading the data of 8 bytes in length and decoding it to int 64 in big-endian order.
Golang-demo
Here is the result of running the demo.
Read consumer-queue-commit-log
The following is my understanding of the preceding questions.
The consumer queue offset is continuous.
The Consumer Queue Offset refers to the subscript index in each queue, which is continuous. Consumers also take advantage of this continuity to avoid consumer offset holes.
Each index occupies 20 bytes. The following figure illustrates the structure of the index.
Index Structure of consumer-queue
The physical offset in the figure refers to the Commit Log Offset.
The Commit Log Offset is not continuous.
The Commit Log Offset refers to the byte offset of each message in all commit log files. The size of each message is uncertain, so the Commit Log Offset, or the byte offset, varies.
Additionally, it's worth noting that the absolute value of the difference between every two offsets is the total bytes of the previous message.
Moreover, the previous figure "Abstract model of commit log file distribution" contains a misunderstanding. The size of each small square is different.
Bytes in Java files are stored in big-endian order. Byte order includes the data storage order and the network transmission order. By default, Java adopts the big-endian format, which aligns with the order in which data is transmitted in a network, making encoding and decoding more convenient.
The first byte of the data message of each network transmission layer indicates the protocol used to transmit the following data. This allows the data receiver to first analyze the protocol according to the byte order when receiving the data, and then decode the subsequent byte sequence based on the protocol, aligning with the way humans think about and solve problems.
The above is my understanding. Please feel free to leave comments if you have any questions.
Note: There may be differences between different versions of RocketMQ. This article only discusses RocketMQ 4.9.3.
How to Configure TLS-encrypted Transmission in RocketMQ 5.0?
Koordinator: Supporting Hybrid Deployment of Kubernetes and YARN
495 posts | 48 followers
FollowAlibaba Cloud Native Community - October 26, 2023
Alibaba Cloud Native - June 12, 2024
Alibaba Cloud Native Community - February 7, 2023
Alibaba Cloud Native Community - February 1, 2024
Alibaba Cloud Native Community - March 20, 2023
Alibaba Cloud Native - June 11, 2024
495 posts | 48 followers
FollowApsaraMQ for RocketMQ is a distributed message queue service that supports reliable message-based asynchronous communication among microservices, distributed systems, and serverless applications.
Learn MorePlan and optimize your storage budget with flexible storage services
Learn MoreA cost-effective, efficient and easy-to-manage hybrid cloud storage solution.
Learn MoreProvides scalable, distributed, and high-performance block storage and object storage services in a software-defined manner.
Learn MoreMore Posts by Alibaba Cloud Native Community