This article summarizes five distributed transaction solutions and analyzes their features: XA Specification, TCC, Saga, Local-Message-Based Distributed Transactions, and Transactional-Message-Based Distributed Transactions. Considering its length, I recommend that you add it to your favorites and read it at your convenience.
A transaction is a group of integral operations. These operations are either executed successfully as a whole or canceled as a whole. The most typical transaction scenario is a transfer between bank accounts. If account A wants to transfer RMB 100 to account B, RMB 100 will be deduced from A and added to B. In this scenario, a successful transfer can be considered only when amount changes are made to both accounts successfully. More strictly, a transaction is defined to have the ACID features, which refer to atomicity, consistency, isolation, and durability.
A single database does not involve network interaction, so it is relatively easy to handle transactions between multiple tables. Such transactions are called local transactions.
However, when the performance of a single database reaches the bottleneck, database sharding (into physical instances) is required, which results in cross-database (cross-database-instance) transactions. As the enterprise application expands, the enterprise will undergo further service-oriented transformation to meet the business growth. As the microservice architecture becomes increasingly popular, we will be looking at more and more cross-service transactions.
These are all requirements on distributed transactions. A distributed transaction means that the initiator, participant, resource server, and transaction manager of a transaction are located on different nodes of a distributed system.
In general, there are three scenarios for distributed transactions:
Participants in a distributed transaction system are distributed over an asynchronous network. They achieve consistency through network communication, during which failures and timeouts are inevitable. Therefore, distributed transactions are more difficult to implement than local transactions. Here are some common distributed transaction solutions:
The earliest distributed transaction product may be Tuxedo (Transactions for UNIX, Extended for Distributed Operations) launched by AT&T in the 1980s. Tuxedo was first developed as a distributed transaction middleware for OLTP systems in the telecommunications field. Later, the standardization organization X/Open adopted the design and some interfaces of Tuxedo, and introduced the distributed transaction specification, XA specification.
The XA specification defines a distributed transaction processing model that includes four core roles:
The following figure shows the transaction model defined in the XA specification. The TM instance that initiates the distributed transactions is called the root node, and other TM instances are collectively called transaction participants. The initiator starts a global transaction, and transaction participants run their own transaction branches. If a TM instance initiates a service call to other TM instances, the initiator is the Superior, while the instance being called is the subordinate node.
In the XA specification, a distributed transaction is built on an RM's local transactions, and is then considered a branch transaction. The TM is responsible for coordinating these branch transactions and ensuring that either all of them are committed successfully or all of them are rolled back. The XA specification divides the distributed transaction processing process into the following two phases, hence, it is also called a two-phase commit protocol:
1) Preparation phase
The TM logs transaction startup and queries each RM about whether it is ready to perform preparatory operations.
After an RM receive the order, it evaluates its own state and tries to perform preparatory operations for the local transaction, such as reserving resources, locking resources, and performing operations. Then, the RM waits for subsequent orders from the TM without committing the transaction. If the preceding attempt fails, the RM informs the TM that the execution of this phase fails and rolls back the performed operations. Then, the RM no longer participates in this transaction. For example, MySQL will lock resources and write the redo and undo logs at this phase.
The TM collects the RM's responses and logs the transaction preparation as completed.
2) Commit or rollback phase
This phase initiates the transaction commit or rollback operation based on the coordination result of the previous phase.
If all RMs responded success in the previous step, then the following actions take place:
If any RM responds with an execution failure or does not respond timely in the previous step, the TM will regard the transaction as failed. Then, the following actions take place:
The XA specification also defines the following optimization measures:
The XA specification defines interaction interfaces between core components in detail. Take the interaction interfaces between the TM and an RM as an example. The following figure shows a complete global transaction, in which the interaction between the TM and the RM is quite frequent.
Errors and network timeouts may occur during transaction execution. For these exceptions, different implementations may have different exception handling methods, as described below.
The XA two-phase commit protocol is designed to implement the atomicity, consistency, isolation, and durability of a transaction like local transactions.
The XA specification is the earliest distributed transaction specification. Mainstream database products, such as Oracle, MySQL, and SQL Server, support the XA specification. Note that the JTA specification in J2EE is also based on the XA specification, and therefore is compatible with the XA specification.
XA is a distributed transaction model implemented at the resource management layer. It features a low degree of intrusion into businesses.
XA's two-phase commit protocol can cover the three scenarios of distributed transactions. However, an RM keeps locking resources during the execution of a global transaction. If the transaction involves too many RMs, especially in cross-service scenarios, the number and time consumption of network communications will rapidly increase. Consequently, the blocking time will be prolonged, the throughput of the system declines, and the probability of transaction deadlocks increases. Therefore, the two-phase commit protocol is not suitable for the cross-service distributed transaction mode in microservice scenarios.
Each TM realm creates a single point, which may incur a single point of failure. If the TM crashes after phase one, the participating RMs will not receive the order for phase two and therefore hold the resource locks for a long time. This, as a result, affects the throughput of the business. On the other hand, in a complete global transaction, the TM interacts with RMs eight times, which causes complexity and performance decline.
In addition, the two-phase protocol may cause split-brain exceptions. If the TM goes down after instructing RMs to commit the transaction in phase two, and only some of the RMs receive the commit order, then, when the TM restores, it cannot coordinate all the RMs in maintaining the local transaction consistency.
XA has to deal with many exception scenarios, which are challenging to the framework implementation. Regarding open-source implementations, you can refer to Atomikos and Bitronix.
To solve problems in the two-phase commit protocol, an improved three-phase commit solution was proposed. This new solution removes the single point of failure (SPOF) and adds a timeout mechanism for RMs to avoid long-term locking of resources. However, the three-phase solution cannot solve the split-brain problem and is rarely applied to practical cases. If you are interested in this solution, you can read relevant material on it.
Try, Commit, and Cancel (TCC) is a model of compensating transactions. This model requires each service of an application to provide three interfaces, that is, the try, commit, and cancel interfaces. The core idea of this model is to release the locking of resources at the quickest possible time by reserving the resources (providing intermediate states). If the transaction can be committed, the reserved resources are confirmed. If the transaction needs to be rolled back, the reserved resources are released.
TCC is also a two-phase commit protocol, and can be considered to be a variant of the two-phase-commit XA, which does not hold resource locks for a long time.
The TCC model divides the commitment of transactions into two phases:
1) Phase one
In phase one, TCC performs business check (for consistency) and reserves business resources (for quasi-isolation). This defines the try operation of TCC.
2) Phase two
If the reservation of all business resources is successful in the try phase, the confirm operation is performed. Otherwise, the cancel operation is performed.
In the TCC model, both the initiator and participants of a transaction need to record logs for the transaction. The initiator records the status and information of the global transaction and each branch transaction. Participants record the status of the branch transactions.
At any stage of the TCC transaction execution process, exceptions such as downtime, restart, and network disconnection may occur. If this is the case, the transaction enters a non-atomic and non-consistent state. Hence, it is necessary to commit or roll back the remaining branch transactions based on the logs of the main transaction and the branch transactions. In this way, all transactions in the entire distributed system can reach the final consistency and atomicity.
Take an e-commerce system as an example. Xiao Ming bought a book on Taobao for RMB 100 and received 10 credits for this purchase. The purchase involves the following operations in the system:
These operations need to be executed as one transaction and must be all successful or all canceled.
If the TCC model is used in this case, the systems must be transformed as follows:
1) The order system
2) The payment system
3) The inventory system
4) The membership system
TCC transactions have four features:
The TCC transaction model is intrusive for the business as the business side needs to split one interface to three, which results in high development costs.
At the same time, in order to avoid exceptions caused by communication failures or timeouts in asynchronous networks, TCC requires the business side to follow three policies in design and implementation:
TCC implements distributed transactions at the business layer rather than the resource layer, which allows businesses to flexibly select a resource locking granularity. In addition, locks will not always be held in the global transaction execution process, therefore, the system throughput is much higher than that in the two-phase-commit XA mode.
Open-source frameworks that support TCC transactions include ByteTCC, Himly, and TCC-transaction.
Saga, another model for compensating transactions, is not a new concept. Saga-related papers were published in 1987, almost the same time as the XA two-phase commit protocol specification.
Saga, like TCC, is also a compensating transaction model, but it does not include a try phase. Saga regards distributed transactions as a transaction chain that is composed of a group of local transactions.
Each forward transaction operation in the transaction chain corresponds to a reversible transaction operation. The Saga transaction coordinator executes branch transactions in the transaction chain in sequence. After the branch transactions are all executed, resources are released. However, if a branch transaction fails, a compensating operation is performed in the opposite direction.
Assume that a Saga distributed transaction chain is composed of n branch transactions, that is, [T1, T2, ..., Tn]. Then, there are three possible conditions where the distributed transaction executes:
Assume that Xiao Ming wants to take a trip on the National Day holiday. He plans to depart from Beijing, spend three days in London, and then pay a three-day visit to Paris before returning to Beijing. The whole trip involves ticket reservations from different airlines and hotel reservations in London and Paris. Xiao Ming's plan is to cancel the trip if any of the reservations fail. Assume that a comprehensive travel service platform can make all reservations with one click, which resembles a long transaction. If the service is arranged by using Saga, as shown in the following figure, the trip reservation will be canceled through compensating operations when any of the reservations fail.
Saga transactions guarantee three transaction features:
However, Saga does not guarantee the isolation of transactions. A local transaction will be visible to other transactions after it is committed. If other transactions have changed the data that has been submitted successfully, the compensating operation may fail. For example, the deduction fails, but the money in the account has gone. Therefore, we need to consider this scenario and avoid this problem from business design.
Saga transactions, like TCC transactions, require the business design and implementation to follow three policies:
Although Saga and TCC are both compensating transaction models, they are different due to different commit phases.
The Saga mode is suitable for long transactions and microservices, as it is less intrusive to business. At the same time, Saga uses the one-phase commit mode, which does not lock resources for a long time and has no "cask effects." Therefore, systems with this architecture have high performance and high throughput.
Both Alibaba's open-source project Seata and Huawei's open-source project ServiceComb support Saga.
The core concept of the message-based distributed transaction mode is to inform other transaction participants of its own execution status through the message system.
The introduction of message systems can more effectively decouple transaction participants and allow each participant to act asynchronously.
The difficulty in implementing this mode is to ensure the consistency between the execution of local transactions and the sending of messages. Because, both actions must all be successful or all canceled.
There are two main solutions for the message-based distributed transaction mode:
Normal messages cannot ensure consistency between local transaction execution and message sending. The reason is that message sending is a network communication process, and an error or timeout may occur during message sending. If timeout happens for a message, the sender cannot determine whether the message reaches the destination or not. Therefore, no matter whether the message sender commits or rolls back the transaction, inconsistency still may occur.
To solve this problem, transactional messages are needed. Transactional messages differ from normal messages in that after a transactional message is successfully sent, it enters the prepared state and cannot be consumed by subscribers. The downstream subscribers can perceive this message only after the state of the transactional message changes to consumable.
The following figure shows the processes for sending local transactions and transactional messages.
The transactional-message-based model places high requirements on the MQ system. Not all MQ systems support transactional messages. RocketMQ is one of the few MQ systems that support transactional messages. If the MQ system does not support transactional messages, the local message mode can be used.
The core concept of this mode is that the transaction initiator maintains a local message table. Business and local message table operations are executed in the same local transaction. If the service is successfully executed, a message with the "to be sent" state is also recorded in the local message table. The system starts a scheduled task to regularly scan the local message table for messages that are in the "to be sent" state and sends them to MQ. If the sending fails or times out, the message will be resent until it is sent successfully. Then, the task will delete the state record from the local message table. The subsequent consumption and subscription process is similar to that of the transactional message mode.
The message-based distributed transaction mode supports ACID as follows:
Message-based distributed transactions can effectively decouple distributed systems from other systems, hence the calls between transaction participants are not synchronous calls.
Message-based distributed transactions have high requirements for the MQ system and bring certain intrusion to the business. For such transactions, either the interface for querying the transaction message status must be provided or the local message table needs to be maintained. In addition, message-based distributed transactions do not support transaction rollback. If a transaction fails, it must retry until it is successful. This feature makes it applicable to limited business scenarios that are less sensitive to eventual consistency. For example, calls between cross-enterprise systems.
The best-effort notification-based distributed transaction is another solution based on MQ systems, but it does not require MQ messages to be reliable.
Suppose Xiao Ming uses the Unicom mobile app to pay phone bills, and the prepayment method is Alipay. The whole process is as follows:
The best-effort notification solution is essentially to guarantee the eventual consistency by introducing a periodic verification mechanism. This solution is relatively easy to implement as it is less intrusive to business and has undemanding requirements on the MQ system. This method is applicable to scenarios with low sensitivity to the eventual consistency and short business paths, such as cross-platform and cross-enterprise business interactions between systems.
Alibaba provides two distributed transaction middleware:
XTS and TXC have similar functions, and both support the TCC transaction mode. They also provide distributed transaction solutions that are less intrusive to business. Currently, the two teams are working on developing the open-source distributed transaction middleware, Simple Extensible Autonomous Transaction Architecture (Seata). Next, let's take a look at Seata.
The following section describes the brief history of Seata:
Seata supports the TCC and Saga modes. Specially, Seata provides a solution with zero intrusion to TCC. This solution is called the Automatic Transaction (AT) mode. The mechanism of the AT mode is as follows:
The AT mode automatically generates rollback logs, which lowers access cost and business intrusion. However, the AT mode has the following limits:
Single database transactions can easily meet the ACID features and ensure strong consistency. However, it is difficult for distributed transactions to fully comply with ACID. In order to pursue the high availability and high throughput of distributed systems, distributed transaction solutions usually provide eventual consistency instead.
Transactions that provide strong consistency are called rigid transactions. Transactions that provide eventual consistency are called flexible transactions. Rigid transactions fully meet the ACID feature, whereas flexible transactions comply with ACID as follows:
Flexible transactions generally follow the Basically Available, Soft state, Eventual consistency (BASE) theory in the distributed field.
The BASE theory is an extension of the Consistency, Availability, Partition tolerance (CAP) theory. It is a balance between consistency and availability in CAP. The core concept of the BASE theory is that: we cannot achieve strong consistency, however, each application can achieve eventual consistency by using an appropriate method according to its own characteristics.
CAP shows that a distributed system cannot achieve consistency, availability, and partition fault tolerance at the same time. Therefore, we need to make a choice when designing the system. Rigid transactions pursue strong consistency, and therefore they sacrifice high availability. Flexible transactions sacrifice consistency in exchange for the high availability of the system.
When we select a distributed solution for the system, we can make the decision based on the requirements for consistency. For scenarios that require strong consistency, XA-defined two-phase-commit solutions should be prioritized. For scenarios that require eventual consistency only, flexible transaction solutions are preferred.
ApsaraDB - July 3, 2019
Alibaba Clouder - July 16, 2020
ApsaraDB - April 19, 2019
Alibaba Cloud MaxCompute - March 25, 2021
Alibaba Cloud Blockchain Service Team - August 19, 2019
Alibaba Clouder - May 17, 2018
AlibabaMQ for Apache RocketMQ is a distributed message queue service that supports reliable message-based asynchronous communication among microservices, distributed systems, and serverless applications.Learn More
A message service designed for IoT and mobile Internet (MI).Learn More
Message Queue is a distributed message queue service that supports reliable message-based asynchronous communication among microservices, distributed systems, and serverless applications.Learn More
A distributed, fully managed, and professional messaging service that features high throughput, low latency, and high scalability.Learn More