Change Request: A More Flexible Software Integration and Release Model

By Dong Yue (Nickname: He Chu), Senior Product Expert at Alibaba R&D Effectiveness Department

Today, the typical software integration and release model is to manage feature branches by using methods like Pull Request on GitHub or MergeRequest on GitLab. After changes to a feature branch are proven correct and proper through code review, they are merged into the integration branch. Next, code changes are included in the continuous delivery pipeline running on the integration branch until they are released.

Although the R&D collaboration tool Aone (publicly known as Yunxiao) also supports this model, most researchers and developers at Alibaba do not choose this model. Instead, they use an object called a Change Request to manage feature branches before the release.

At first glance, a Change Request may look very similar to a Pull or MergeRequest. Conceptually, they are actually very different.

This article describes the similarities and differences between these methods and explains why users prefer change requests. Risk and disadvantages of change requests are also shown in this article. I hope this article will be helpful for you.

Similarities

The similarities between a Change Request and a Pull/MergeRequest mainly lie in the quality and process control of feature branches themselves:

Like a Pull/MergeRequest, a change request generally corresponds to a feature branch.
When a Pull/Merge Request is used, code changes to the corresponding feature branch are made visible for code review. Similarly, code review can also be performed by changing the granularity.
Code submission on a feature branch can automatically trigger continuous integration tools to do the creation work and implement various automatic detections. The results can be shown in the Pull/Merge Request. Similarly, results can also be shown in a change request.
You can deploy the latest code in a Pull/MergeRequest into its exclusive test environment and run the code for testing and debugging. This can also be done in a change request.
In a Pull/MergeRequest, you can set the conditions of approval. For example, at least two reviewers approve the changes, all problems found in code review are fixed or clarified, and the continuous integration pipeline on a feature branch are running successfully. Similar settings are also supported in a change request.

Differences and main advantages

The main difference between a change request and a Pull/MergeRequest is how the corresponding feature branch is integrated and delivered together with other feature branches.

The feature branch corresponding to a Pull/MergeRequest is incorporated into the integration branch. After that, code changes are not managed at the granularity of feature branches. That feature branch has been incorporated into the integration branch, and code changes on that feature branch have been merged into the integration and release procedures together with code changes on other feature branches to go through each integration and release stage.

However, a change request is different. Even if a change request has been integrated with other change requests, it features a certain level of independency and flexibility. When necessary, operations can be performed on that change request separately. Next we will use two examples to give more detailed description.

Example 1

For simplicity, assume that the integration and delivery are implemented in three phases: daily integration testing, pre-release environment testing, and general availability release. An application has five changes: changes A to E.

After going through the daily integration testing, application changes are now in the phase of pre-release environment testing. In this testing phase, change C is founded to have one fault. This fault is not exposed in the daily integration testing phase due to the limits of the daily integration testing environment.

After proper analysis is made, change C is found independent of the other four changes. To minimize the impact on the release of the other four changes, we decide to remove change C from the pre-release environment testing. Then this fault does not occur during the test and validation of the other four changes. These four changes pass the pre-release environment testing and proceed to the general availability release phase.

In this example, after change C is removed, the other four changes do not receive the daily integration testing again for two reasons. One is that the daily integration testing environment has been used by new changes. It takes time to test these new changes and repeated adjustments may be necessary. It is obviously improper and disadvantageous to banish new changes out of the daily integration testing environment, mix the four existing changes with new changes or put the four changes on the waiting list.

The other reason is that changes A, B, D, and E together with change C have already completed the daily integration testing. In addition, change C is not related to the other four changes. It is very unlikely to find new problems if daily integration tests are performed again on these four changes. Testing also needs to be cost-effective instead of solely ensuring zero fault. For these reasons, a development team may evaluate the actual conditions and decide to test the four other changes in pre-release environment testing after change C is removed instead of putting these four changes into the daily integration testing phase again.

Example 2

Still, let's assume that the integration and delivery are implemented in three phases: daily integration testing, pre-release environment testing, and general availability release. An application has five changes: changes A to E. After going through the daily integration testing, application changes are now in the phase of pre-release environment testing.

At this point, we need to make a few adjustments to the new feature represented by change C based on some market condition changes, for example, changing a few words in the page description. Since we either need to release both the new change and change C or release neither of them, we decide to make the new change right in the feature branch corresponding to change C for easy management. Then test the updated content of change C and the other four changes in the pre-release environment and if all the changes pass the pre-release test, make them generally available.

The preceding two examples inject a certain level of flexibility into the traditional integration and release model: a change can be removed, modified or improved during the process of integration and release when necessary. Some teams may adopt a simpler approach to deal with change requests. They no longer have roles such as integration engineers and no longer plan unified integration and release as well as points in time. Each developer is responsible for their own changes and keeps track of the changes until the quality is improved adequately enough to integrate changes.

In addition, each developer is responsible for pushing the changes that they handle into individual stages of the integration and releases and tracks changes until they are made generally available. In other words, even individual changes are in the integration and release stages, they are tracked and pushed by the corresponding developers. These changes may be pushed at a different speed and pace, without intervening each other.

Mutually independent changes use a specific test environment and go through the same batch testing just by coincidence to improve test efficiency, and are made generally available at the same time to avoid queuing. Therefore, the duration between development and release is minimized and this supports relatively frequent releases. This also conforms to the "You build it, you run it" DevOps principle.

This change trend is similar to what happens in software development management practices. The waterfall model is an exception. The waterfall model is replaced with iteration methods. Sprint in the Scrum method is a typical example. Lean development weakens the role of iterations. The focus switches from what each iteration does and which stage each iteration is experiencing to which stage each work-in-process is currently in and what the total number of the work-in-process products is in each stage.

Similarly, the focus of the preceding change management methods evolves from the current integration and release phase of each specific integration version to the current integration and release phase of each change and the changes included in each phase.

Additional Advantages of Change Request

The preceding section describes the flexibility enabled by the change request management method and the improved efficiency due to business flexibility. The change request management has some advantages in information recording and tracking.

To know what features are included in a test and release, simply see what changes are included. A change itself has related descriptive text. A change can also be associated with items such as requirements, tasks, and faults to explain the purpose of the change in detail. Except by using change requests, no other code modifications can be directly submitted to the integration.

All modifications on a change request are made on the corresponding feature branch and will not be dispersed in different places after multiple modifications. Therefore, we can conveniently view and operate on these modifications. In the meantime, we always know the status of a change and its current phase. Does the development process finish? Is it now in the daily integration testing phase? Is it generally available? We can easily answer these questions.

A change can be associated with items such as requirements, tasks, and defects. Meanwhile, the status of a change can be automatically obtained. Therefore, in principle, the tracking items on the kanban can be automatically moved to reflect the actual status. The kanban clearly shows the collaboration and progress.

Disadvantages and Potential Risks

The preceding sections all talk about the advantages of the change request method. However, does this method have disadvantages or risks?

Yes, of course it does. From the explosive integration to the continuous deployment pipeline, the industry has been adopting a basic model for decades to always have one integration version experience integration and release phases in sequence. This ensures that the content in the next phase has been well tested in the preceding stage. The change request method described here brings more flexibility and significantly improves this basic model. Flexibility has always been a double-edged sword. Flexibility can also mean increased risk and misuse.

"Individuals and Interactions over processes and tools" is one of the values in the agile software development manifesto. This change request method is also in accordance with this value.

However, using this method in real scenarios puts higher requirements on team members. Team members are required to evaluate the change relevancy and risk in real specific scenarios, understand the impact of different choices on the efficiency, and make decisions based on a comprehensive view of specific scenarios. Specifically:

The fewer the code modifications corresponding to a change, the smaller the risk of removing the change halfway.
The fewer the code modifications corresponding to the modification and improvement of a change halfway, the smaller the risk.
The better the software architecture, the smaller the risk of removing, modifying or improving a change halfway.
The smaller the relevancy between one change and other changes, the smaller the risk of removing, modifying or improving a change halfway.
The more urgent it is, the more you should consider flexible handling.
From the business perspective, the higher the software quality requirement is, the less you should consider flexible handling.

In fact, in the era of microservices or function services, the aforementioned risks still exist even if the change request method is not used. As mentioned before, team members need to have the ability to make proper judgment themselves. Why?

One main reason why a single application is split into microservices or even function services is that each service can be tested and released separately. However, when microservices or function services are used, the test object is not strictly a service, but the service and all the other services that directly or indirectly interact with that service in the test environment. Testing and releasing each service separately may often cause that the service version in the current test phase is different from that in the next test phase or the general availability release. This poses a risk similar to that of the change request method.

Accordingly, manual judgment (perhaps with the assistance of intelligent algorithms) is required to determine which service changes must be tested and released together and which service changes can operate separately this time. Next time, the situation may vary and proper judgment is required accordingly.

Judging from this, the integration and release is silently already beyond this basic model of "always having one integration version experience integration and release stages in sequence". The Change Request method just makes this more obvious.

Implementation and Tool Support

The preceding sections describe the unique Change Request method, its advantages, and the risk. Now let's see how this method is implemented at Alibaba.

First, a branch solution is required to support this method. Basically, the procedures are like the following:

The master branch always represents the latest release version.
Code changes are always done on feature branches. Feature branches are always pulled from the master branch and synchronized from the master again when necessary.
Not a single long-lasting integration and release branch exists. Instead, each integration and release stage corresponds to a short-term and automatically managed integration and release branch. An integration and release branch is automatically pulled from the master branch and each feature branch is automatically incorporated into that branch (manual intervention required in the case of conflicts). Then the branch has all desired features.
To further modify and improve a feature, do this on the corresponding feature branch and incorporate that feature branch into the integration and release branch.

This solution puts high requirements on the tool platform. Users only need to manage changes that are included in each integration and release stage from the interface perspective. The tool platform needs to map this solution into the management of integration and release branches, including creating new branches, reusing existing branches, and incorporating feature branches to an integration and release branch. Many useful algorithms are also included to minimize the recurrence of the same incorporation conflict.

The high requirement on tool platforms is possibly why this solution has been widely used at Alibaba for many years.

Many colleagues that previously worked for Alibaba complain that they can't use such a good tool any more. Now this is no longer a problem. Just as Google has publicly provided Kubernetes based on its internal Borg, Alibaba also publicly provides Yunxiao based on its internal R&D collaboration tool platform Aone.

Community

Change Request: A More Flexible Software Integration and Release Model

Similarities

Differences and main advantages

Example 1

Example 2

Additional Advantages of Change Request

Disadvantages and Potential Risks

Implementation and Tool Support

Read previous post:

Read next post:

Alibaba Clouder

You may also like

Comments

Alibaba Clouder

Related Products

DevOps Solution

Alibaba Cloud Flow

Offline Visual Intelligence Software Packages

ActionTrail