Test service performance when facing system time jumps in ECS - Elastic Compute Service

A time jump refers to a sudden change in the system clock. The accuracy of system time and the time consistency across system components are crucial for production systems. Without proper time management, various time-dependent services such as logging, synchronization, and backup may experience abnormal behavior. Through time jump drills, you can verify whether the system can promptly synchronize and recover the correct time and resume business operations when a system time jump occurs.

Implementation

This solution uses the Cloud Assistant plugin ACS-ECS-TimeJump, which can adjust the time forward or backward by setting the system time offset. The fault recovery is achieved through the Chrony service synchronizing with NTP time servers. Therefore, the Chronyc service must be installed to use this plugin for fault injection.

Procedure

Prerequisites

Cloud Assistant Agent is installed on the ECS instance for which you want to perform a drill.
The status of Cloud Assistant is Normal on the ECS instance. For more information, see View the status of Cloud Assistant and handle anomalies.

Fault injection

Connect to the ECS instance.
For more information, see Use Workbench to connect to a Linux instance through SSH.
Use a user with sudo access permissions to run the Cloud Assistant plugin ACS-ECS-TimeJump.
```
sudo acs-plugin-manager --exec --plugin ACS-ECS-TimeJump --params inject,[time=paramA],[duration=paramB]
```
Parameters in [] are optional fault injection parameters. Parameter descriptions:
- time (optional): Time offset in seconds. A positive integer adjusts the time forward, and a negative integer adjusts the time backward. The default value is 1.
- duration (optional): Duration in seconds. The default value is 300.
When the following information is displayed, the Cloud Assistant plugin ACS-ECS-TimeJump has started.
Run the date command to check whether the current system time meets your expectations.

Fault recovery

Method 1: Wait for automatic recovery after timeout.
Method 2: Execute the fault recovery command in the ECS instance.
```
sudo acs-plugin-manager --exec --plugin ACS-ECS-TimeJump --params recover
```
Important
The recovery command actually triggers a time synchronization. The time required for Chrony service synchronization depends on multiple factors. Therefore, you must check whether the time has been recovered after the drill ends.

Drill example

To better observe the time jump process, run the following script in the terminal. This script prints the current system time every second.
```
while true; do
    date +"%Y-%m-%d %H:%M:%S"
    sleep 1
done
```
Inject data to the instance to simulate faults.
```
sudo acs-plugin-manager --exec --plugin ACS-ECS-TimeJump --params inject,time=1800
```
This command adjusts the current system time forward by 1800 seconds. The following information is returned, which includes the adjusted system time.
Check the fault injection effect.
You can see that at 15:35:01, the system time jumps.
Execute the recovery command.
The system automatically synchronizes time with the NTP server.
sudo acs-plugin-manager --exec --plugin ACS-ECS-TimeJump --params recover
```
sudo acs-plugin-manager --exec --plugin ACS-ECS-TimeJump --params recover
```
The system time returns to normal.

Set system time

Configure the NTP time synchronization service and check whether the time synchronization service has started. For more information, see Manage time synchronization service.