Use the agent troubleshooting feature - Security Center - Alibaba Cloud Documentation Center

When the Security Center agent goes offline, fails to be installed or uninstalled, or causes high CPU usage, use the automated troubleshooting tool for quick diagnosis or follow the manual troubleshooting procedure to check items one by one.

Background

If the Security Center console shows that the agent is not installed or is offline, the server loses the protection of Security Center and may be exposed to intrusion risks. The following table describes the common causes of agent offline issues.

Cause	Description
Abnormal agent process	The core agent processes (AliYunDun or AliYunDunUpdate) are not running properly. Possible causes include process crashes, manual termination, or system errors.
Network connection failure	The network between the server and the Security Center backend is disconnected, preventing heartbeat reporting.
DNS resolution failure	The DNS service on the server is abnormal and cannot resolve the domain name of the Security Center backend.
Firewall or security group restrictions	The firewall ACL rules or Alibaba Cloud security group rules on the server block communication between the agent and the Security Center backend.
Insufficient server resources	Sustained high server resource usage (CPU or memory above 95%) disrupts agent operation.
Third-party security software conflicts	Third-party antivirus software installed on the server blocks network access for the Security Center agent.

Troubleshooting methods

Method	Use case	Description
Console	The server is connected to Security Center.	Use the agent troubleshooting feature in the console to automatically collect and analyze agent data.
CLI	The server is not connected to Security Center.	Run the `aegis_checker` troubleshooting tool on the server to automatically diagnose agent issues.
Manual	The server does not support Agent Troubleshooting or the `aegis_checker` troubleshooting tool.	Check the agent process, network connection, and system resources one by one to identify the cause of the offline issue.

Console troubleshooting

In the Security Center console, use the Agent Troubleshooting feature to automatically detect and analyze agent issues.

Scope

Supported operating systems:
- Windows Server 2008 and later
- 64-bit Linux (CentOS 5 and earlier are not supported)
The server is connected to Security Center.

Procedure

Log on to the Security Center console.
In the navigation pane on the left, choose Assets > Host. In the upper-left corner of the console, select the region where your assets are located: Chinese Mainland or Outside Chinese Mainland.
On the Host page, click the Server tab. In the server list, select the server that you want to troubleshoot, and then click Agent Troubleshooting below the list.

In the Agent Troubleshooting dialog box, select the Issue Type and Mode for the agent troubleshooting task, and then click Start Check.

Parameter

Description

Issue Type

Select the type of issue that the agent is experiencing. If you are unsure about the issue, select Overall Check (Unknown Issues).

Mode

Select the troubleshooting mode. The following modes are available:

Standard Mode: Standard mode collects agent-related log data and reports the data to Security Center for analysis. The troubleshooting takes about 1 minute.
Enhancement Mode: Enhanced mode collects agent-related network, process, and log data and reports the data to Security Center for analysis. The troubleshooting takes about 5 minutes.

Note

The diagnostic program collects agent-related network, process, and log data from the server and reports the data to Security Center for analysis.

In the Note dialog box, click OK to open the Task Management panel and view all agent troubleshooting tasks.
Note
Open the Task Management panel by clicking Agent Task Management in the upper-right corner of the Host page.

Find the troubleshooting task that you want to view and click Details in the Actions column to open the Run Logs panel. The Run Logs panel displays the troubleshooting details for each server.

Column	Description
Start Time/End Time	The start time and end time of the agent troubleshooting task.
Server Information	The information about the servers included in the troubleshooting task.
Status	The status of the troubleshooting task. Valid values: Started: The troubleshooting command has been sent. Timed Out: The troubleshooting command has not returned results within the expected time. Succeeded: The troubleshooting results have been generated.
Issue	The issues detected during the troubleshooting task.
Result	The solutions for the issues detected during the troubleshooting task.
Actions	The diagnostic log of the troubleshooting task. Download the diagnostic log for further analysis.

Result analysis:
- For some of the detected issues, a solution is provided in the Result column. Follow the provided solution to resolve the issue.
- If no solution is provided in the Result column, click Download Diagnostic Logs in the Actions column to download the diagnostic log. Provide the exported diagnostic log and your AliUid to Alibaba Cloud technical support for further analysis.

CLI troubleshooting

Run the troubleshooting tool command on the server to automatically diagnose agent issues.

Scope

Supported server operating systems:
- Windows Server 2008 and later
- 64-bit Linux (CentOS 5 and earlier are not supported)
The server is connected to Security Center.

Procedure

Log on to the server.
Note
- Log on to Windows with administrator privileges.
- Log on to Linux with root privileges.

Run the following command on the server.

Alibaba Cloud ECS - Linux

Standard mode (about 1 minute):
- If the ECS instance can connect to Security Center over the network, run the following command as root:
```
wget "http://update2.aegis.aliyun.com/download/aegis_client_self_check/linux64/aegis_checker.bin" && chmod +x aegis_checker.bin && ./aegis_checker.bin
```
- If the ECS instance cannot connect to Security Center over the network, download aegis_checker and copy it to the server. Then, run the following command as root:
```
chmod +x aegis_checker.bin
 ./aegis_checker.bin
```

Enhanced mode (about 5 minutes): Run the following command as root:

wget "http://update2.aegis.aliyun.com/download/aegis_client_self_check/linux64/aegis_checker.bin" && chmod +x aegis_checker.bin && ./aegis_checker.bin -b "ew0KICAgICJ1dWlkIjogIiIsDQogICAgImNtZF9pZHgiOiAiIiwNCiAgICAiaXNzdWUiOiAib3RoZXJfaXNzdWUiLA0KICAgICJtb2RlIjogMywNCiAgICAianNydl9kb21haW4iOiBbXSwNCiAgICAidXBkYXRlX2RvbWFpbiI6IFtdDQp9"

Alibaba Cloud ECS - Windows

Standard mode (about 1 minute): Use one of the following methods to troubleshoot:

Download the aegis_checker program and run it as an administrator.

Run the following command in a cmd window as an administrator:

powershell -executionpolicy bypass -c "(New-Object Net.WebClient).DownloadFile('http://update2.aegis.aliyun.com/download/aegis_client_self_check/win32/aegis_checker.exe', $ExecutionContext.SessionState.Path.GetUnresolvedProviderPathFromPSPath('.\aegis_checker.exe'))"; "./aegis_checker.exe"

Note

Enhanced mode is not supported on Windows.

Non-Alibaba Cloud - Linux

Standard mode (about 1 minute): Run the following command as root:

wget "http://aegis.alicdn.com/download/aegis_client_self_check/linux64/aegis_checker.bin" && chmod +x aegis_checker.bin && ./aegis_checker.bin

Enhanced mode (about 5 minutes): Run the following command as root:

wget "http://aegis.alicdn.com/download/aegis_client_self_check/linux64/aegis_checker.bin" && chmod +x aegis_checker.bin && ./aegis_checker.bin -b "ew0KICAgICJ1dWlkIjogIiIsDQogICAgImNtZF9pZHgiOiAiIiwNCiAgICAiaXNzdWUiOiAib3RoZXJfaXNzdWUiLA0KICAgICJtb2RlIjogMywNCiAgICAianNydl9kb21haW4iOiBbXSwNCiAgICAidXBkYXRlX2RvbWFpbiI6IFtdDQp9"

Non-Alibaba Cloud - Windows

Standard mode (about 1 minute): Use one of the following methods to troubleshoot:

Download the aegis_checker program and run it as an administrator.

Run the following command in a cmd window as an administrator:

powershell -executionpolicy bypass -c "(New-Object Net.WebClient).DownloadFile('http://aegis.alicdn.com/download/aegis_client_self_check/win32/aegis_checker.exe', $ExecutionContext.SessionState.Path.GetUnresolvedProviderPathFromPSPath('.\aegis_checker.exe'))"; "./aegis_checker.exe"

Note

Enhanced mode is not supported on Windows.

After the check is complete, export the generated log archive. The storage path of the log archive varies depending on the operating system of the server.
- Linux: The log archive is stored in the /root/miniconda2/aegis_checker/output directory.
- Windows: The log archive is stored in the ./miniconda2/aegis_checker/output directory under the current directory.
Result analysis: In the check result log, entries prefixed with [root cause] indicate the issues detected by aegis_checker.
1. For some issues, a solution or a handled status is provided. Follow the instructions to resolve the issue.
2. If aegis_checker does not provide a solution, provide the screenshots of the check results, the log archive, and your AliUid to Alibaba Cloud technical support for further analysis.

Manual troubleshooting

If the agent is offline, log on to the server and follow the steps below to check items one by one to identify the cause.

Agent processes

Diagnostic steps: Verify that the two core processes, AliYunDun and AliYunDunUpdate, are running.

Linux: Run the ps -ef | grep AliYunDun command to check.
Windows: Open **Task Manager** and go to the **Details** or **Services** tab to find the related processes and services.

Resolution: Manually restart the agent processes.

Linux

Run the following commands to restart the processes.

Stop the related processes:

killall AliYunDun
killall AliYunDunUpdate

Start the latest version of the agent.

In the /usr/local/aegis/aegis_client directory, find the aegis_10_xx folders and select the one with the highest version number.

For example, among aegis_10_70, aegis_10_73, and aegis_10_75, select aegis_10_75.
```
/usr/local/aegis/aegis_client/aegis_10_xx/AliYunDun
```

Windows

In the Services panel, restart the two Security Center services: Alibaba Security Aegis Detect Service and Alibaba Security Aegis Update Service. To do so, right-click each service and select Restart.

Network connection

Diagnostic steps: Verify that your firewall or security group allows outbound traffic to the Security Center service IP addresses or domain names, such as jsrv.aegis.aliyun.com or update.aegis.aliyun.com. The agent can also go offline if the server cannot connect to the Security Center service.

Note

For more information about the Security Center service IP addresses and domain names, see Appendix: Agent communication endpoints (domain names and IP addresses).

Resolution:

Verify that the DNS service on the server is running correctly.

If the DNS service is not running, restart the server or troubleshoot the DNS service.
Check if network access policies are configured on the server.
1. Firewall ACL rules
  
  Add the Security Center service IP addresses or domain names to your firewall's allowlist to permit network access. You only need to configure rules for outbound traffic.
  
  Note
  If you use Alibaba Cloud Firewall, see Create an outbound access control policy for traffic from an internal network to the Internet for instructions.
  
  Example firewall configuration (iptables):
```
# Allow access to the control service
iptables -A OUTPUT -p tcp -d jsrv.aegis.aliyun.com --dport 443 -j ACCEPT
iptables -A OUTPUT -p tcp -d jsrv.aegis.aliyun.com --dport 80 -j ACCEPT

# Allow access to the update service
iptables -A OUTPUT -p tcp -d update.aegis.aliyun.com --dport 443 -j ACCEPT
iptables -A OUTPUT -p tcp -d update.aegis.aliyun.com --dport 80 -j ACCEPT
```
2. Alibaba Cloud security group rules
  
  If you use an ECS instance, see Manage security groups for specific steps.
  
  Note
  Allow outbound traffic to the Security Center CIDR blocks. You can either leave the port unrestricted or allow traffic on ports 80 and 443.
  
  The following is an example configuration for the 100.100.0.0/16 CIDR block:
  - Direction: Outbound
  - Authorization policy: Allow
  - Protocol type: TCP
  - Port range: 80/443
  - Authorization object: 100.100.0.0/16

System resources

Diagnostic steps:

Verify that the server has sufficient resources. The agent may stop running if server resources are exhausted.

CPU/Memory: Use top (Linux) or **Task Manager** (Windows) to check the usage.
Disk space: Use df -h (Linux) or **This PC** (Windows) to check the remaining disk space.

Resolution:

High resource usage
- If the AliYunDun process is the cause, contact technical support and provide the relevant logs.
- If other business processes are the cause, optimize your applications or consider upgrading the server configuration.
Insufficient disk space: Delete unnecessary files to free up disk space.

Duplicate agent IDs

Diagnostic steps: This issue often occurs when you create multiple servers from the same system image. Check if the uuid field in the following configuration files is duplicated across multiple servers.

Linux: /usr/local/aegis/aegis_client.conf
Windows:
- 32-bit: C:\Program Files\Alibaba\aegis\aegis_client.conf
- 64-bit: C:\Program Files (x86)\Alibaba\aegis\aegis_client.conf

Resolution:

Before creating multiple images from a single template server, uninstall and clean up the old agent, and then obtain a new installation command.

Software conflicts

Diagnostic steps: Check if other Host-based Intrusion Detection System (HIDS), Endpoint Detection and Response (EDR), or antivirus software is installed on the server. Such software can conflict with the Security Center agent.

Resolution:

Disable or uninstall the third-party security software. After the Security Center agent is installed, you can restart or reinstall the original software as needed.

Agent logs

Diagnostic steps: Review the agent logs for specific error messages. The log files are located in the following directories:

Linux: /usr/local/aegis/aegis_client/aegis_12_xx/data/.

Note
The aegis_xx_xx placeholder represents the version directory for the running agent. To find the exact path, check the output of the ps -ef|grep AliYunDun command.
Windows: C:\Program Files (x86)\Alibaba\Aegis\aegis_client\aegis_12_xx\data\.

Resolution:

Troubleshoot the issue based on the error messages in the logs. If you cannot resolve the issue, contact technical support and provide the complete log files.