Troubleshoot host deployments - Alibaba Cloud DevOps - Alibaba Cloud Documentation Center

Host agent issues — Flow Runner

Troubleshoot Flow Runner issues

Important

Before you troubleshoot issues related to host deployments or environment provisioning for private build machines, first check the Flow Runner status on the machine.

Use the diagnostic tool

This tool supports only Linux systems.

Download the tool:

wget "https://rdc-public-software.oss-cn-hangzhou.aliyuncs.com/runner/runnerStatusCheck" -O runnerStatusCheck

Grant execute permissions:
```
chmod u+x runnerStatusCheck
```
Run the tool:
```
./runnerStatusCheck
```

Follow the instructions in the tool's output, as shown in the example below.

[INFO] prepare to check disk
check disk output
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        99G   59G   36G  62% /
devtmpfs        3.7G     0  3.7G   0% /dev
tmpfs           3.7G     0  3.7G   0% /dev/shm
tmpfs           3.7G  600K  3.7G   1% /run
tmpfs           3.7G     0  3.7G   0% /sys/fs/cgroup
tmpfs           748M     0  748M   0% /run/user/0
[INFO] prepare to check runner status
RunnerService: runner-v0.0.7-be-1fy1ifhg1ylcvqv7rwinqpgh.service  status is active running
RunnerService: runner-v0.0.7-be-3n5goas4jkp3clmcaopzeebj.service  status is active running
RunnerService: runner-v0.0.7-be-6fcm3z72s2fsnrkcjl7pbpsa.service  status is not running , please exeute 'systemctl restart runner-v0.0.7-be-6fcm3z72s2fsnrkcjl7pbpsa.service ' to restart runner service
RunnerService: runner-v0.0.7-be-7srpetjhkcy40btoontpsy8a.service  status is not running , please exeute 'systemctl restart runner-v0.0.7-be-7srpetjhkcy40btoontpsy8a.service ' to restart runner service
RunnerService: runner-v0.0.7-be-7ypo76lza21lljapgp5xtopi.service  status is active running
RunnerService: runner-v0.0.7-be-9zouh99ta6ycx2dq3iggekk9.service  status is not running , please exeute 'systemctl restart runner-v0.0.7-be-9zouh99ta6ycx2dq3iggekk9.service ' to restart runner service
RunnerService: runner-v0.0.7-be-ao81clhgw56bo0jopj5jgwn3.service  status is active running
RunnerService: runner-v0.0.7-be-bn9wuh8n4kiba0aa0a8dcirb.service  status is active running
RunnerService: runner-v0.0.7-be-fsr2gvdefe5yxl3nyijp36d3.service  status is active running
RunnerService: runner-v0.0.7-be-haphfbrhuudbxkzwpcuhn0hk.service  status is not running , please exeute 'systemctl restart runner-v0.0.7-be-haphfbrhuudbxkzwpcuhn0hk.service ' to restart runner service
RunnerService: runner-v0.0.7-be-mwkcqcv2hbwozrwmmczopzgm.service  status is active running
RunnerService: runner-v0.0.7-be-qdiyjifmn0bbkafobtfpkrst.service  status is not running , please exeute 'systemctl restart runner-v0.0.7-be-qdiyjifmn0bbkafobtfpkrst.service ' to restart runner service
RunnerService: runner-v0.0.7-be-qx6gaxojjjpzeblor692hrwk.service  status is active running
RunnerService: runner-v0.0.7-be-s5f7tixziynazefazqhls9ob.service  status is active running
RunnerService: runner-v0.0.7-be-uwdlmn516wd8emqebhgqwq6x.service  status is active running
RunnerService: runner-v0.0.7-be-xjmxrlbzrtksuafv4oyg8qtf.service  status is active running
Check runner service status finished.

Manual troubleshooting

Check if your Linux version is supported.

Run the lsb_release -a command to check your Linux distribution. The unified Flow Runner supports the following Linux distributions:
- CentOS 6+
- Ubuntu 16.04+
- Alibaba Cloud Linux 2/3

Check the Flow Runner service status and logs.

Run the ls -al /etc/systemd/system | grep runner command to find the service name. The service name is typically in the format runner-{version}-{tenant-name}.service.

[root@ecs-for-batch-deploy-1 ~]# lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-noarch
Distributor ID: CentOS
Description:    CentOS Linux release 8.2.2004 (Core)
Release:        8.2.2004
Codename:       Core
[root@ecs-for-batch-deploy-1 ~]# ls -al /etc/systemd/system | grep runner
-rw-r--r--    1 root root  550 Oct 11 10:50 runner-v0.xxx.service
[root@ecs-for-batch-deploy-1 ~]#

Run the systemctl status runner-{version}-{tenant-name}.service command to check the service status. If the status is active (running), the service is operating correctly.

Release:        8.2.2004
Codename:       Core
[root@ecs-for-batch-deploy-1 ~]# ls -al /etc/systemd/system | gr
-rw-r--r--   1 root root  550 Oct 11 10:50 runner-v0.0.5-be-bnyxxx...yt.service
[root@ecs-for-batch-deploy-1 ~]# systemctl status runner-v0.0.5-be-bnyxxx...yt.service
● runner-v0.0.5-be-bnxxx...yt.service - Aliyun yunxiao runner polling jo
     Loaded: loaded (/etc/systemd/system/runner-v0.xxx...yt.service)
     Active: active (running) since Tue 2022-10-11 10:50:56 CST; 2h ago
   Main PID: 1405243 (runner)
      Tasks: 10 (limit: 48875)
     Memory: 13.6M
     CGroup: /system.slice/runner-v0.0.5-be-brxxx...yt.service
             └─1405243 /usr/local/share/yunxiao-runner/v0.0.5/runner run --configPath=/r...
Oct 11 11:01:03 ecs-for-batch-deploy-1 runner[1405243]: INFO[2022-10-11T11:01:03+08:00]
Oct 11 11:01:03 ecs-for-batch-deploy-1 runner[1405243]: INFO[2022-10-11T11:01:03+08:00]
Oct 11 11:01:18 ecs-for-batch-deploy-1 runner[1405243]: WARN[2022-10-11T11:01:18+08:00]

To view real-time execution logs, run the journalctl -u runner-{version}-{tenant-name}.service -a --no-pager --since '5 minutes ago' -f command.

Cross-region deployment failures

When deploying across geographic regions (for example, from Hong Kong (China) to Malaysia) over the Internet, deployment failures may occur due to network latency or instability.

In addition to checking the Flow Runner status and disk space, verify the following:

Test network latency and packet loss from the target host to the Alibaba Cloud DevOps service endpoint using ping or traceroute.
If the network is unstable, consider optimizing the network link or using a dedicated cross-region connection to improve reliability.

No logs generated or host is offline during build or deployment

Run df -hl to check if the host's disk is full. If it is, free up space and retry.
Check the Flow Runner service status. If the status is not active(running), restart the service by following these steps:
1. Get the service name. The format is runner-{version}-{tenant-name}.service.
```
systemctl | grep "runner-v" | awk '{print $1}'
```
2. Replace $SERVICE_NAME in the following command with the service name and run it to restart the Flow Runner service.
```
systemctl restart $SERVICE_NAME
```

Check for network connectivity issues.

Run systemctl status runner-{version}-{tenant-name}.service to query the Flow Runner service status and find the --configPath parameter, as shown below:

[root@ecs-for-batch-deploy-1 ~]# systemctl status runner-v0.0.5-be-bnyxxx1llzpci0q606yt.service
● runner-v0.0.5-be-xxx...yt.service - Aliyun yunxiao runner polling jobs to execute
   Loaded: loaded (/etc/systemd/system/runner-v0.0.5-be-bxxx...yt.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-10-11 10:50:56 CST; 41min ago
 Main PID: 1405243 (runner)
    Tasks: 10 (limit: 48875)
   Memory: 13.7M
   CGroup: /system.slice/runner-v0.0.5-be-bnyxxx...yt.service
           └─1405243 /usr/local/share/yunxiao-runner/v0.0.5/runner run --configPath=/root/yunxiao/be-bnykv60t831llzpci0q606yt/runner/config
Oct 11 11:31:46 ecs-for-batch-deploy-1 runner[1405243]: INFO[2022-10-11T11:31:46+08:00] [runner] no new job, skip.                    runner=9e7073c92b2fa8b35bc0efd3eec051f8
Oct 11 11:31:46 ecs-for-batch-deploy-1 runner[1405243]: INFO[2022-10-11T11:31:46+08:00] counter current: 0, limit: 50
Oct 11 11:32:01 ecs-for-batch-deploy-1 runner[1405243]: WARN[2022-10-11T11:32:01+08:00] Response status code: 204 body data:
Oct 11 11:32:01 ecs-for-batch-deploy-1 runner[1405243]: INFO[2022-10-11T11:32:01+08:00] POST /api/v2/builds/request, time spent 15.07 s
Oct 11 11:32:01 ecs-for-batch-deploy-1 runner[1405243]: INFO[2022-10-11T11:32:01+08:00] [runner] no new job, skip.                    runner=9e7073c92b2fa8b35bc0efd3eec051f8
Oct 11 11:32:01 ecs-for-batch-deploy-1 runner[1405243]: INFO[2022-10-11T11:32:01+08:00] counter current: 0, limit: 50
Oct 11 11:32:16 ecs-for-batch-deploy-1 runner[1405243]: WARN[2022-10-11T11:32:16+08:00] ...

View the URL in the configPath by running cat {PATH_TO_CONFIG}/config.yml | grep url.

[root@ecs-for-batch-deploy-1 config]# cat /root/yunxiao/be-bnykv60t83lllzpci0q606yt/runner/config/config.yml | grep url
url: https://xxx.com
[root@ecs-for-batch-deploy-1 config]#

Run the following command to check URL accessibility.

# Note: Replace {url} in the command below with your actual URL.
curl '{url}/api/v2/runner/storage/latest?os=linux&arch=amd64'

Host deployment fails with a MachineGroupNotHaveMachine error

This error indicates that the deployment group has no available machines, so the deployment task cannot be dispatched to the target host.

To troubleshoot this issue:

In the Alibaba Cloud DevOps console, go to the deployment task configuration for the pipeline and check whether the associated deployment group contains machines. If the deployment group is empty, add the target hosts first.
If the deployment group already contains machines, verify that they are online. If a machine is offline, log in to the server, check the Runner service status, and restart the service.
After confirming that the deployment group has available online machines, re-trigger the deployment task.

How do I prevent errors from requests reaching services during startup in a multi-node deployment?

Alibaba Cloud DevOps does not provide built-in traffic control during deployments. To manage traffic and prevent requests from reaching services that are still starting up, use the following approach:

Split your hosts into two host groups and configure sequential deployments in the pipeline. This controls the deployment order and ensures one group is fully available before the other is updated.
Use gateway-level request interception or configure forwarding rules with Application Load Balancer (ALB) to route traffic away from nodes that are still starting up.