主機部署問題 - Alibaba Cloud DevOps

主機代理問題 — 流水線 Runner

同一台 ECS 執行個體被添加到多個主機群組導致部署失敗或頻繁斷連怎麼辦？

同一台 ECS 執行個體被添加到多個主機群組時，可能出現 Runner 離線、版本過舊或配置衝突，導致部署失敗或 Runner 頻繁斷連。

快速恢複

在伺服器上查看 Runner 服務名稱：
```
ls -al /etc/systemd/system | grep runner
```

重啟對應的 Runner 服務：

systemctl restart runner-{版本}-{租戶名稱}.service

徹底解決

如果重啟後問題持續，建議卸載舊版 Runner 並安裝最新版本（如 0.3.4）。操作步驟：

停止 Runner 服務：

systemctl stop runner-{版本}-{租戶名稱}.service

刪除 systemd 設定檔：

rm -rf /etc/systemd/system/runner-{版本}-{租戶名稱}.service

刪除 Runner 配置目錄：
```
rm -rf /root/yunxiao
```
在雲效控制台重新安裝最新版 Runner 外掛程式。

流水線 Runner 問題排查方法

重要

在開始處理主機部署、私人構建機申請環境失敗問題之前請先按以下步驟確認機器上雲效runner狀態。

使用工具排查

只支援Linux系統

下載工具：

wget "https://rdc-public-software.oss-cn-hangzhou.aliyuncs.com/runner/runnerStatusCheck" -O runnerStatusCheck

設定執行許可權：
```
chmod u+x runnerStatusCheck
```
執行工具：
```
./runnerStatusCheck
```

按照工具提示資訊進行處理，樣本如下。

[INFO] prepare to check disk
check disk output
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        99G   59G   36G  62% /
devtmpfs        3.7G     0  3.7G   0% /dev
tmpfs           3.7G     0  3.7G   0% /dev/shm
tmpfs           3.7G  600K  3.7G   1% /run
tmpfs           3.7G     0  3.7G   0% /sys/fs/cgroup
tmpfs           748M     0  748M   0% /run/user/0
[INFO] prepare to check runner status
RunnerService: runner-v0.0.7-be-1fy1ifhg1ylcvqv7rwinqpgh.service  status is active running
RunnerService: runner-v0.0.7-be-3n5goas4jkp3clmcaopzeebj.service  status is active running
RunnerService: runner-v0.0.7-be-6fcm3z72s2fsnrkcjl7pbpsa.service  status is not running , please exeute 'systemctl restart runner-v0.0.7-be-6fcm3z72s2fsnrkcjl7pbpsa.service ' to restart runner service
RunnerService: runner-v0.0.7-be-7srpetjhkcy40btoontpsy8a.service  status is not running , please exeute 'systemctl restart runner-v0.0.7-be-7srpetjhkcy40btoontpsy8a.service ' to restart runner service
RunnerService: runner-v0.0.7-be-7ypo76lza21lljapgp5xtopi.service  status is active running
RunnerService: runner-v0.0.7-be-9zouh99ta6ycx2dq3iggekk9.service  status is not running , please exeute 'systemctl restart runner-v0.0.7-be-9zouh99ta6ycx2dq3iggekk9.service ' to restart runner service
RunnerService: runner-v0.0.7-be-ao81clhgw56bo0jopj5jgwn3.service  status is active running
RunnerService: runner-v0.0.7-be-bn9wuh8n4kiba0aa0a8dcirb.service  status is active running
RunnerService: runner-v0.0.7-be-fsr2gvdefe5yxl3nyijp36d3.service  status is active running
RunnerService: runner-v0.0.7-be-haphfbrhuudbxkzwpcuhn0hk.service  status is not running , please exeute 'systemctl restart runner-v0.0.7-be-haphfbrhuudbxkzwpcuhn0hk.service ' to restart runner service
RunnerService: runner-v0.0.7-be-mwkcqcv2hbwozrwmmczopzgm.service  status is active running
RunnerService: runner-v0.0.7-be-qdiyjifmn0bbkafobtfpkrst.service  status is not running , please exeute 'systemctl restart runner-v0.0.7-be-qdiyjifmn0bbkafobtfpkrst.service ' to restart runner service
RunnerService: runner-v0.0.7-be-qx6gaxojjjpzeblor692hrwk.service  status is active running
RunnerService: runner-v0.0.7-be-s5f7tixziynazefazqhls9ob.service  status is active running
RunnerService: runner-v0.0.7-be-uwdlmn516wd8emqebhgqwq6x.service  status is active running
RunnerService: runner-v0.0.7-be-xjmxrlbzrtksuafv4oyg8qtf.service  status is active running
Check runner service status finished.

手動排查

檢查系統 Linux 版本是否匹配。

通過命令lsb_release -a查看 Linux 發行版本資訊是否滿足條件。目前雲效統一 Runner 支援以下 Linux 發行版本：
- centos 6+ 及以上
- ubuntu 16.04+ 及以上
- Alibaba Cloud Linux 2/3

檢查 Runner service 狀態和日誌。

通過命令ls -al /etc/systemd/system | grep runner查看 Runner 服務名稱，通常 Runner 服務的名稱為 runner-{版本}-{租戶名稱}.service，如下所示。

[root@ecs-for-batch-deploy-1 ~]# lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-noarch
Distributor ID: CentOS
Description:    CentOS Linux release 8.2.2004 (Core)
Release:        8.2.2004
Codename:       Core
[root@ecs-for-batch-deploy-1 ~]# ls -al /etc/systemd/system | grep runner
-rw-r--r--    1 root root  550 10月 11 10:50 runner-v0.xxx                    t.service
[root@ecs-for-batch-deploy-1 ~]#

通過命令systemctl status runner-{版本}-{租戶名稱}.service查看 Runner 服務狀態，如果服務狀態為 active(running)，則服務正常，如下所示。

Release:        8.2.2004
Codename:       Core
[root@ecs-for-batch-deploy-1 ~]# ls -al /etc/systemd/system | gr
-rw-r--r--   1 root root  550 10月 11 10:50 runner-v0.0.5-be-bnyxxx  xxx yt.service
[root@ecs-for-batch-deploy-1 ~]# systemctl status runner-v0.0.5-be-bnyxxxpci0q6|06yt.service
● runner-v0.0.5-be-bnxxx yt.service - Aliyun yunxiao runner polling jo
     Loaded: loaded (/etc/systemd/system/runner-v0.xxx yt.service)
     Active: active (running) since Tue 2022-10-11 10.xxx ago
   Main PID: 1405243 (runner)
      Tasks: 10 (limit: 48875)
     Memory: 13.6M
     CGroup: /system.slice/runner-v0.0.5-be-brxxx06yt.service
             └─1405243 /usr/local/share/yunxiao-runner/v0.0.5/runner run --configPath=/r
10月 11 11:01:03 ecs-for-batch-deploy-1 runner[1405243]: INFO[2022-10-11T11:01:03+08:00]
10月 11 11:01:03 ecs-for-batch-deploy-1 runner[1405243]: INFO[2022-10-11T11:01:03+08:00]
10月 11 11:01:18 ecs-for-batch-deploy-1 runner[1405243]: WARN[2022-10-11T11:01:18+08:00]

另外，你也可以通過命令journalctl -u runner-{版本}-{租戶名稱}.service -a --no-pager --since '5 minutes ago' -f查看 Runner 執行日誌。

構建/部署運行中，無日誌或者主機狀態顯示為離線

檢查主機磁碟是否滿了df -hl ，如果磁碟已滿請嘗試清理後重試。
檢查 Runner service 狀態，如果狀態非 active(running)，通過以下步驟重啟 Runner service：
1. 擷取服務名稱，Runner 的服務名稱格式為：runner-{版本}-{租戶名稱}.service。
```
systemctl | grep "runner-v" | awk '{print $1}'
```
2. 使用擷取到的服務名稱替換以下命令中$SERVICE_NAME並執行，重啟 Runner service。
```
systemctl restart $SERVICE_NAME
```

檢查網路連通性是否正常。

通過systemctl status runner-{版本}-{租戶名稱}.service查詢 Runner service 狀態，並截取 service 進程參數 --configPath=***，如下所示：

[root@ecs-for-batch-deploy-1 ~]# systemctl status runner-v0.0.5-be-bnyxxx1llzpci0q606yt.service
● runner-v0.0.5-be-xxx_llzpci0q606yt.service - Aliyun yunxiao runner polling jobs to execute
   Loaded: loaded (/etc/systemd/system/runner-v0.0.5-be-bxxx pci0q606yt.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-10-11 10:50:56 CST; 41min ago
 Main PID: 1405243 (runner)
    Tasks: 10 (limit: 48875)
   Memory: 13.7M
   CGroup: /system.slice/runner-v0.0.5-be-bnyxxx q606yt.service
           └─1405243 /usr/local/share/yunxiao-runner/v0.0.5/runner run --configPath=/root/yunxiao/be-bnykv60t831llzpci0q606yt/runner/config
10月 11 11:31:46 ecs-for-batch-deploy-1 runner[1405243]: INFO[2022-10-11T11:31:46+08:00] [runner] no new job, skip.                    runner=9e7073c92b2fa8b35bc0efd3eec051f8
10月 11 11:31:46 ecs-for-batch-deploy-1 runner[1405243]: INFO[2022-10-11T11:31:46+08:00] counter current: 0, limit: 50
10月 11 11:32:01 ecs-for-batch-deploy-1 runner[1405243]: WARN[2022-10-11T11:32:01+08:00] Response status code: 204 body data:
10月 11 11:32:01 ecs-for-batch-deploy-1 runner[1405243]: INFO[2022-10-11T11:32:01+08:00] POST /api/v2/builds/request, time spent 15.07 s
10月 11 11:32:01 ecs-for-batch-deploy-1 runner[1405243]: INFO[2022-10-11T11:32:01+08:00] [runner] no new job, skip.                    runner=9e7073c92b2fa8b35bc0efd3eec051f8
10月 11 11:32:01 ecs-for-batch-deploy-1 runner[1405243]: INFO[2022-10-11T11:32:01+08:00] counter current: 0, limit: 50
10月 11 11:32:16 ecs-for-batch-deploy-1 runner[1405243]: WARN[2022-10-11T11:32:16+08:00] ...

查看 configPath 中URL cat {***}/config.yml | grep url。

[root@ecs-for-batch-deploy-1 config]# cat /root/yunxiao/be-bnykv60t83lllzpci0q606yt/runner/config/config.yml | grep url
url: https://xxx.com
[root@ecs-for-batch-deploy-1 config]#

通過以下命令判斷是否可以正常訪問。

# 注意替換下面命令中的url
curl '{url}/api/v2/runner/storage/latest?os=linux&arch=amd64'

跨地區部署失敗排查

如果部署涉及跨地區情境（如從中國香港部署到馬來西亞等），除了檢查 Runner 狀態和磁碟空間外，還需重點排查跨地區網路的連通性和穩定性。

在目標主機上測試到雲效服務端的網路延遲與丟包率：
```
# 測試網路延遲
ping {雲效服務端地址}
# 測試網路連通性和丟包
mtr {雲效服務端地址}
```
如網路品質不佳，建議最佳化網路鏈路（如使用專線或加速服務），或聯絡網路系統管理員排查跨地區公網鏈路問題。

多節點部署時如何避免服務啟動期間請求進入導致報錯？

雲效無法直接控制外部流量，建議採取以下措施實現部署期間的流量控制：

將主機分成兩個主機群組，在流水線中配置為順序部署（先部署第一組，完成後再部署第二組），確保始終有可用節點對外提供服務。
結合網關層面的流量攔截或使用應用型負載平衡（ALB）配置轉寄規則，在部署期間將流量從待部署節點上摘除，待服務啟動完成後再恢複轉寄，避免請求進入未完全啟動的服務。