When your service mesh encounters configuration errors, certificate issues, or data-plane synchronization failures, control-plane logs are the primary source for diagnosis. Service Mesh (ASM) Log Center allows you to view control-plane logs and data-plane logs in the ASM console. After you enable log collection, you can use Log Center to view control-plane logs, configure log-based alert rules, and view data-plane logs and related dashboards. This topic describes how to enable control-plane log collection and log-based alerting.
This topic applies to ASM instances of version 1.17.2.35 or later. For earlier versions, see Enable control-plane log collection and log-based alerting in an ASM instance of a version earlier than 1.17.2.35.
Prerequisites
Before you begin, make sure that you have:
Simple Log Service activated for your Alibaba Cloud account. See "Step 1: Activate Simple Log Service" in Getting Started
The logtail-ds component installed in the data-plane cluster. See "Step 1: Install Logtail" in Collect log data from containers by using Simple Log Service
A cluster added to an ASM instance of version 1.17.2.35 or later. See Add a cluster to an ASM instance and Update an ASM instance
ASM does not charge for log collection. However, Simple Log Service bills separately in pay-by-feature or pay-by-ingested-data mode. For details, see Billing overview.
Enable control-plane log collection
Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of your ASM instance. In the left-side navigation pane, choose Observability Management Center > Log Center.
On the Log Center page, click the Control-Plane Logs tab. Select New project or Use existed project, configure the parameters, and then click Enable Control-Plane Log Collection.
In the Submit confirmation dialog, click OK.
After log collection is enabled:
View control-plane logs in the ASM console or Simple Log Service console.
Configure log-based alerting to detect issues early.
To disable this feature, click Disable Control-Plane Log Collection in the upper-right corner of the Control-Plane Logs tab, and then click OK to confirm.
View control-plane logs
After you enable log collection, control-plane component logs stream into Simple Log Service. View them in the ASM console or the Simple Log Service console.
Logs may not appear immediately after you enable collection. The system needs a short time to set up the log distribution task.
View logs in the ASM console
Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of your ASM instance. In the left-side navigation pane, choose Observability Management Center > Log Center.
On the Log Center page, click the Control-Plane Logs tab to view log information.
View logs in the Simple Log Service console
Log on to the Simple Log Service console.
In the Projects section, click the name of the Simple Log Service project used for your cluster.
On the Logstores page, click the desired Logstore to view log information.
Configure log-based alerting
Set up alert rules to get notified about control-plane issues such as configuration push failures, configuration risks, and certificate expiration.
Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of your ASM instance. In the left-side navigation pane, choose Observability Management Center > Log Center.
On the Log Center page, click the Control-Plane Logs tab. In the upper-left corner, click Alert Setting.
In the Control-Plane Alert Setting dialog box, find the alert policy you want, select an action policy, and click Enable Alert. In the Note message, click OK.
An action policy defines how notifications are sent when an alert fires. Two types are available:
| Action policy type | Description |
|---|---|
| ASM Built-in Action Strategy | A default policy with no notification channels preconfigured. Click the |
| Custom action strategy | A policy you define with your own notification channels. |
Add alert contacts
You can configure built-in action policies for gateways, alert contacts, and notification templates in Simple Log Service.
Log on to the Simple Log Service console.
In the Projects section, click the name of the desired project. In the left-side navigation pane, click Alerts.
On the Alert Center page, choose Notification Objects > User Group Management.
On the User Group Management tab, find sls.app.asm.builtin and click Edit in the Actions column.
In the Edit User Group dialog box, select the members to add, click the
icon to add them to the user group, and then click OK.
Verify alert notification settings
Trigger a test alert to confirm that notifications are delivered correctly.
This example triggers only one alert type. For related error messages, see Configuration push failure alerts.
Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of your ASM instance. In the left-side navigation pane, choose ASM Gateways > Gateway. On the page that appears, click Create from YAML.
On the Create page, select a namespace and a template, paste the following YAML, and click Create. This example deploys an Istio gateway in the default namespace with a non-existent TLS credential to intentionally trigger an error:
apiVersion: networking.istio.io/v1beta1 kind: Gateway metadata: name: gateway-test namespace: default spec: selector: istio: ingressgateway servers: - hosts: - '*console.aliyun.com' port: name: https number: 443 protocol: HTTPS tls: credentialName: not-existing-credential mode: SIMPLECheck the alert in the ASM console:
In the left-side navigation pane, choose Observability Management Center > Log Center.
On the Control-Plane Logs tab, search for
ACK ERRORto view the alert information.
If you configured email notifications, check your inbox for the alert email.
Troubleshoot alerts
Potential configuration risk alerts
ASM raises these alerts when it detects configurations that may produce unexpected results. Check the Mesh Diagnosis page and follow the instructions to correct the configurations. See Diagnose ASM instances.
Incorrect configuration alerts
ASM raises these alerts when it detects configurations likely to cause unexpected behavior. Check the Mesh Diagnosis page and correct the configurations promptly. See Diagnose ASM instances.
Configuration push failure alerts
The following table lists common errors when the control plane fails to push configurations to the data plane. If your error is not listed, submit a ticket.
| Error message | Fix |
|---|---|
Internal:Error adding/updating listener(s) 0.0.0.0_443: Failed to load certificate chain from <inline>, only P-256 ECDSA certificates are supported | The data-plane cluster does not support the configured certificate. Replace it with a P-256 ECDSA certificate. See Use an ingress gateway to enable HTTPS. |
Internal:Error adding/updating listener(s) 0.0.0.0_443: Invalid path: **** | The certificate path is invalid or the certificate does not exist. Verify that the certificate mount path matches the path in the gateway configuration. See Use an ingress gateway to enable HTTPS. |
Internal:Error adding/updating listener(s) 0.0.0.0_xx: duplicate listener 0.0.0.0_xx found | The gateway has duplicate listening ports. Remove the duplicate port from the gateway configuration. |
Internal:Error adding/updating listener(s) 192.168.33.189_15021: Didn't find a registered implementation for name: '***' | The name *** referenced based on the 15021 listener patch by using EnvoyFilter cannot be found in sidecar proxies or ingress gateway services. Remove the invalid reference. |
Internal:Error adding/updating listener(s) 0.0.0.0_80: V2 (and AUTO) xDS transport protocol versions are deprecated in grpc_service *** | The xDS v2 protocol on the data plane will be deprecated soon. This is usually because the version of sidecar proxies on the data plane does not match that on the control plane. Delete existing pods to trigger re-creation. Sidecar proxies of the latest version are automatically injected into the recreated pods. |
Certificate expiration alerts
| Alert policy | Action |
|---|---|
| Certificate Management has an expired certificate, which may cause the gateway to fail to process TLS traffic. | Replace the expired certificate immediately to restore TLS traffic processing on the gateway. |
| Certificate Management has a certificate that is about to expire. Please update the certificate in time. | Renew or replace the certificate before it expires to avoid TLS traffic disruption. |
Change the control-plane log project
Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of your ASM instance. In the left-side navigation pane, choose Observability Management Center > Log Center.
On the Control-Plane Logs tab, click Change Log Project. In the Change Log Project dialog box, update the settings and click Submit.