Host monitoring for ECS with Managed Service for Prometheus - Managed Service for Prometheus

Managed Service for Prometheus supports collecting OS-level metrics from Elastic Compute Service (ECS) Linux or Windows hosts using node_exporter, gathering process-related monitoring data with process_exporter, and retrieving custom monitoring metrics that you write to files through the textfile collector.

Prerequisites

Managed Service for Prometheus is activated. Instance billing.
An ECS instance is created. Create and manage an ECS instance by using the ECS console (express version).
Resource Center is activated. Activate Resource Center.

Benefits of host monitoring

Host monitoring provides an automated observability solution for Alibaba Cloud ECS instances, covering host discovery, agent installation, metric scraping, and alerting.

Supports Alibaba Cloud ECS instances, self-managed data center servers, and third-party cloud servers. For ECS instances, the service auto-installs exporters and generates scrape configurations. A managed Prometheus agent handles metric scraping, storage, visualization, and alerting. For non-Alibaba Cloud hosts without automatic discovery support, manually install the Alibaba Cloud agent to push data.

Benefit	Description
Near real-time host discovery	Adaptability: Auto-discovery detects dynamic changes in cloud resources, ensuring all running instances are promptly monitored. Versatility: Supports multiple discovery types, including Kubernetes service discovery and integration with other cloud services.
Near real-time agent installation	Plug-and-play: Automated exporter installation lets the system recognize new nodes and collect metrics without manual intervention. Comprehensive monitoring: Includes node-exporter, process-exporter, GPU-exporter, and middleware exporters for full-stack performance tracking.
Near real-time metric scraping	Simplified configuration: Automated config generation reduces manual work and ensures accurate metric scraping across all nodes and services. Flexibility: Adjustable configurations scale to complex monitoring environments. A host is integrated within 30–60 seconds of creation. Scrape intervals are adjustable from 1 to 60 seconds.
Serverless agent	Centralized management: A managed Prometheus agent unifies data collection and simplifies monitoring architecture. The collection pipeline is transparent to users. High efficiency: Abstracted monitoring complexity reduces misconfiguration risk and improves data accuracy.
Intelligent metric labels	Automatically extracts tags, resource groups, and regions from ECS instances and injects them as metric labels. Custom labels can be added for business, environment, and data source identification.
Large-scale data collection and storage	Supports large-scale integration with a hybrid model of dedicated and shared resources. Resources scale dynamically based on the number of integrated hosts. The storage system handles massive metric volumes with high-performance querying.
Comprehensive upstream and downstream monitoring data	End-to-end observability requires integrating monitoring data across dimensions to reflect the health of the entire application and service ecosystem. Covers the full stack from hardware to the application layer, including hosts, networks, dependent services, and external services like RDMA networks, OSS, and Redis.
Process-level monitoring	Tracks process performance and resource utilization on the OS, showing the health of applications running on the server. Captures CPU usage, memory, disk I/O, start time, open file handles, and thread count per process. Near-real-time feedback enables quick issue identification. Helps identify processes causing performance degradation, such as memory leaks, high CPU usage, or resource contention.
Expert-level Grafana dashboards included by default	Includes expert-curated Grafana dashboards: ECS Overview, ECS Detail, GPU Overview, GPU Detail, and Node Process. One-click, out-of-the-box host monitoring experience.

Step 1: Integrate host monitoring data

Log on to the ARMS console. In the left-side navigation pane, click Integration Center.
On the Integration Center page, click Infrastructure in the left-side navigation pane, and then click Host Monitoring.
Note
- Managed Service for Prometheus uses Resource Center to discover VPCs and ECS instances. If Resource Center is not activated, the integration process guides you to activate it. Activate Resource Center.
- Activating Resource Center is an asynchronous operation. If the status does not update, wait 10 to 20 seconds and click Redetect.

In the panel that appears, select the target VPC and configure the Configuration Information as described in the following table.

Parameter	Description
Policy name	Optional. Name for this integration.
Notes	This feature collects agentless metrics for ECS from Cloud Monitor. You can also install exporters, such as Node-Exporter, Process-Exporter, or Windows-Exporter, to collect host agent metrics. Installing an agent on a host in a production environment is considered a production change. Submit a change request according to your organization's standards. If cross-region access is enabled, you cannot use an agent to collect node, process, or Windows metrics. Exporter installation is not currently supported on LifseaOS. Instances that run LifseaOS are skipped automatically. Resource usage: The agent runs as a systemd service with GOMAXPROCS=1, a CPU limit of 30%, and a memory limit of 256 MB. Resource usage is low under normal conditions. The collection overhead increases with the number of processes on the host. Risk warning: Enabling the Collect Process PSS Memory Metrics feature reads the smaps file for each process and briefly holds the mmap_lock. This can introduce latency glitches for processes that use large amounts of memory or perform frequent mmap operations. If you do not need Proportional Set Size (PSS) metrics, disable this feature.
Node-exporter installation mode	Automatic Installation (Recommended): Installs node-exporter on selected ECS instances automatically. No manual steps required. Self-installation: Install node-exporter yourself.
Host discovery mode	Stain label selection: Blacklist mechanism. Instances with matching labels are excluded. By default, container monitoring service nodes are not scraped. Unconditional: Install exporters and collect monitoring metrics from all ECS hosts in the current VPC. Tag selection: Whitelist mechanism. Only instances with matching tags are integrated. IP CIDR selection: Integrates instances whose IP falls within the specified CIDR block. Enter the VPC CIDR block to select all instances in the VPC. Instance ID: Specify instance IDs, separated by commas (,).




ECS stain label	Each stain label is a key-value pair. You can set multiple labels.
Collect TextFile	Whether to scrape Prometheus metrics from a specified file.
Collect process status metrics	Collects process monitoring data by default.
Collect Process PSS Memory Metrics	When enabled, reads the `smaps` file for each process to collect PSS (proportional set size resident memory) and related metrics.
Node-Exporter Service Port	The default port is 9100. If both arms-prometheus and ack-prometheus-operator are installed in the same ACK cluster, both node-exporter sets use port 9100 by default, which may cause a port conflict and result in service errors.
Metric scrape interval (seconds)	Scrape interval. Default: 15 seconds.
Automatically configure security groups	Enabled by default.
Custom ECS tag injection	Specify an ECS tag key. The system injects the tag key-value pair into Prometheus metrics as labels.

Click OK. Wait 1 to 2 minutes for the integration to complete.

Note

If no data appears on the dashboard after integration, verify that the ECS security group inbound rules allow the 100.64.0.0/10 and 192.168.0.0/18 CIDR blocks to access ports 9100 and 9256 (default ports for node-exporter and process-exporter). View security group rules. If you use different ports, adjust the rules accordingly.

Troubleshoot and resolve node-exporter port conflicts

When arms-prometheus and ack-prometheus-operator are both deployed in the same ACK cluster, their respective node-exporter DaemonSets compete for hostPort 9100. This prevents one set of node-exporter pods from starting. To resolve this, change the port of one node-exporter DaemonSet to 9101.

Run the following commands to confirm which DaemonSet's node-exporter pods have failed to schedule:
```
kubectl get pods -A | grep node-exporter
kubectl describe pod <pod-name> -n <namespace>
```

Edit the conflicting DaemonSet:

kubectl edit daemonset <daemonset-name> -n <namespace>

In the DaemonSet YAML, locate the --secure-listen-address argument and change the port from 9100 to 9101. Also update the containerPort and hostPort values to 9101:
```
args:
- --secure-listen-address=0.0.0.0:9101
...
ports:
- containerPort: 9101
  hostPort: 9101
  name: https
```
Save and exit. Verify that the node-exporter pods restart and are running on the new port.
Note
If the component was originally installed using Helm, a subsequent helm upgrade may overwrite your changes. Before upgrading, back up the modified configuration or lock the chart version.

Configure a ServiceMonitor for custom monitoring

After resolving the port conflict, configure a ServiceMonitor in the ARMS console to scrape metrics from the node-exporter instance on the new port.

Log on to the ARMS console.
In the left-side navigation pane, choose Prometheus Monitoring > Service Discovery.
Click Create ServiceMonitor. In the creation panel, select YAML Mode.

Enter a ServiceMonitor configuration that targets your node-exporter service on port 9101. The following is an example:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: node-exporter-custom
  namespace: <namespace>
spec:
  endpoints:
  - port: https
    interval: 15s
    scheme: https
    tlsConfig:
      insecureSkipVerify: true
  namespaceSelector:
    matchNames:
    - <namespace>
  selector:
    matchLabels:
      app.kubernetes.io/name: node-exporter

Replace <namespace> with the namespace where your node-exporter is deployed.

Click OK. The ServiceMonitor takes effect within a few minutes.

Step 2: View monitoring dashboards

In the left-side navigation pane, click Integration Management.
On the Integration Management page, click the Integrated Environments tab, and select ECS Environment.
In the ECS Environment list, click the name of the target environment to open its details page.
On the Component Management tab, in the Addon Type section, click Dashboard to view the built-in Grafana dashboards.

Step 3: Configure alerts

Log on to the Managed Service for Prometheus console. In the left-side navigation pane, click Integration Management.
On the Integration Management page, click the Integrated Environments tab, and select ECS Environment.
In the ECS Environment list, click the name of the target environment to open its details page.
On the Component Management tab, in the Addon Type section, click Alarm Rules to view the built-in alert rules.

Note

Built-in alert rules generate events but do not send notifications. To send notifications, click Edit to configure a notification method. You can also customize thresholds, duration, and content. Create a Prometheus alert rule.
In Simple Mode, you can set the notification recipients, notification period, and repeat policy.

Managed Service for Prometheus:ECS observability