Import data from AWS or Tencent Cloud to Alibaba Cloud Monitor - Cloud Monitor

You can create metric import tasks for Amazon Web Services (AWS) or Tencent Cloud to import monitoring data into Cloud Monitor. After the data is imported, you can create enterprise dashboards and monitoring charts to view metrics for these cloud services.

Prerequisites

Ensure Cloud Monitor can communicate with the hosts running your AWS or Tencent Cloud services.
Ensure you have created a namespace. For more information, see Create a namespace.

Important

You can install the Cloud Monitor agent and exporters only by using an administrator account (the root user for Linux or the Administrator user for Windows). However, using an administrator account may introduce certain risks, such as system instability or data security issues. Proceed with caution.

Create a metric import task

Install the Cloud Monitor agent on the host running your AWS or Tencent Cloud service.
For more information, see Install and uninstall the Cloud Monitor agent.
Install an exporter on the host where the Tencent Cloud service runs.
This section demonstrates how to install the exporter for a Tencent Cloud service on a Linux server.
1. Log on to the host where the Tencent Cloud service runs as the root user.
2. Run the following command to download the exporter installation package.
  wget https://cms-prometheus.oss-cn-hangzhou.aliyuncs.com/tencentyun/qcloud-exporter-linux-amd64.tar.gz
3. Run the following command to decompress the qcloud-exporter-linux-amd64.tar.gz file.
  tar zxvf qcloud-exporter-linux-amd64.tar.gz
4. Run the following commands to view all files in the qcloud-exporter-linux-amd64 directory.
  cd qcloud-exporter-linux-amd64
  ll qcloud-exporter-linux-amd64
```
qcloud-cdb-product.yml   qcloud-clb-product.yml     qcloud-cvm-product.yml  qcloud_exporter                     qcloud-redis-cluster-product.yml
qcloud-cdn-product.yml   qcloud-cmongo-product.yml  qcloud-dc-product.yml   qcloud-exporter-linux-amd64.tar.gz  qcloud-redis-product.yml
qcloud-clb7-product.yml  qcloud-cos-product.yml     qcloud-dcx-product.yml  qcloud-nat-product.yml
```
  Files with the .yml suffix are configuration files for Tencent Cloud services. You can modify the target configuration file directly, or copy a configuration file and modify its configuration.
5. Configure the qcloud-redis-product.yml file for the Tencent Cloud service.
  Note
  This example uses the qcloud-redis-product.yml file.
  1. Run the following command to edit the qcloud-redis-product.yml file.
    vi qcloud-redis-product.yml
  2. Configure the qcloud-redis-product.yml file.
```
       credential:
         access_key: "access_key" # The AccessKey ID of your Tencent Cloud account.
         secret_key: "secret_key" # The AccessKey secret of your Tencent Cloud account.
         region: "region" # The region where the Redis instance runs.
     
       rate_limit: 15   # The API rate limit for Tencent Cloud's monitoring service. The maximum rate is 20 calls per second or 1,200 calls per minute.
       products:
         - namespace: QCE/REDIS    # The namespace of Redis.
          all_metrics: true # true: Monitors all Redis metrics.
          all_instances: true # true: Monitors all Redis instances.
          #only_include_metrics: []
          #only_include_instances: [crs-xxxxxxxx]
          #extra_labels: [InstanceName]
          #statistics_types: [last]
          #period_seconds: 60
          #metric_name_type: 2
```
  3. Press the Esc key, enter :wq, and then press the Enter key to save and exit the qcloud-redis-product.yml file.
6. Run the following command to install the exporter and start the service.
  qcloud_exporter --config.file "qcloud-redis-product.yml"

Install an exporter on the host where the AWS service runs.

This section demonstrates how to install the exporter for an AWS service on a Linux server.

Log on to the host where the AWS service runs as the root user.
Run the following commands to set the environment variables.
AWS_ACCESS_KEY_ID="AKIAZUF26NQPE****"
AWS_SECRET_ACCESS_KEY="8mSpf+XRHBuJEi4tjJN4YU6h3LRS****"

Create a configuration file named example.yml for the AWS service.

Run the following command to create an empty file named example.yml.
touch example.yml
Run the following command to edit the example.yml file.
vi example.yml

Configure the example.yml file.

region: us-east-2
metrics:
- aws_namespace: AWS/EC2
  aws_metric_name: CPUUtilization
  aws_dimensions: [InstanceId]
  aws_dimenstion_select:
          InstanceID: [i-03c0a044201c5****,i-04e5e3be6f194****]
  aws_statistics: [Maximum]
- aws_namespace: AWS/EC2
  aws_metric_name: DiskReadOps
  aws_dimensions: [InstanceId]
  aws_dimenstion_select:
          InstanceID: [i-03c0a044201c5****,i-04e5e3be6f194****]
  aws_statistics: [Maximum]
- aws_namespace: AWS/EC2
  aws_metric_name: DiskWriteOps
  aws_dimensions: [InstanceId]
  aws_dimenstion_select:
          InstanceID: [i-03c0a044201c5****,i-04e5e3be6f194****]
  aws_statistics: [Maximum]
- aws_namespace: AWS/EC2
  aws_metric_name: DiskReadBytes
  aws_dimensions: [InstanceId]
  aws_dimenstion_select:
          InstanceID: [i-03c0a044201c5****,i-04e5e3be6f194****]
  aws_statistics: [Maximum]

Parameter	Description
`region`	The region where the AWS service runs.
`aws_namespace`	The namespace of the AWS service.
`aws_metric_name`	The metric for the AWS service.
`aws_dimensions`	The dimension used to query metric data for the AWS service. Example: `InstanceId`.
`InstanceID`	The ID of a specific AWS instance. This key is used under `aws_dimension_select` to filter instances.
`aws_statistics`	The statistical method for the AWS service metric.

Press the Esc key, enter :wq, and then press the Enter key to save and exit the example.yml file.

Run the following command to download the exporter installation package.
wget https://cms-prometheus.oss-cn-hangzhou.aliyuncs.com/aws-cloudwatch/cloudwatch_exporter-0.9.0-jar-with-dependencies.jar
Run the following command to create the target directory.
mkdir target
Run the following command to move the cloudwatch_exporter-0.9.0-jar-with-dependencies.jar installation package to the target directory.
mv cloudwatch_exporter-0.9.0-jar-with-dependencies.jar target

Run the following command to install the exporter and start the service.

java -jar target/cloudwatch_exporter-0.9.0-jar-with-dependencies.jar 9106 example.yml

The following output indicates the exporter was installed successfully.

2021-06-11 03:24:55.743:INFO::main: Logging initialized @1775ms to org.eclipse.jetty.util.log.StdErrLog
2021-06-11 03:24:55.844:INFO:oejs.Server:main: jetty-9.4.z-SNAPSHOT; built: 2019-02-15T16:53:49.381Z; git: eb70b240169fcf1abbd86af36482d1c49826****; jvm 1.8.0_282-b08
2021-06-11 03:24:55.906:INFO:oejsh.ContextHandler:main: Started o.e.j.s.ServletContextHandler@3de8f619{/,null,AVAILABLE}
2021-06-11 03:24:55.964:INFO:oejs.AbstractConnector:main: Started ServerConnector@3feb2dda{HTTP/1.1,[http/1.1]}{0.0.0.0:9106}
2021-06-11 03:24:55.965:INFO:oejs.Server:main: Started @1997ms

Create the metric import task.

Log on to the Cloud Monitor console.
In the left-side navigation pane, choose Hybrid Cloud Monitoring > Namespace.
Find the target namespace and click the number of accounts, cloud services, metrics, or tasks in the Metric data column.
Note
This number is the total count of metric import tasks from sources such as on-premises data centers, AWS, Tencent Cloud, Prometheus, on-premises logs, and Simple Log Service (SLS).
On the Metric management panel, click the other IDC tab, and then click Metric data.

On the Basic task information page of the Metric data dialog box, configure the task settings, and then click Next.

Parameter	Description
Select middleware to be monitored	Select the middleware to monitor.
Namespace	Select the namespace to store the imported data.
Task name	The name of the metric import task.
Task description	The description of the metric import task.
Select host	The host of the target cloud provider. Note The Cloud Monitor agent must be installed on these hosts.
Region	The region where the host is located. Note Click Advanced options to configure this parameter.
Network type	The network type of the host. Valid values: VPC classic network Note Click Advanced options to configure this parameter.

On the Data access check page, verify that the latest versions of the Cloud Monitor agent and the exporter are installed, and then click Next.
On the Task details page, enter the URL of the collection target, select the data collection interval and timeout period, set tags for the task, and then click Next.
On the Dashboard page, click Close.

Modify a metric import task

Log on to the Cloud Monitor console.
In the left-side navigation pane, choose Hybrid Cloud Monitoring > Namespace.
Find the target namespace and click the number of accounts, cloud services, metrics, or tasks in the Metric data column.
Note
This number is the total count of metric import tasks from sources such as on-premises data centers, AWS, Tencent Cloud, Prometheus, on-premises logs, and Simple Log Service (SLS).
On the Metric management panel, click the other IDC tab. Find the task to modify and click Modify in the Actions column.
On the Basic task information page of the Metric data dialog box, modify the task settings, and then click Next.
On the Data access check page, verify that the latest versions of the Cloud Monitor agent and the exporter are installed, and then click Next.
On the Task details page, modify the URL of the collection target, the data collection interval, the timeout period, or the tags for the task, and then click Next.
On the Dashboard page, click Close.

Delete a metric import task

Log on to the Cloud Monitor console.
In the left-side navigation pane, choose Hybrid Cloud Monitoring > Namespace.
Find the target namespace and click the number of accounts, cloud services, metrics, or tasks in the Metric data column.
Note
This number is the total count of metric import tasks from sources such as on-premises data centers, AWS, Tencent Cloud, Prometheus, on-premises logs, and Simple Log Service (SLS).
On the Metric management panel, click the other IDC tab. Find the task to delete and click Delete in the Actions column.
In the Confirm deletion dialog box, click Confirm.