All Products
Search
Document Center

Simple Log Service:Import data from CloudMonitor to Log Service

Last Updated:Dec 12, 2023

This topic describes how to import data from CloudMonitor to Log Service. After you import data from CloudMonitor to Log Service, you can query, analyze, and transform the data.

Prerequisites

A project and a Logstore are created. For more information, see Create a project and Create a Logstore.

Background information

Log Service allows you to import data from CloudMonitor to a Logstore as log data or to a Metricstore as metric data. For more information, see Log or Metric. Most data from CloudMonitor is statistical data. We recommend that you store the data imported from CloudMonitor in a Metricstore. For more information, see Import metric data from CloudMonitor to Simple Log Service.

Create a data import configuration

  1. Log on to the Simple Log Service console.

  2. On the Data Import tab in the Import Data section, click CloudMonitor - Data Import.

  3. Select the project and Logstore. Then, click Next.
  4. Configure the data import settings and click Next.

    Parameter

    Description

    Configuration Name

    The name of the data import configuration.

    Import All

    Select Yes to import all metric data from CloudMonitor to Simple Log Service. For more information, see Predefined metrics.

    NameSpace

    If you select No for the Import All parameter, select the metrics that you want to import from the NameSpace drop-down list.

    AccessKey ID

    The AccessKey ID of your Alibaba Cloud account.

    We recommend that you use the AccessKey pair of a Resource Access Management (RAM) user that is granted the AliyunCloudMonitorReadOnlyAccess permission.

    AccesssKey Secret

    The AccessKey secret of your Alibaba Cloud account.

    Use Hybrid Cloud Monitoring API

    Simple Log Service uses only Hybrid Cloud Monitoring API to pull data from CloudMonitor.

    Hybrid Cloud Monitoring allows you to pull the latest data in real time and supports higher queries per second (QPS). You are charged traffic fees when you use Hybrid Cloud Monitoring. For more information, see Hybrid Cloud Monitoring.

View a data import configuration

After you create a data import configuration, you can view the configuration details and related statistical reports in the Log Service console.

  1. In the Projects section, click the project that you want to manage.

  2. Choose Log Storage > Logstores. On the Logstores tab, find the Logstore to which the data import configuration belongs and choose Data Import > Data Import.

  3. Click the name of the import configuration.

  4. On the Import Configuration Overview page, view the basic information and statistical reports of the data import configuration.

Related operations

On the Import Configuration Overview page, you can perform the following operations:

  • Modify the data import configuration.

    To modify the data import configuration, click Modify Settings. For more information, see Create a data import configuration.

  • Delete the data import configuration.

    To delete the data import configuration, click Delete Configuration.

    Important

    After a data import configuration is deleted, it cannot be restored. Proceed with caution.

  • Stop the data import job.

    To stop the data import job, click Stop.

FAQ

Issue

Possible cause

Solution

Some metrics cannot be found.

If Hybrid Cloud Monitoring API is used to pull data, some virtual metrics cannot be pulled.

If some metrics cannot be found after you turn on Use Hybrid Cloud Monitoring API but you can query the metrics by calling the DescribeMetricList operation, the cause is that the metrics are not supported by Hybrid Cloud Monitoring API. To resolve this issue, submit a ticket to CloudMonitor technical support.

Long data latency exists.

  • Latency exists when metrics are collected from data sources.

  • The API quota of CloudMonitor is exhausted.

  • The network is unstable.

For more information, see Limits.

  • For metrics in the acs_global_acceleration namespace, long latency exists when the metrics are reported to CloudMonitor. As a result, a latency of 10 to 15 minutes exists when the metrics are imported to Log Service.

  • If Use Hybrid Cloud Monitoring API is turned off and the Data Processing Insight dashboard displays API quota-related errors, we recommend that you turn on Use Hybrid Cloud Monitoring API in the data import configuration.

  • Reduce the number of namespaces from which data is imported or submit a ticket to CloudMonitor technical support to request an increase in the API quota.

  • Use a Log Service project in regions where CloudMonitor is deployed, such as the China (Hangzhou) and China (Shanghai) regions.

Error handling

Item

Description

Error handling method

After an error occurs, the import job retries at a regular interval until the job is recovered. You can view the number of errors on the Data Processing Insight dashboard.

If the import job is recovered within 3 hours, the job continues to pull data from the position at which the job stops pulling data due to the error. Otherwise, the import job continues to pull data from the latest position regardless of the stop position.