Simple Log Service:Import Elasticsearch/OpenSearch data

Version guide

This feature supports Elasticsearch 6.3 and later, and OpenSearch 1.0.0 and later.

Create a data import configuration

1. Log on to the Simple Log Service console.

2. In the Access Data area, on the Data Import tab, select ES/OpenSearch - Data Import.

   

3. Select the project and logstore. Then, click Next.

4. Set the import configuration.

   1. In the Import Configuration step, set the following parameters.

      Parameter	Description
      Task Name	The unique name of the SLS task.
      Display Name	The display name of the task.
      Task Description	The description of the import task.
      Service Instance URL	The URL of the Elasticsearch or OpenSearch server. The format is http://host:port/. You can specify multiple URLs. Separate them with commas (,), for example, http://host1:port1/,http://host2:port2/. The service port for Elasticsearch or OpenSearch servers is typically 9200. Important If you set the VPC Instance ID, you must set host to the IPv4 address of the corresponding ECS instance.
      Index List	The indexes to import. Separate multiple indexes with commas (,), for example, index1,index2,index3.
      Username	The username for the Elasticsearch or OpenSearch cluster. This parameter is required only if the cluster requires user authentication.
      Password	The password for the Elasticsearch or OpenSearch user.
      Time Field	The name of the column in the Elasticsearch or OpenSearch index that represents time. This is used to specify the log time. If you do not specify a time field, Simple Log Service uses the system time of the data import by default. Important If you want to perform an incremental import, you must set the Time Field.
      Time Field Format	The time format used to parse the value of the time field. The Java SimpleDateFormat syntax is supported, for example, yyyy-MM-dd HH:mm:ss. For more information about the time format syntax, see Class SimpleDateFormat. For common time formats, see Time formats. The epoch format is also supported. Valid values are epoch, epochMillis, epochMacro, and epochNano. Important Java SimpleDateFormat does not support Unix timestamps. If you want to use Unix timestamps, you must set the Time Field Format to the epoch format.
      Time Field Time Zone	The time zone of the time field. When the Time Field Format is set to epoch format, you do not need to set the time zone.
      Search Statement	The search statement to filter data. The statement must follow the Elasticsearch or OpenSearch query_string format, for example, gender:male and city:Shanghai. For more information, see Query string query.
      Import Mode	The mode for data import. Import Historical Data Only: The import task automatically stops after the data is imported. Automatically Import Incremental Data: The import task runs continuously. Important If you select Auto Import New Data, you must set the Time Field.
      Start Time	Data is imported to Simple Log Service only if the value of the time field is greater than or equal to the specified start time. Important This configuration is valid only after the Time Field is set.
      End Time	Data is imported to Simple Log Service only if the value of the time field is less than or equal to the specified end time. Important This configuration is effective only if the Time Field is set and the Import Mode is set to Import Historical Data Only.
      Maximum Data Latency (Seconds)	The maximum latency from when data is generated to when it is written to Elasticsearch or OpenSearch. Important If the value you set is smaller than the actual latency, some data may fail to be imported from Elasticsearch or OpenSearch to Simple Log Service. This configuration is effective only after you set the Time Field and set the Import Mode to Auto Import New Data.
      Incremental Data Check Interval (Seconds)	The interval at which to check for new data in Elasticsearch or OpenSearch. Default value: 300 seconds. Minimum value: 60 seconds.
      VPC Instance ID	If your cluster is an Alibaba Cloud Elasticsearch or OpenSearch cluster in a VPC, or a self-managed cluster on an ECS instance, set the VPC Instance ID. This allows Simple Log Service to read data from the cluster over the Alibaba Cloud private network for better security and stability. Important The Elasticsearch or OpenSearch cluster must allow access from the 100.104.0.0/16 CIDR block.

   2. Click Preview to view the import results.

   3. After you confirm the information is correct, click Next.

5. Complete the Preview Data and Create Index steps, and then click Next. By default, full-text indexing is enabled in Simple Log Service. You can also manually create field indexes based on the collected logs, or click Auto Generate Index to have Simple Log Service automatically generate field indexes. For more information, see Create Index.

   Important

   A full-text index lets you query all fields in a log. Field indexes allow you to query specific fields and can reduce index traffic. You must create field indexes to analyze fields with SELECT statements.

6. Click Query Log. Then, you are redirected to the query and analysis page of your Logstore.

   You must wait approximately 1 minute for the indexes to take effect. Then, you can view the collected logs on the Raw Logs tab. For more information about how to query and analyze logs, see Get started with log query and analysis.

View the import configuration

After you create an import configuration, you can view the configuration and its statistical reports in the console.

1. Click the destination Project.

2. For the destination Logstore, choose Data Ingestion > Data Import, and then click the name of the import configuration.

3. On the Import Configuration Overview page, you can view the basic information and statistical reports for the data import configuration.

Related operations

• Delete an import configuration

  On the Import Configuration Overview page, you can click Delete Configuration to delete the configuration.

  Warning

  This operation cannot be undone. Proceed with caution.

• Stop and restart an import task

  After you create a data import configuration, Simple Log Service creates an import task. On the Import Configuration Overview page, you can stop the import task by clicking Stop and then restart it.

  Important

  After an import task is stopped, its state is retained for a maximum of 24 hours. If you do not restart the task within 24 hours, the task becomes unavailable. If you restart the task after 24 hours, errors will occur when the task runs again.

FAQ

Error handling mechanism