Simple Log Service:Import Elasticsearch/OpenSearch data

Supported versions

Only Elasticsearch 6.3 or later and OpenSearch 1.0.0 or later are supported.

Create a data import configuration

1. Log on to the Simple Log Service console.

2. In the Import Data section, click the Data Import tab, and then select ES/OpenSearch - Data Import.

3. Select the destination project and Logstore, and then click Next.

4. Configure the import settings.

   1. In the Import Configuration step, configure the following parameters.

      Parameter	Description
      Task Name	A unique name for the import job.
      Display Name	The display name of the job.
      Job Description	The description of the import job.
      Service Instance URL	The URL of the Elasticsearch/OpenSearch server. Use the format http://host:port/. You can specify multiple URLs separated by commas (,), for example, http://host1:port1/,http://host2:port2/. The default service port for Elasticsearch/OpenSearch is 9200. Important If you set VPC ID, you must set host to the IPv4 address of the corresponding ECS instance.
      Index List	The indexes to import. Separate multiple index names with commas (,), for example, index1,index2,index3.
      User Name	The username to access the Elasticsearch/OpenSearch cluster. Required only if the cluster uses authentication.
      User Password	The password to access the Elasticsearch/OpenSearch cluster.
      Time Field	The field in the source index that represents time, used as the log time. If you do not specify a time field, SLS uses the system time when the data is imported as the log time. Important To perform an incremental import, you must set Time Field.
      Time Field Format	The time format used to parse the value of the time field. Supports time formats that follow Java's SimpleDateFormat syntax, such as yyyy-MM-dd HH:mm:ss. For more information about the syntax, see Class SimpleDateFormat. For common time formats, see Time formats. Supports epoch formats. Valid values: epoch, epochMillis, epochMacro, and epochNano. Important Java's SimpleDateFormat does not support UNIX timestamps. If you want to use UNIX timestamps, you must set Time Field Format to an epoch format.
      Time Zone	The time zone of the time field. You do not need to set a time zone if Time Field Format is set to an epoch format.
      Query	The search query used to filter data. The query must follow the Elasticsearch/OpenSearch query_string format. Example: gender:male and city:Shanghai. For more information, see Query string query.
      Import Method	The method used to import data. Import Only Historical Data: The job stops automatically after importing the data. Automatically Import Incremental Data: The import job runs continuously. Important If you select Automatically Import Incremental Data, you must set Time Field.
      Start At	The start time for the import. SLS imports data only if its timestamp is on or after this time. Important This parameter takes effect only if Time Field is set.
      End Time	The end time for the import. SLS imports data only if its timestamp is on or before this time. Important This parameter takes effect only if Time Field is set and Import Method is set to Import Only Historical Data.
      Maximum Latency in Seconds	The maximum allowed delay, in seconds, between data generation and its ingestion into the source cluster. Important Setting this value lower than the actual latency may cause data loss. This parameter takes effect only if Time Field is set and Import Method is set to Automatically Import Incremental Data.
      Incremental Data Check Interval (Seconds)	The interval in seconds at which to check for new data in Elasticsearch/OpenSearch. Default value: 300. Minimum value: 60.
      VPC ID	If your source resides in a VPC (either an Alibaba Cloud Elasticsearch/OpenSearch cluster or a self-hosted cluster on an ECS instance), set this parameter so that SLS reads data over the internal network for better security and stability. Important The Elasticsearch/OpenSearch cluster must allow access from the 100.104.0.0/16 CIDR block.

   2. Click Preview to preview the import results.

   3. After you confirm the settings, click Next.

5. Configure the settings for Preview Data and Create Index, and then click Next. Simple Log Service enables the full-text index by default. You can also manually create field indexes based on the collected logs, or click Automatic Index Generation, and Simple Log Service will automatically generate field indexes. For more information, see Create an index.

   Important

   Use a full-text index to query all log fields. To query specific fields, reduce index traffic, or perform analysis with SELECT statements, use a field index.

6. Click Query Log. Then, you are redirected to the query and analysis page of your Logstore.

   You must wait approximately 1 minute for the indexes to take effect. Then, you can view the collected logs on the Raw Logs tab. For more information about how to query and analyze logs, see Query and analysis quick start.

View the data import configuration

After you create a data import configuration, you can view its details and the generated report in the console.

1. Click the destination Project.

2. In the navigation pane of the destination Logstore, choose Data Collection > Data Import and click the name of the configuration.

3. On the Import Configuration Overview page, view the basic information and report for the configuration.

More operations

• Delete an import configuration

  On the Import Configuration Overview page, you can click Delete Configuration to delete the data import configuration.

  Warning

  This action cannot be undone. Proceed with caution.

• Stop and restart an import job

  When you create a data import configuration, SLS creates a corresponding import job. On the Import Configuration Overview page, you can click Stop to stop the job. You can restart it later.

  Important

  A stopped job's status is retained for 24 hours. If not restarted within this period, the job becomes unavailable and will fail if you attempt to restart it later.

FAQ

Error handling