This topic describes how to import data from Elasticsearch to Log Service. After you import data to Log Service, you can query, analyze, and transform the data in the Log Service console.

Prerequisites

Create a data import configuration

Notice
  • You can import data from Elasticsearch to a Logstore that resides in the China (Hangzhou), China (Shanghai), China (Beijing), China (Shenzhen), or China (Chengdu) region.
  • You can import data from Elasticsearch to Log Service only once. You cannot import data in batches.
  1. Log on to the Log Service console.
  2. In the Projects section, click the project that you want to view.
  3. Choose Log Storage > Logstores. On the Logstores tab, find the Logstore that you want to manage and click >. Then, move the pointer over Data Import > Data Import and click +.
  4. In the Import Data dialog box, click Elasticsearch - Data Import.
  5. In the Configure Import Settings step, create a data import configuration.
    1. In the Specify Data Source step, configure the parameters. The following table describes the parameters.
      Parameter Description
      Configuration Name The name of the data import configuration.
      Service Instance URL The URL of the Elasticsearch server. Format: http://host:port/.

      You can specify multiple URLs. Separate multiple URLs with commas (,). Example: http://host1:port1/,http://host2:port2/,http://host3:port3/.

      In most cases, the service port of an Elasticsearch server is port 9200.
      Note If you configure the VPC-based Instance ID parameter, you must set the host variable to the IPv4 address of the corresponding Elastic Compute Service (ECS) instance.
      Elasticsearch Index List The indexes to be imported. Separate multiple indexes with commas (,). Example: index1,index2,index3.
      Elasticsearch User Name The username that is used to access the Elasticsearch cluster. This parameter is required only if user authentication is required to access the Elasticsearch cluster.
      Elasticsearch User Password The password of the username that is used to access the Elasticsearch cluster.
      Time The name of the column that represents the time in the Elasticsearch indexes. This column records the time when data is imported to Log Service.
      Note If you do not specify a time field, Log Service uses the system time by default when data is imported.
      Time Field Format Specifies the time format that is used to parse the value of the time field. Example: yyyy-MM-dd HH:mm:ss.
      • The time formats that use the Java SimpleDateFormat syntax are supported. For more information about the time format syntax, see Class SimpleDateFormat. For more information about the common time formats, see Time formats.
      • The time can be an epoch timestamp, such as epoch, epochMillis, epochMacro, or epochNano.
      Note Java SimpleDateFormat does not support UNIX timestamps. If you want to use UNIX timestamps, you can set this parameter to epoch.
      Time Zone The time zone that corresponds to the time field.

      If the time format is set to epoch, you do not need to configure the Time Zone parameter.

      Elasticsearch Query String The query statement that is used to filter data. The specified query statement must conform to the Elasticsearch query_string format. Example: gender:male and city:Shanghai. For more information, see Query string query.
      Start At Only the data whose value of the time field is greater than or equal to the specified start time can be imported to Log Service.
      Note This parameter is valid only if you configure the Time parameter. The format of the start time must be in the format that is specified by the Time Field Format parameter.
      End Time Only the data whose value of the time field is less than or equal to the specified end time can be imported to Log Service.

      If you set the End Time parameter to a point in time in the future, Log Service automatically creates an import task to import incremental data. When the point in time that is specified by the End Time is reached, the import task is automatically completed. If you want to import incremental data for a long time, you can set the End Time parameter to a large value.

      Note This parameter is valid only if you configure the Time parameter. The format of the end time must be in the format that is specified by the Time Field Format parameter.
      VPC-based Instance ID If the Elasticsearch cluster is an Alibaba Cloud Elasticsearch cluster in a virtual private cloud (VPC) or a self-managed Elasticsearch cluster on an ECS instance, you can configure this parameter to allow Log Service to read data from the Elasticsearch cluster over the internal network of Alibaba Cloud. If you read data over the internal network of Alibaba Cloud, higher security and network stability can be ensured.
      Note The Elasticsearch cluster must allow access from the CIDR block 100.104.0.0/16.
    2. Click Preview to preview the import result.
    3. After you confirm the settings, click Next.
  6. Preview data, configure indexes, and then click Next.
    By default, full-text indexing is enabled for Log Service. You can also configure field indexes based on collected logs in manual or automatic mode. For more information, see Configure indexes.
    Note If you want to query and analyze logs, you must enable full-text indexing or field indexing. If you enable both full-text indexing and field indexing, the system uses only field indexes.

View the data import configuration

After you create the data import configuration, you can view the configuration details and related statistical reports in the Log Service console.

  1. In the Projects section, click the project to which the data import configuration belongs.
  2. Find the Logstore to which the data import configuration belongs, choose Data Import > Data Import, and then click the name of the data import configuration.
  3. On the Import Configuration Overview page, view the basic information and statistical reports of the data import configuration.

What to do next

  • Delete the data import configuration
    On the Import Configuration Overview page, you can click Delete Configuration to delete the data import configuration.
    Notice If the data import configuration is deleted, it cannot be recovered. Proceed with caution.
  • Stop and restart the import task of the data import configuration
    After you create the data import configuration, Log Service generates an import task. On the Import Configuration Overview page, click Stop to stop the import task. Then, you can restart the import task.
    Notice After an import task is stopped, the task is in the stopped state for up to 24 hours. If the import task is not restarted during this period, it becomes unavailable. If you restart an unavailable import task, errors may occur.