Simple Log Service:Import data from Kafka

Supported versions

Only Kafka 2.0.0 and later are supported.

Create a data import configuration

1. Log on to the Simple Log Service console.

2. In the Import Data section, on the Data Import tab, click Kafka - Data Import.

3. Select the destination project and Logstore, and then click Next.

4. Configure the import settings.

   1. In the Import Settings step, configure the following parameters.

      Parameter	Description
      Task Name	A unique name for the data import configuration.
      Display Name	The display name of the configuration.
      Job Description	A description of the import configuration.
      Endpoint	The Kafka bootstrap server endpoint. You can specify multiple endpoints separated by commas (,). If you use ApsaraMQ for Kafka, enter the IP address or domain name of the endpoint. If you use a self-managed Kafka cluster on an Alibaba Cloud ECS instance, enter the IP address of the ECS instance. If you use another type of Kafka cluster, enter the public IP address or domain name of a Kafka broker.
      Topics	The Kafka topics from which to import data. You can specify multiple topics separated by commas (,).
      Consumer Group	If you use ApsaraMQ for Kafka and have not enabled the Free-to-Use Group feature, you must select a consumer group. For more information about how to create a consumer group, see Create a consumer group.
      Starting Position	The position from which to start importing data. Earliest: Start importing from the first available data record in the topic. Latest: Start importing from the most recent data record in the topic.
      Data Format	The format of the data to import. Simple Mode: If your data is in a single-line format, select Simple Mode. JSON String: If your data is in JSON format, select JSON String. This setting parses the first layer of JSON data into key-value pairs.
      Parse Array Elements	If you enable Parse Array Elements, Simple Log Service splits JSON arrays into separate log entries by array element before import.
      Encoding Format	The encoding format of the data to import. UTF-8 and GBK are supported.
      VPC ID	If your Kafka cluster is in a Virtual Private Cloud (VPC), such as a ApsaraMQ for Kafka cluster or a self-managed cluster on an ECS instance, specify the VPC ID to allow Simple Log Service to read data over the internal network. Internal network access provides improved security and network stability. Important The Kafka cluster must be accessible from the CIDR block 100.104.0.0/16.
      Time Configuration
      Time Field	The field in your Kafka data that represents the log time. Simple Log Service uses this field as the timestamp for each imported log entry.
      Time Field Extraction Regular Expression	If you set Data Format to Simple Mode, you must specify a regular expression to extract the time from your Kafka data. For example, for the log content message with time 2022-08-08 14:20:20, you can set Time Field Extraction Regular Expression to \d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d.
      Time Field Format	The format for parsing the time field's value. Java SimpleDateFormat patterns, such as yyyy-MM-dd HH:mm:ss. For more information, see Class SimpleDateFormat. For common time formats, see Time formats. Epoch time formats. Valid values are epoch, epochMillis, epochMacro, and epochNano.
      Time Zone	The time zone of the time field. This setting is not required if the time format is epoch.
      Default Time Source	The fallback time source when time extraction is not configured or fails. Options include the current system time and the Kafka message timestamp.
      Advanced Settings
      Log Context	If you enable Log Context, you can view the log entries that precede or follow a specific log entry within the original Kafka partition.
      Communication Protocol	For data imported over the public network, we recommend using an encrypted connection with user authentication. Configure the communication protocol as shown in the following example. The protocol field supports plaintext, ssl, sasl_plaintext, and sasl_ssl. We recommend setting this to sasl_ssl, which requires both connection encryption and user authentication. If you set protocol to sasl_plaintext or sasl_ssl, you must configure the sasl node. The mechanism field can be PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512, which represents a username and password authentication mechanism. { "protocol":"sasl_plaintext", "sasl":{ "mechanism":"PLAIN", "username":"xxx", "password":"yyy" } }
      Private Domain Resolution	If the Kafka brokers deployed on Alibaba Cloud ECS instances use internal domain names to communicate with each other, you must specify the corresponding domain name and IP address for each broker. The following code provides an example. { "hostname#1":"192.168.XX.XX", "hostname#2":"192.168.XX.XX", "hostname#3":"192.168.XX.XX" }

   2. Click Preview to check the import result.

   3. After you confirm the result, click Next.

5. Create indexes and preview data. Then, click Next. By default, full-text indexing is enabled in Simple Log Service. You can also manually create field indexes for the collected logs or click Automatic Index Generation. Then, Simple Log Service generates field indexes. For more information, see Create indexes.

   Important

   If you want to query all fields in logs, we recommend that you use full-text indexes. If you want to query only specific fields, we recommend that you use field indexes. This helps reduce index traffic. If you want to analyze fields, you must create field indexes. You must include a SELECT statement in your query statement for analysis.

6. Click Query Log to go to the query and analysis page and verify the data import.

   Wait about one minute. If the Kafka data appears, the import is successful.

View the data import configuration

You can view the details and statistical reports of a data import configuration in the console.

1. In the Projects section, click the destination Project.

2. Navigate to the destination Logstore, choose Data Collection > Data Import, and then click the configuration name.

3. On the Import Configuration Overview page, view the basic information and statistical reports for the configuration.

Related operations

On the Import Configuration Overview page, you can also perform the following operations.

• Modify a configuration

  Click Edit Configurations to change the configuration settings. For more information about the parameters, see Create a data import configuration.

• Delete a configuration

  Click Delete Configuration to delete the import configuration.

  Warning

  This action cannot be undone. Proceed with caution.

• Stop a task

  Click Stop to stop the import task.

FAQ

Error handling mechanism