Simple Log Service:Import data from Amazon S3 to Simple Log Service

Prerequisites

• Log files are uploaded to Amazon S3.

• A project and logstore are created. For more information, see Manage a project and Create a logstore.

• Custom permissions:

  • Create a custom policy that grants permissions to manage S3 resources, as shown in the example that follows. For more information, see Create custom permissions on AWS.

    Note

    You must configure custom permissions for S3 to import objects into SLS.

    {
      "Version": "1",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "s3:GetObject",
            "s3:ListBucket"
          ],
          "Resource": [
            "arn:aws:s3:::your_bucket_name",
            "arn:aws:s3:::your_bucket_name/*"
          ]
        }
      ]
    }

  • Create a custom policy that grants permissions to manage Amazon Simple Queue Service (SQS) resources. For more information, see Create custom permissions on AWS.

    Note

    This is required only if you enable SQS.

    {
      "Effect": "Allow",
      "Action": [
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage",
        "sqs:GetQueueAttributes",
        "kms:Decrypt"
      ],
      "Resource": "*"
    }

Create a data import configuration

1. Log on to the Simple Log Service console.

2. On the Data Import tab in the Import Data area, select S3 - Data Import.

3. Select the target project and logstore, then click Next.

4. Configure import settings.

   1. In the Import Configuration step, set the following parameters.

      Parameter	Description
      Job Name	The unique name of the SLS task.
      S3 Region	The region where the bucket that stores the objects to import is located.
      AWS AccessKey ID	The AccessKey ID used to access AWS. Important Make sure your AccessKey has permissions to access the corresponding AWS resources.
      AWS Secret AccessKey	The Secret AccessKey used to access AWS.
      SQS Queue URL	The identifier of the SQS queue. For more information, see Queue and message identifiers. Important This feature takes effect only after you enable SQS.
      File Path Prefix Filter	Filter S3 objects by file path prefix to locate the objects to import. For example, if all objects to import are in the `csv/` directory, set the prefix to `csv/`. If you do not set this parameter, the entire S3 bucket is traversed. Note Set this parameter. If the bucket contains many objects, traversing the entire bucket is inefficient.
      File Path Regex Filter	Filter S3 objects by a regular expression on the file path to locate the objects to import. Only objects whose names, including their paths, match the regular expression are imported. By default, this parameter is empty, which means no filtering is performed. For example, if an S3 object is named testdata/csv/bill.csv, set the regular expression to (testdata/csv/)(.*). For more information about how to debug a regular expression, see Debug a regular expression.
      File Modification Time Filter	Filter S3 objects by their modification time to locate the objects to import. All: To import all S3 objects that meet specified conditions, select this option. From Specific Time: To import files modified after a point in time, select this option. Specific Time Range: If you want to import files modified within a specific time range, select this option.
      Data Format	The parsing format of the objects. The options are described as follows. CSV: A delimited text file. Specify the first line as field names or manually specify field names. All lines except the first line are parsed as the values of log fields. Single-line JSON: An S3 file is read line by line. Each line is parsed as a JSON object. The fields in JSON objects are log fields. Single-line Text Log: Each line in an S3 file is parsed as a log. Multi-line Text Logs: Multiple lines in an S3 object are parsed as a log. Specify a regular expression to match the first line or the last line of a log.
      Compression Format	The compression format of the S3 objects to import. SLS decompresses the objects and reads the data based on the specified format.
      Encoding Format	The encoding format of the S3 objects to import. Only UTF-8 and GBK are supported.
      New File Check Cycle	If new objects are constantly generated in the specified directory of S3 objects, configure New File Check Cycle as needed. After you configure this parameter, the data import job is continuously running in the background, and new objects are automatically detected and read at regular intervals. The system ensures that data in an S3 object is not repeatedly written to SLS. If new objects are no longer generated in the target S3 path, set the option to Never Check. The import job will then automatically exit after reading all objects that meet the specified conditions.
      Log Time Configuration
      Time Field	When you set Data Format to CSV or Single-line JSON, set a time field. This field is the name of the column in the file that represents time and is used to specify the timestamp for logs when they are imported to SLS.
      Regular Expression To Extract Time	Use a regular expression to extract the time from a log. For example, if a sample log is 127.0.0.1 - - [10/Sep/2018:12:36:49 0800] "GET /index.html HTTP/1.1", set Regular Expression To Extract Time to [0-9]{0,2}\/[0-9a-zA-Z]+\/[0-9:,]+. Note For other data formats, you can also use a regular expression to extract only part of the time field.
      Time Field Format	Specify the time format to parse the value of the time field. Specify a time format that follows the Java SimpleDateFormat syntax. For example: yyyy-MM-dd HH:mm:ss. For more information about the syntax, see Class SimpleDateFormat. For more information about common time formats, see Time formats. Epoch formats are supported, including epoch, epochMillis, epochMicro, and epochNano.
      Time Zone	Select the time zone for the time field. When the time field format is an epoch type, you do not need to set a time zone. If daylight saving time needs to be considered when parsing log time, select a UTC format. Otherwise, select a GMT format. Note The default time zone is UTC+8.

      If you set Data Format to CSV, you must configure the additional parameters described in the following table.

      • CSV-specific parameters

        Parameter	Description
        Delimiter	Set the delimiter for logs. The default value is a comma (,).
        Quote	The quote character used for a CSV string.
        Escape Character	Configure the escape character for logs. The default value is a backslash (\).
        Maximum Lines	If you turn on First Line As Field Name, the first line in a CSV file is used as the field name.
        Custom Fields	If you turn off First Line As Field Name, you can specify custom field names. Separate multiple field names with commas (,).
        Lines To Skip	Specify the number of log lines to skip. For example, if you set this to 1, log collection starts from the second line of the CSV file.

      • Multi-line text log-specific parameters

        Parameter	Description
        Position to Match Regular Expression	Set the position for the regular expression to match. The options are described as follows: Regular Expression To Match First Line: If you select this option, the regular expression that you specify is used to match the first line of a log entry. The unmatched lines are collected as a part of the log entry until the maximum number of lines that you specify is reached. Regular Expression To Match Last Line: Uses a regular expression to match the last line of a log entry. Unmatched lines are considered part of the next log entry until the maximum number of lines is reached.
        Regular Expression	Set the correct regular expression based on the log content. For more information about how to debug a regular expression, see Debug a regular expression.
        Maximum Lines	The maximum number of lines for a single log entry.

   2. Click Preview to view the import result.

   3. After confirming the information, click Next.

5. Preview data, configure indexes, and click Next.

   By default, full-text indexing is enabled in SLS. You can also manually create field indexes based on collected logs, or click Automatic Index Generation to have SLS generate them automatically. For more information, see Create indexes.

   Important

   To query and analyze logs, you must enable either full-text indexing or field indexing. If both are enabled, field indexes take precedence.

View the import configuration

After you create an import configuration, view the configuration and its statistical reports in the console.

1. In the Project list, click the target project.

2. In Log Storage > Logstores, find the target logstore, select Data Collection > Data Import, and click the configuration name.

3. View basic information and statistical reports for the data import configuration.

   You can also modify the configurations, start or stop the import, and delete the configuration.

   Warning

   The delete operation is irreversible. Proceed with caution.

Billing

The import feature in SLS is free of charge. However, this feature requires access to service provider APIs and will incur traffic and request fees. The pricing model is as follows, and the final fees are determined by the bill from the service provider.

FAQ

Error handling