Simple Log Service:Import Amazon S3 files

Prerequisites

• Upload log files to Amazon S3.

• Create a project and a logstore. Manage a project, Create a logstore.

• Configure custom permissions:

  • Create a custom policy that grants permissions to manage S3 resources, as shown in the following example (AWS: Custom permission sets).

    Note

    Custom S3 permissions are required for importing files into SLS.

    {
      "Version": "1",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "s3:GetObject",
            "s3:ListBucket"
          ],
          "Resource": [
            "arn:aws:s3:::your_bucket_name",
            "arn:aws:s3:::your_bucket_name/*"
          ]
        }
      ]
    }

  • Create a custom policy that grants permissions to manage SQS resources (AWS: Custom permission sets).

    Note

    This is required only if you enable SQS integration.

    {
      "Effect": "Allow",
      "Action": [
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage",
        "sqs:GetQueueAttributes",
        "kms:Decrypt"
      ],
      "Resource": "*"
    }

Create a data import configuration

1. Log on to the Simple Log Service console.

2. Navigate to Import Data > Data Import and select S3 - Data Import.

3. Select the destination project and logstore, and then click Next.

4. Configure the import settings.

   1. In the Import Configuration step, set the following parameters.

      Parameter	Description
      Task Name	The unique name for the Data Import job.
      S3 Region	The region where the source bucket is located.
      AWS AccessKey ID	The AWS AccessKey ID used to access your AWS account. Important Make sure that your AccessKey ID has the required permissions to access the relevant AWS resources.
      AWS Secret AccessKey	The AWS Secret AccessKey that corresponds to your AccessKey ID.
      SQS Queue URL	The URL of the SQS queue (AWS: Queue and message identifiers). Important This parameter is required only if you enable SQS integration.
      File Path Prefix Filter	Filters S3 objects by key prefix. For example, set the prefix to csv/ to import only objects in that directory. If you leave this parameter empty, the entire S3 bucket is scanned. Note Configure a prefix for large buckets. Scanning an entire bucket slows down the import.
      File Path Regex Filter	Filters S3 objects by a regular expression matching the full object path. Only matching objects are imported. Empty by default (no filtering). For example, if an S3 object is at testdata/csv/bill.csv, you can set the regular expression to (testdata/csv/)(.*). Debug a regular expression.
      File Modification Time Filter	Filters S3 objects by their last modified time. All: Select this option to import all objects that meet the specified conditions. From Specific Time: Select this option to import objects that were modified after a specific point in time. Specific Time Range: Select this option to import objects that were modified within a specific time range.
      Data Format	The format for parsing log files: CSV: Delimited text. You can use the first row of the file as field names or manually specify field names. Each subsequent row is parsed as the values for a log entry. Single-line JSON: Reads the S3 object line by line and parses each line as a JSON object. The keys in the JSON object become the field names in the log. Single-line Text Log: Parses each line in the S3 object as a separate log entry. Multi-line Text Logs: Parses multiple lines as a single log entry. You must specify a regular expression to match the beginning or end of a log entry.
      Compression Format	The compression format of the source S3 objects. SLS decompresses the objects based on the selected format before reading the data.
      Encoding Format	The encoding format of the source S3 objects. Only UTF-8 and GBK are supported.
      New File Check Cycle	If new objects are continuously added to the source S3 path, set the New File Check Cycle as needed. The import job periodically discovers and reads new objects. Each S3 object is imported at most once. If no new objects will be added, select Never Check. The job stops after importing all matching objects.
      Log Time Configuration
      Time Field	If Data Format or Single-line JSON is selected as the Single-line JSON, you can specify a field that contains the timestamp. SLS uses this field as the log's timestamp.
      Time Field Extraction Regular Expression	You can use a regular expression to extract the timestamp from the log content. For example, if a sample log entry is 127.0.0.1 - - [10/Sep/2018:12:36:49 0800] "GET /index.html HTTP/1.1", you can set the Time Field Extraction Regular Expression to [0-9]{0,2}\/[0-9a-zA-Z]+\/[0-9:,]+. Note For other data formats, you can also use a regular expression to extract only part of a time field.
      Time Field Format	The format used to parse the time field's value. Supports Java SimpleDateFormat syntax, such as yyyy-MM-dd HH:mm:ss. Time formats. Epoch formats are supported, including epoch, epochMillis, epochMicro, and epochNano.
      Time Zone	Select the time zone for the time field. A time zone is not required for epoch formats. To account for daylight saving time, select a UTC format. Otherwise, select a GMT format. Note The default time zone is UTC+8.

      If you select Data Format as the Data Format, you must configure the additional parameters described in the following table.

      • CSV-specific parameters

        Parameter	Description
        Delimiter	The delimiter used to separate fields. The default value is a comma (,).
        Quote	The quote character used for CSV strings.
        Quote	The escape character used to escape special characters. The default value is a backslash (\).
        Maximum Lines	If you enable First Line as Field Name, the first line of the CSV file is used as the field names.
        Custom Fields	If you disable First Line as Field Name, you must specify custom field names for the logs. Separate multiple field names with commas (,).
        Lines to Skip	The number of log lines to skip from the beginning of the file. For example, if you set this to 1, data collection starts from the second line of the CSV file.

      • Multi-line text log-specific parameters

        Parameter	Description
        Position to Match Regular Expression	The position at which to match the regular expression. The following options are available: Regular Expression to Match First Line: Uses a regular expression to match the first line of a log entry. Unmatched lines are considered part of the current log entry until the maximum number of lines is reached. Regular Expression to Match Last Line: Uses a regular expression to match the last line of a log entry. Unmatched lines are considered part of the next log entry until the maximum number of lines is reached.
        Regular Expression	The regular expression that defines the start or end of a log entry. Debug a regular expression.
        Maximum Lines	The maximum number of lines for a single log entry.

   2. Click Preview to check the import result.

   3. After you confirm the settings, click Next.

5. Preview the data, create an index, and then click Next.

   SLS enables the full-text index by default. You can also manually create field indexes or click Automatic Index Generation to generate them automatically. Create an index.

   Important

   To query and analyze logs, you must enable either a full-text index or a field index. If both are enabled, the field index takes precedence.

View the import configuration

View configuration details and statistical reports in the console.

1. In the project list, click the destination project.

2. In the Log Storage > Logstore section, find the destination logstore, choose Data Collection > Data Import, and then click the configuration name.

3. View the basic information and statistical reports for the import configuration.

4. You can also modify, start, stop, or delete the import configuration.

   Warning

   This operation cannot be undone. Proceed with caution.

Billing

SLS does not charge for the Data Import feature. However, data transfer and API request fees from your cloud provider may apply. The following model outlines potential costs. Refer to your provider's bill for actual charges.

FAQ

Error handling