Import and analyze OSS files with Simple Log Service - Simple Log Service

You can import log files from an Object Storage Service (OSS) bucket to Simple Log Service (SLS) for query, analysis, and processing. Simple Log Service can import OSS files up to 5 GB. For compressed files, this limit applies to the compressed file size.

Billing

Simple Log Service does not charge for the data import feature. However, this feature incurs OSS API request and traffic fees. For more information about the pricing of billable items, see OSS Pricing. The daily OSS fees for importing data from OSS are calculated using the following formula:

Billing parameters

Parameter	Description
`N`	The number of files imported per day.
`T`	The total amount of data imported per day, in GB.
`p_read`	The traffic fee per GB of data. If you import data from a bucket in the same region as your SLS project, outbound traffic over the internal network is generated. This traffic is free of charge. Importing data across regions generates outbound traffic over the Internet.
`p_put`	The fee per 10,000 PUT requests. Simple Log Service calls the ListObjects API operation to list the files in a destination bucket. This operation is billed as PUT requests and returns a maximum of 1,000 entries at a time. Therefore, listing 1,000,000 new files requires 1,000 requests.
`p_get`	The fee per 10,000 GET requests.
`M`	The interval at which to check for new files, in minutes. You can set the New File Check Cycle parameter when you create a data import configuration.

Prerequisites

You have uploaded log files to an OSS bucket. For more information, see Upload objects.
You have created a project and a Logstore. For more information, see Manage projects and Create a Logstore.
You have completed Cloud Resource Access Authorization, granting Simple Log Service permission to access your OSS resources using the AliyunLogImportOSSRole role.

Your account has the oss:ListBuckets permission. For more information, see Grant custom permissions to a RAM user.

If you use a RAM user, you must also grant the ram:PassRole permission. The following policy is an example. For more information, see Create a custom policy and Grant permissions to a RAM user.

{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["ram:PassRole", "ram:GetRole"],
      "Resource": "acs:ram:*:*:role/aliyunlogimportossrole"
    },
    {
      "Effect": "Allow",
      "Action": "oss:GetBucketWebsite",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "oss:ListBuckets",
      "Resource": "*"
    }
  ],
  "Version": "1"
}

Create a data import configuration

Important

If an imported file is updated, the job re-imports the entire file, including existing data.

Log on to the Simple Log Service console.
In the Import Data section, on the Data Import tab, click OSS - Data Import.
Select the destination project and Logstore, and then click Next.

Set the import parameters.

In the Import Configuration step, set the following parameters.

Parameters

Parameter	Description
J*******	A unique name for the SLS job.
Display name	The display name of the job.
Job description	The description of the import job.
OSS region	The region where the bucket that stores the OSS files resides. If the OSS bucket and the Simple Log Service project are in the same region, you benefit from faster transfer speeds and avoid charges for outbound traffic over the Internet.
Bucket	The bucket that contains the OSS files to import.
File path prefix filter	Filters files by their path prefix. For example, if all files to import are in the csv/ directory, you can specify the prefix as csv/. If you do not set this parameter, the entire OSS bucket is traversed. Note We recommend that you set this parameter. If the bucket contains a large number of files, traversing the entire bucket is inefficient.
File path regex filter	Imports only files whose full paths match the specified regular expression. If left empty, this filter is not applied. For example, if an OSS file is named `testdata/csv/bill.csv`, you can set the regular expression to `(testdata/csv/)(.*)`. For more information about how to debug a regular expression, see How to test a regular expression.
File modification time filter	Filters OSS files by their modification time. All: Select this option if you want to import all eligible files. From Specific Time: Select this option if you want to import files modified after a specific point in time. Specific Time Range: Select this option if you want to import files modified within a specific time range.
Data format	The parsing format for the files. CSV: A text file delimited by separators. You can specify the first line of the file as field names or manually specify field names. Each line other than the field names is parsed as the values of log fields. Single-line JSON: Reads the OSS file line by line and parses each line as a JSON object. After parsing, each field in the JSON object corresponds to a field in the log. JSON array: Reads the entire OSS file at once. The content is an array that contains one or more JSON objects. CloudTrail: Reads the entire OSS file at once. The content is in the standard CloudTrail data structure format. Single-line Text Log: Parses each line in the OSS file as a single log entry. Multi-line Text Log: A multi-line mode that supports log parsing by using regular expressions to match the first or last line of a log entry. ORC: An ORC file. No configuration is required. The file is automatically parsed into a log format. Parquet: A Parquet file. No configuration is required. The file is automatically parsed into a log format. Alibaba Cloud OSS Access Log: The format of an Alibaba Cloud OSS access log. For more information, see Log shipping. Alibaba Cloud CDN Download Log: The format of an Alibaba Cloud CDN download log. For more information, see Quick start.
Compression format	The compression format of the OSS files to import. Simple Log Service decompresses the files based on the specified format and reads the data.
Encoding format	The encoding format of the OSS files to import. Only UTF-8 and GBK are supported.
New File Check Cycle	If new files are continuously generated in the destination OSS file path, you can set a New File Check Cycle based on your requirements. After you set this parameter, the import job runs in the background and periodically discovers and reads new files. The system ensures that data from the same OSS file is not repeatedly written to Simple Log Service. For example, if you create a job at 12:00 with a check cycle of 30 minutes, the job's first execution is at 12:00. If a new file is generated afterward, the second execution is at 12:30. If new files are no longer generated in the destination OSS file path, set this parameter to Never Check. The import job then stops automatically after reading all eligible files.
Import Archive Files	Files in the Archive or Cold Archive storage class must be restored before they can be read. Enabling this feature automates the restoration process. Deep Cold Archive files are not supported. Note Restoring Archive files takes about 1 minute, which may cause the first preview to time out. If a timeout occurs, wait a moment and try again. Restoring Cold Archive files takes about 1 hour. If the preview times out, you can skip the preview or wait for 1 hour before trying again. When Cold Archive files are restored, the default restoration period is 7 days to ensure sufficient time to import the files.
Log time configuration
Time field	If you set Data Format to CSV, Single-line JSON, JSON array, CloudTrail, ORC, Parquet, Alibaba Cloud OSS Access Log, or Alibaba Cloud CDN Download Log, you must set a time field. This field is the column name that represents time in the OSS file and is used to specify the log time when importing to Simple Log Service.
Regular expression to extract time	If you set Data Format to Single-line Text Log or Multi-line Text Log, you must use a regular expression to extract the time from the log. For example, if a log sample is 127.0.0.1 - - [10/Sep/2018:12:36:49 +0800] "GET /index.html HTTP/1.1", you can set Regular expression to extract time to `[0-9]{0,2}\/[0-9a-zA-Z]+\/[0-9\: +]+`. Note For other data formats, you can also use a regular expression if you need to extract only a part of the time field.
Time field format	Specifies the time format to use for parsing the value of the time field. Time formats that follow the Java SimpleDateFormat syntax are supported, such as `yyyy-MM-dd HH:mm:ss`. For more information about the syntax, see Class SimpleDateFormat. For more information about common time formats, see Time formats. Epoch formats are supported, including epoch, epochMillis, epochMicro, and epochNano.
Time zone	Select the time zone that corresponds to the time field. If the time field format is an epoch type, you do not need to set the time zone. If daylight saving time needs to be considered when the log time is parsed, select a UTC format. Otherwise, select a GMT format.
Advanced settings
OSS Metadata Indexing	We strongly recommend that you enable this feature when the number of OSS files exceeds one million. Otherwise, discovering new files is very inefficient. With OSS Metadata Indexing, new files are discovered within seconds, enabling near-real-time data import. Before you use OSS Metadata Indexing, you must first enable the Meta Query feature in OSS. For more information, see Meta Query.

If you set Data Format to CSV or Multi-line Text Log, you must configure additional parameters. The following tables describe these parameters.

CSV

Parameter	Description
Delimiter	The delimiter for logs. The default value is a comma (,).
Quote	The quote character used for the CSV string.
Escape character	The escape character for logs. The default value is a backslash (\).
Maximum lines	If a log spans multiple lines, you must specify the maximum number of lines. The default value is 1.
First Line as Field Name	If you enable First Line as Field Name, the first line of the CSV file is used as the field names. For example, the first line in the following figure is extracted as the field names for the log.
Custom fields	If you disable First Line as Field Name, define the custom field names. Separate multiple field names with commas (,).
Lines to skip	Specifies the number of log lines to skip. For example, a value of 1 means that log collection starts from the second line of the CSV file.

Multi-line text log

Parameter	Description
Position to match regular expression	Specifies where the regular expression should match: Regular Expression to Match First Line: Uses a regular expression to match the beginning of a log entry. The unmatched part is considered part of that log entry until the maximum number of lines is reached. Regular Expression to Match Last Line: Uses a regular expression to match the end of a log entry. The unmatched part is considered part of the next log entry until the maximum number of lines is reached.
Regular expression	Set the correct regular expression based on the log content. For more information about how to debug a regular expression, see How to test a regular expression.
Maximum lines	The maximum number of lines for a single log entry.

Click Preview to view the import results.
After you confirm that the configurations are correct, click Next.

Create indexes and preview data. Then, click Next. By default, full-text indexing is enabled in Simple Log Service. You can also manually create field indexes for the collected logs or click Automatic Index Generation. Then, Simple Log Service generates field indexes. For more information, see Create indexes.
Important
If you want to query all fields in logs, we recommend that you use full-text indexes. If you want to query only specific fields, we recommend that you use field indexes. This helps reduce index traffic. If you want to analyze fields, you must create field indexes. You must include a SELECT statement in your query statement for analysis.
Click Query Log to go to the query and analysis page and confirm whether the OSS data has been successfully imported.
Wait about one minute. The import is successful if the data appears in the Logstore.

Related operations

After creating a data import configuration, you can view the configuration details and related statistical reports in the Simple Log Service console.

In the Projects section, click the destination project.
Navigate to Log Storage > Logstores, select the destination Logstore, select Data Ingestion > Data Import, and then click the configuration name.
View job
On the Import Configuration Overview page, view the basic information and statistical reports of the data import job.
Modify configuration
Click Modify Configuration to edit the import settings. For more information, see Set the import parameters.
Delete configuration
Click Delete Configuration to delete the import configuration.
Warning
This operation cannot be undone. Proceed with caution.
Stop job
Click Stop to stop the data import job.
Start job
Click Start to start the data import job.

FAQ

Issue	Cause	Solution
No data is available for preview when importing files from the HDFS directory of a bucket.	Importing files from the HDFS directory is not supported.	If the HDFS service is enabled for the bucket, a .dlsdata directory is created by default under the OSS directory. You can import files from the .dlsdata directory.
No data is displayed during preview.	The OSS bucket contains no files, the files have no data, or no files match the filter conditions.	Check that the bucket contains non-empty files, for example, CSV files with more than just a header row. If no data exists, wait for data to become available before you start the import. Adjust the File Path Prefix Filter, File Path Regex Filter, and File Modification Time Filter settings.
The data contains garbled characters.	The settings for Data Format, Compression Format, or Encoding Format are incorrect.	Confirm the actual format of the OSS files and adjust the Data Format, Compression Format, or Encoding Format settings. To fix existing garbled data, create a new Logstore and a new data import configuration.
The timestamps of logs in Simple Log Service do not match the timestamps in the source data.	The time field was not specified when the import job was configured, or the time format or time zone was set incorrectly.	Specify the log time field and the correct time format and time zone. For more information, see Create a data import configuration.
Data cannot be queried or analyzed after being imported.	The data is outside the query time range. An index is not configured. The index has not taken effect.	Check whether the time of the data you want to query is within the query time range. If not, adjust the time range and query again. Check whether an index is configured for the Logstore. If not, create an index first. For more information, see Create an index and Rebuild an index. If an index is already configured and the amount of successfully imported data shown on the Data Processing Insight dashboard is as expected, the index may not have taken effect. Try to rebuild the index. For more information, see Rebuild an index.
The number of imported data entries is less than expected.	Some files contain single lines of data that exceed 3 MB in size, causing the data to be dropped during the import process. For more information, see Data collection limits.	When you write data to OSS files, ensure that the size of a single line does not exceed 3 MB.
The number of files and total amount of data are both large, but the import speed is slower than expected. Import speeds can normally reach 80 MB/s.	The number of Logstore shards is too small. For more information, see Performance limits.	If the number of Logstore shards is small, try increasing the number of shards to 10 or more and observe the latency. For more information, see Manage shards.
Cannot select an OSS bucket when you create a data import configuration.	The AliyunLogImportOSSRole role has not been authorized.	Complete the authorization as described in the Prerequisites section of this topic.
Some files are not imported.	The filter conditions are set incorrectly, or some files exceed 5 GB in size. For more information, see Data collection limits.	Check whether the files to be imported meet the filter conditions. If not, modify the filter conditions. Make sure that the size of each file to be imported is less than 5 GB. If a file exceeds 5 GB, reduce its size.
Archived files are not imported.	The Import Archive Files switch is turned off. For more information, see Data collection limits.	Method 1: Modify the import configuration and turn on the Import Archive Files switch. Method 2: Create a new import configuration and turn on the Import Archive Files switch.
Multi-line text logs are parsed incorrectly.	The regular expression for the first line or last line is incorrect.	Verify that the regular expression for the first or last line is correct.
The import of new files is significantly delayed.	There are too many existing files that match the file path prefix, and the OSS Metadata Indexing switch is turned off in the import configuration.	If the number of files that match the file path prefix is too large (over 1 million), you need to turn on the OSS Metadata Indexing switch in the import configuration. Otherwise, discovering new files will be very inefficient.
An STS-related permission error occurs during creation.	The RAM user has insufficient permissions.	Check whether the AccessKey pair of the RAM user is valid. Verify that the AccessKey pair is correct and enabled. For more information, see AccessKey pairs. Check whether the STS configuration for the RAM user has expired. If the temporary access credential has expired, extend its validity period. For more information, see Use temporary credentials provided by STS to access OSS. Verify that you have completed Cloud Resource Access Authorization to grant Simple Log Service the permission to access your OSS resources using the AliyunLogImportOSSRole role.

Error handling

Error	Description
File read failure	If an incomplete file error occurs during reading, for example, due to network anomalies or file damage, the import job automatically retries. If the read fails after three retries, the file is skipped. The retry interval is the same as the New File Check Cycle. If New File Check Cycle is set to Never Check, the retry interval is 5 minutes.
Compression format parsing error	If an invalid file compression format error occurs during decompression, the import job skips that file.
Data format parsing error	If parsing binary format data (ORC, Parquet) fails, the import job skips that file. If parsing data in other formats fails, the import job stores the original text content in the content field of the log.
OSS bucket does not exist	The import job retries periodically. After the bucket is re-created, the import job automatically resumes.
Permission error	If a permission error occurs while reading from an OSS bucket or writing to an SLS Logstore, the import job retries periodically. After the permissions are corrected, the job automatically resumes. The import job does not skip any files when a permission error occurs. Therefore, after the permissions are corrected, the job automatically imports data from the unprocessed files in the bucket to the SLS Logstore.

API reference

Actions	API operation
Create an OSS import job	CreateOSSIngestion
Update an OSS import job	UpdateOSSIngestion
Get an OSS import job	GetOSSIngestion
Delete an OSS import job	DeleteOSSIngestion
Start an OSS import job	StartOSSIngestion
Stop an OSS import job	StopOSSIngestion

Billing

Prerequisites

Create a data import configuration

CSV

Multi-line text log

Related operations

View job

Modify configuration

Delete configuration

Stop job

Start job

FAQ

Error handling

API reference