This topic describes how to import data from Object Storage Service (OSS) to Log Service. After you store logs in OSS, you can import these logs as OSS data to Log Service. Then, you can query, analyze, and transform the data in Log Service. Log Service can only import OSS objects that are less than 5 GB in size. The size of a compressed object is calculated based on the compressed size.
Prerequisites
- Logs are uploaded to an OSS bucket. For more information, see Upload objects.
- A project and a Logstore are created. For more information, see Create a project and a Logstore.
- Log Service is authorized to use the AliyunLogImportOSSRole role to access your OSS
resources. For more information, see Cloud Resource Access Authorization.
If you use a RAM user, you must attach the PassRole policy to the RAM user. The following script shows an example. For more information, see Create a custom policy and Grant permissions to a RAM user.
{ "Statement": [ { "Effect": "Allow", "Action": "ram:PassRole", "Resource": "acs:ram:*:*:role/aliyunlogimportossrole" } ], "Version": "1" }
Create a data import configuration
View the data import configuration
After you create a data import configuration, you can view the configuration details and related statistical report in the Log Service console.
Related operations
On the Import Configuration Overview page, you can perform the following operations:
- Modify the data import configuration.
You can click Modify Settings to modify the data import configuration. For more information, see Configure data import.
- Delete the data import configuration.
You can click Delete Configuration to delete the data import configuration.Warning After the data import configuration is deleted, it cannot be recovered.
Common issues
Issue | Cause | Solution |
---|---|---|
You cannot select an OSS bucket when you create a data import configuration. | The AliyunLogImportOSSRole role is not authorized. | See the "Prerequisites" section in this topic to complete the authorization. |
Data cannot be imported. | The object size exceeds 5 GB. | Reduce the size of each OSS object. |
After the data is imported, you cannot query or analyze the data. | No indexes are configured or the indexes failed to take effect. | To make indexes take effect after data is imported, we recommend that you create indexes for the specified Logstore in advance. For more information, see Enable and configure the indexing feature for a Logstore. If the indexes have failed to take effect, you can reindex logs for the Logstore. For more information, see Reindex logs for a Logstore. |
Archived objects cannot be imported. | The Restore Archived Files switch is not turned on. |
|
The Regular Expression Filter parameter is specified, but no data is collected. |
|
Reset the Regular Expression Filter parameter. If data still cannot be collected, the probable cause is that the number of objects is too large. You must specify a more specific directory to reduce the number of objects to be traversed. |
The logs are imported, but no data can be found in the Log Service console. | The log time is not included in the data retention period of the Logstore. Expired data is deleted. | Check the query time range and the retention period of data in the Logstore. |
The extracted log time is used to query data, but no data is queried at this time. | The time format is incorrectly specified. | Check whether the time format is Java SimpleDateFormat. For more information, see Class SimpleDateFormat. |
Multi-line text logs are incorrectly parsed. | The regular expression that is used to match the first line or the last line of a log entry is incorrectly specified. | Check whether the regular expression that is used to match the first line or the last line of a log entry is correctly specified. |
The import speed suddenly slows down. |
|
|
Appendix
- Sample time formats
Time format Parsing syntax Parsed value (Unit: seconds) 2020-05-02 17:30:30 yyyy-MM-dd HH:mm:ss 1588411830 2020-05-02 17:30:30:123 yyyy-MM-dd HH:mm:ss:SSS 1588411830 2020-05-02 17:30 yyyy-MM-dd HH:mm 1588411800 2020-05-02 17 yyyy-MM-dd HH 1588410000 20-05-02 17:30:30 yy-MM-dd HH:mm:ss 1588411830 2020-05-02T17:30:30V yyyy-MM-dd'T'HH:mm:ss'V' 1588411830 Sat May 02 17:30:30 CST 2020 EEE MMM dd HH:mm:ss zzz yyyy 1588411830 - Time parsing syntax
Character Description Example G Era indicator AD y 4-digit year number 2001 M Month name or number July or 07 d Day of the month as a number 10 h Hour (12-hour clock, 1 to 12) 12 H Hour (24-hour clock, 0 to 23) 22 m Minute 30 s Second 55 S Millisecond 234 E Day of the week (Sunday to Saturday) Tuesday D Day of the year 360 F Day of the week in a month 2 w Week number of the year 40 W Week number of the month 1 a AM/PM PM k Hour (24-hour clock, 1 to 24) 24 k Hour (12-hour clock, 0 to 11) 10 z Time zone Eastern Standard Time ' Text delimiter Delimiter " Single quotation mark "