You can use Log Service to collect log data and ship the log data to Object Storage Service (OSS) for storage and analysis. This topic describes how to ship log data from Log Service to OSS.
- A project and a Logstore are created. For more information, see Create a project and a Logstore.
- Log data is collected. For more information, see Log collection methods.
- OSS is activated. A bucket is created in the region where the Log Service project resides. For more information, see Activate OSS.
- Log Service is authorized to access OSS resources. You can authorize Log Service on
the Cloud Resource Access Authorization page.
To ship log data across Alibaba Cloud accounts or by using a RAM user, see Authorization in the RAM console.
- You can set a custom retention period for the log data in the OSS bucket. Permanent retention is supported.
- You can use data processing platforms such as E-MapReduce and Data Lake Analytics (DLA) or use custom programs to consume log data from the OSS bucket.
Ship log data
- Log on to the Log Service console.
- In the Projects section, click the name of the project.
- On the > icon of the Logstore. Then, choose . tab, click the
- On the OSS Shipper page, click Enable.
- In the Shipping Notes dialog box, click Ship.
- Configure a rule in the OSS LogShipper panel to ship log data to an OSS bucket. The following table describes the required parameters.
Parameter Description OSS Shipper Name The name of the shipping rule.
The name can contain only lowercase letters, digits, hyphens (-), and underscores (_). The name must start and end with a lowercase letter or digit and must be 2 to 128 characters in length.
OSS Bucket The name of the OSS bucket to which you want to ship log data.Notice You must specify the name of an existing OSS bucket. The specified OSS bucket must reside in the same region as the Log Service project. OSS Prefix The directory to which log data is shipped in the OSS bucket. Shard Format The partition format of the directory for a shipping task. The directory is generated based on the time when the shipping task is created. The default format is %Y/%m/%d/%H/%M. The value of this parameter cannot start with a forward slash (/). For information about partition format examples, see Partition format. For information about the parameters of partition formats, see strptime API. RAM Role The Alibaba Cloud Resource Name (ARN) of the RAM role. The RAM role is the identity that the OSS bucket owner creates for access control. Example: acs:ram::45643:role/aliyunlogdefaultrole. For more information, see Obtain the ARN of a RAM role. Shipping Size The maximum size of raw log data that can be shipped to the OSS bucket in a shipping task. Valid values: 5 to 256. Unit: MB.
If the size of shipped data exceeds the specified value, a new shipping task is created.
Storage Format The storage format of log data in OSS. Data shipped to OSS can be in the JSON, CSV, or Parquet format. For more information, see CSV-formatted data. Compress Specifies whether to compress log data that is shipped to OSS. Valid values:
- No Compress: Log data that is shipped to OSS is not compressed.
- Compress (snappy): The Snappy utility is used to compress the log data that is shipped to OSS. This releases more storage space of the OSS bucket.
Ship Tags Specifies whether to ship log tags. Shipping Time The time period during which a shipping task runs. Valid values: 300 to 900. Default value: 300. Unit: seconds.
If the specified time is reached, another shipping task is created.
- Click OK. Note
- After you configure a shipping rule, multiple shipping tasks can concurrently run. If the size of data shipped from a shard reaches the specified threshold or the specified shipping time is reached, another task is created.
- After you create a shipping task, you can check whether the shipping rule satisfies your business requirements based on the task status and the data shipped to OSS.
View OSS data
After log data is shipped to OSS, you can view the log data in the OSS console, or by using the OSS API, an SDK, or another method. For more information, see Manage objects.
|OSS Bucket||OSS Prefix||Partition format||OSS directory|
Note This format may prevent platforms such as Hive from parsing related log data in the OSS bucket. We recommend that you do not use this format.
You can use Hive, MaxCompute, or Data Lake Analytics (DLA) to analyze OSS data. In this case, if you want to use partition information, you can set PARTITION-FORMAT in the key=value format. For example, you can set the partition format to oss://test-bucket/log_ship_oss_example/year=2017/mon=01/day=20/log_195043_1484913043351525351_2850008.parquet. In this example, year, mon, and day are specified as three partition keys.
- Modify the shipping rule
Click Settings to modify the shipping rule. For information about parameters, see Ship log data.
- Disable data shipping
Click Disable. The data in the Logstore is not shipped to OSS.
- View the statuses of tasks and error messages
You can view the statuses of log shipping tasks within the last two days.
- Statuses of a shipping task
State Description Succeeded The shipping task succeeded. Running The shipping task is running. Check whether the task succeeds later. Failed The shipping task failed. If the task cannot be restarted due to external causes, troubleshoot the failure based on the error message and retry the task.
- Error message
If a shipping task fails, an error message is returned for the task.
Error message Description Method UnAuthorized The error message returned because the AliyunLogDefaultRole role does not have the required permissions. To troubleshoot the error, check the following configurations:
- Check whether the AliyunLogDefaultRole role is created by the OSS bucket owner.
- Check whether the Alibaba Cloud account ID configured in the permission policy is valid.
- Check whether the AliyunLogDefaultRole role is granted write permissions on the destination OSS bucket.
- Check whether the ARN of the AliyunLogDefaultRole role that you entered in the RAM Role field is valid.
ConfigNotExist The error message returned because the task does not exist. Check whether the data shipping feature is disabled. Enable the data shipping feature, configure a shipping rule for the task, and then retry the task. InvalidOssBucket The error message returned because the destination OSS bucket does not exist. To troubleshoot the error, check the following configurations:
- Check whether the destination OSS bucket resides in the same region as the source project.
- Check whether the specified bucket name is valid.
InternalServerError The error message returned because an internal error has occurred in Log Service. Retry the failed task.
- Retry a task
By default, if a task fails, Log Service retries the task based on the retry policy. You can also manually retry the task. By default, Log Service retries all tasks of the last two days. The minimum interval between two consecutive retries is 15 minutes. If a task fails for the first time, Log Service retries the task 15 minutes later. If the task fails for the second time, Log Service retries the task 30 minutes later. If the task fails for the third time, Log Service retries the task 60 minutes later. A similar method is used for subsequent attempts.
To immediately retry a failed task, you can click Retry All Failed Tasks or Retry on the right of the task. You can also use an API or SDK to retry a task.
- Statuses of a shipping task
How do I obtain the ARN of a RAM role?
- To obtain the ARN of a RAM role, log on to the RAM console.
- In the left-side navigation pane, click RAM Roles.
- In the RAM Role Name column, click the role named AliyunLogDefaultRole.
- On the page that appears, obtain the ARN of the role in the Basic Information section.