Simple Log Service:Create a MaxCompute data shipping job (new version)

Usage notes

• For information about the regions that support shipping data to MaxCompute (ODPS), see Supported regions.

• If the value of a field of the char or varchar type exceeds the specified length, the excess part is truncated when shipped to MaxCompute (ODPS).

  For example, if the length limit is 3 and the field value is 012345, the value becomes 012 after being shipped to MaxCompute (ODPS).

• If the value of a field of the string, char, or varchar type is an empty string, its value becomes Null after being shipped to MaxCompute (ODPS).

• The value of a field of the datetime type must be in the YYYY-MM-DD HH:mm:ss format. Multiple spaces are allowed between DD and HH. An invalid format does not cause a shipping error, but the value becomes Null after being shipped to MaxCompute (ODPS).

• If the value of a field of the date type is in an invalid format, the data shipping job does not report an error, but the value becomes Null after being shipped to MaxCompute (ODPS).

• For a decimal field, if its value exceeds the defined scale, the system rounds it. If the value exceeds the defined precision, the system discards the entire log as dirty data and increments the error count.

• By default, the system discards dirty data during data shipping.

• If a field specified in the shipping configuration does not exist in a log, its value in MaxCompute (ODPS) becomes the default value or Null.

  • If you specified a default value when you created the MaxCompute (ODPS) table, the field is assigned the default value.

  • If you did not specify a default value but the column allows Null values, the field value becomes Null.

• Due to MaxCompute (ODPS) limitations, the maximum data shipping concurrency is 64 write operations. The maximum throughput for a single partition is 10 MB/s.

Procedure

1. Log on to the Simple Log Service console.

2. In the Projects section, click the project you want.

3. On the Log Storage > Logstores tab, click the > icon to the left of the target logstore, and then choose Data Processing > Export > MaxCompute (ODPS).

4. Hover the pointer over MaxCompute (ODPS) and click the + icon.

5. In the Data Shipping to MaxCompute panel, configure the following parameters and click OK.

   Parameter description

   Parameter	Description
   Job name	The unique name of the data shipping job.
   Display name	The display name of the data shipping job.
   Job description	A description of the data shipping job.
   Destination region	The region where the destination MaxCompute (ODPS) table is located.
   MaxCompute endpoint	The endpoint of the MaxCompute (ODPS) service in the destination region. Note The panel for the MaxCompute endpoint is collapsed by default. You can click the icon to expand it, as shown in the following figure:
   Tunnel endpoint	The tunnel endpoint for the MaxCompute (ODPS) service in the destination region. Note The panel for the MaxCompute endpoint is collapsed by default. You can click the icon to expand it, as shown in the following figure:
   Project name	The MaxCompute (ODPS) project that contains the destination table.
   MaxCompute table name	The name of the MaxCompute (ODPS) table.
   Read Log Service authorization	Permissions for the data shipping job to read data from the logstore. Default role: Allows the data shipping job to assume the Alibaba Cloud system role AliyunLogDefaultRole to read data from the logstore. For more information, see Grant permissions to a default role. Custom role: Allows the data shipping job to assume a custom RAM role to read data from the logstore. You must first grant the custom role the required permissions to read data from the logstore. Then, in the Read Log Service authorization section, enter the Alibaba Cloud Resource Name (ARN) of your custom role. For more information, see Authorize a RAM role to read data from a logstore.
   Write MaxCompute authorization	Permissions for the data shipping job to write data to the MaxCompute (ODPS) table. Default role: Allows the data shipping job to assume the Alibaba Cloud system role AliyunLogDefaultRole to write data to the MaxCompute (ODPS) table. For more information, see Authorize a default role to write data to MaxCompute (for data shipping within the same account). Custom role: Allows the data shipping job to assume a custom RAM role to write data to the MaxCompute (ODPS) table. You must first grant the custom role the required permissions to write data to the MaxCompute (ODPS) table. Then, in the Write MaxCompute authorization section, enter the ARN of your custom role. If Log Service (SLS) and MaxCompute (ODPS) belong to the same Alibaba Cloud account, see Procedure. If Log Service (SLS) and MaxCompute (ODPS) belong to different Alibaba Cloud accounts, see Authorize a custom role to write data to MaxCompute (for data shipping across accounts).
   Automatic Authorization	Click Authorize to automatically grant the RAM role permissions to write data to MaxCompute (ODPS). Important If you are using a RAM user, the user must have permissions to manage the MaxCompute (ODPS) account. If automatic authorization fails, you can copy the four provided commands to the MaxCompute (ODPS) console to grant the permissions manually. For more information, see Grant permissions by using the command-line tool. USE xxxxx; ADD USER RAM$xxxxx:`role/xxxxx`; GRANT CreateInstance ON PROJECT xxxxx TO USER RAM$xxxxx:`role/xxxxx`; GRANT Describe, Alter, update ON TABLE xxxxx TO USER RAM$xxxxx:`role/xxxxx`;
   MaxCompute data columns	Map log fields to data columns in the MaxCompute (ODPS) table. Enter a log field name in the left text box and the corresponding data column name in the right text box. For more information, see Data model mapping. Important Data is shipped by mapping log fields to table columns in the specified order. Changing column names in MaxCompute (ODPS) does not affect shipping. However, if you change the table schema, you must reconfigure these mappings. The log field name in the left text box cannot contain double quotation marks (""), single quotation marks (''), or spaces. If your logs contain duplicate field names (for example, two fields named request_time), Log Service may display one as request_time_0 in the console. The underlying stored name remains request_time. Use the original field name request_time when configuring the shipping rule. With duplicate field names, the system randomly ships only one value. Avoid duplicate field names for predictable results.
   MaxCompute partition columns	Map log fields to partition key columns in the MaxCompute (ODPS) table. Enter a log field name in the left text box and the corresponding partition key column name in the right text box. For more information, see Data model mapping. Note The following fields cannot be used as partition fields: _extract_others_, __extract_others__, and __extract_others_all__. Configuring the partition key log field to a value other than __partition_time__ or __receive_time__ may affect delivery performance.
   Time partition format	The time partition format. For configuration examples and details, see Time partition format. Note This parameter takes effect only when the log field name in MaxCompute partition columns is set to __partition_time__ or __receive_time__. Do not use a date format that is precise to the second. This can cause the number of partitions in a single table to exceed the limit of 60,000.
   Time zone	The time zone used to format time values and time partitions. For more information, see Time zone format.
   Shipping mode	Select real-time shipping or batch shipping. Real-time shipping: Reads and ships data from the logstore to MaxCompute (ODPS) in real time. Batch shipping: Reads and ships data from the logstore in batches, processing data that is 5 to 10 minutes old. For more information, see Shipping modes.
   Start time range	The time range of data to ship, based on the log receive time. The following options are available: From Specific Time: The job starts shipping data from a specified start time and continues until it is manually stopped. Specific Time Range: The job ships data within a specified start and end time. The job stops automatically after reaching the end time. Note Shipping historical data is no longer supported because it can easily exceed the MaxCompute (ODPS) write threshold due to slot and QPS limitations.

   After you create a data shipping job, log data is typically imported into MaxCompute (ODPS) within one hour of being written to the logstore. You can then query the data in MaxCompute (ODPS). For more information, see How do I check the integrity of data after it is shipped from Log Service to MaxCompute?

   | log_source | log_time | log_topic | time | ip | thread | log_extract_others | log_partition_time | status |
   +------------+------------+-----------+-----------+-----------+-----------+------------------+--------------------+-----------+
   | 10.10.*.* | 1642942213 | | 24/Jan/2022:20:50:13 +0800 | 10.10.*.* | 414579208 | {"url":"POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=****************&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=******************************** HTTP/1.1","user-agent":"aliyun-sdk-java"} | 2022_01_23_20_50 | 200 |
   +------------+------------+-----------+-----------+-----------+-----------+------------------+--------------------+-----------+

Data model mapping

To ship logs from Log Service (SLS) to a MaxCompute (ODPS) table, you must map the data models of the two services. The following section describes the mapping rules and provides examples.

• A MaxCompute (ODPS) table must contain at least one data column and one partition key column.

• We recommend that you use the following Log Service (SLS) reserved fields: __partition_time__, __source__, and __topic__.

• A MaxCompute (ODPS) table can have a maximum of 60,000 partitions. If this limit is exceeded, data cannot be written to the table.

• The system reserved field __extract_others__ was previously named _extract_others_. The old name is still supported for compatibility.

• The values of MaxCompute (ODPS) partition key columns cannot be MaxCompute (ODPS) reserved words or keywords. For more information, see Reserved words and keywords.

• Partitions in a MaxCompute (ODPS) partitioned table cannot be empty. You must map a specific field, such as a system reserved field or a log field, to a partition key column. The value of the mapped field cannot be empty or null. If a log field is of the string type, it must be convertible to the data type of the partition key column. If the conversion fails and results in an empty partition key, the system discards the log during shipping.

• In Log Service (SLS), a log field can be mapped to only one column in a MaxCompute (ODPS) table, either a data column or a partition key column. Field redundancy is not supported.

Time partition format

Reference information

• The __partition_time__ field

  A common way to filter data in MaxCompute is to use log time as a partition key column.

  • Format

    The __partition_time__ field is derived from the __time__ field in Log Service (SLS). It generates a time string based on the time zone and partition time format. To avoid exceeding the partition limit for a single MaxCompute table, the date partition key value is aligned to 1800-second (half-hour) intervals.

    For example, if the log time in Log Service is 27/Jan/2022 20:50:13 +0800, SLS calculates the reserved field __time__ as 1643287813 (Unix timestamp). The following table shows the values of the time partition key column for different configurations.

    Partition time format	__partition_time__
    %Y_%m_%d_%H_%M_00	2022_01_27_20_30_00
    %Y_%m_%d_%H_%M	2022_01_27_20_30
    %Y%m%d	20220127

  • Usage

    Using __partition_time__ to filter data avoids a full-table scan. For example, to query log data for December 26, 2024:

    select * from {ODPS_TABLE_NAME} where log_partition_time >= "2024_12_26" and log_partition_time < "2024_12_27";

• The __receive_time__ field

  A common way to filter data in MaxCompute is to use the log arrival time as a partition key column.

  • Format

    Partition time format	__receive_time__
    %Y_%m_%d_%H_%M_00	2022_01_27_20_30_00
    %Y_%m_%d_%H_%M	2022_01_27_20_30
    %Y%m%d	20220127

• The __extract_others__ and __extract_others_all__ fields

  • The __extract_others__ field contains all unmapped fields from the log fields, excluding __topic__, __tag__:*, and __source__.

  • The __extract_others_all__ field contains all unmapped fields from the log fields, including __topic__, __tag__:*, and __source__.

Shipping modes

SDK examples

export_odps_sink_demo.py