Simple Log Service:MaxCompute data shipping (new version): Common errors

"User does not exist" error

If the execution record of a MaxCompute data shipping job (new version) shows the error ODPS-0420095: Access Denied - Authorization Failed [4002], You don't exist in project ****. Context ID:11111-1111-111-1111-11111111. --->Tips: Pricipal:INVALID$v4_11111111111; You don't exist in project ****, it indicates that the target user was not added, or was added incorrectly, to the MaxCompute workspace. To resolve this issue, add the user again and complete the authorization. For more information, see Authorize the default role to write data to MaxCompute (for data shipping within the same account) or Authorize a custom role to write data to MaxCompute (for data shipping within the same account).

"RAM role has no permission" error

If the execution record of a MaxCompute data shipping job (new version) shows the error ODPS-0420095: Access Denied - Authorization Failed [4019], You have NO privilege 'odps:Describe' on {acs:odps:*:projects/****/tables/****}. Context ID:111111-1111-11111-1111-1111111. --->Tips: Pricipal:INVALID$v4_111111111, it indicates that the RAM role lacks the required permissions when it was added as a member of the MaxCompute workspace. To resolve this, grant the required permissions to the RAM role again. For more information, see Authorize the default role to write data to MaxCompute (for data shipping within the same account) or Authorize a custom role to write data to MaxCompute (for data shipping within the same account).

FAILED: mismatched input error

{
  "Version": "1",
  "Statement": [
    {
      "Action": "ram:CreateServiceLinkedRole",
      "Resource": "*",
      "Effect": "Allow",
      "Condition": {
        "StringEquals": {
          "ram:ServiceName": "usermgmt.odps.aliyuncs.com"
        }
      }
    }
  ]
}

Insufficient user permissions

If the current user encounters the error FAILED: ODPS-0130013:Authorization exception - Authorization Failed [4003], You have NO privilege to do the PROJECT SECURITY OPERATION for {acs:odps:*:projects/****/authorization/users}. Context ID:1111-11111-1111-1111-11111. when adding a RAM role as a member of a MaxCompute workspace, it indicates that the user lacks the permissions required for project security operations in MaxCompute. To resolve this, assign the admin role to the user. For more information, see Relationship between MaxCompute and DataWorks permissions.

Follow these steps to assign the admin role to a user.

1. Log on to the MaxCompute console.

2. In the upper-left corner of the page, select a region.

3. On the Project Management page, find the target project and click Manage in the Actions column.

4. Click the Role Permissions tab.

5. Find the admin role and click Manage Members in the Actions column.

6. In the Manage Members panel, enter the account name in the Account name to add field.

7. In the Manage Members dialog box, click OK. Select the target account, click the right arrow (>) to move it to the Added Accounts list, and then click OK.

"SlotExceeded" and "io: read/write on closed pipe" errors

If the job execution records of a MaxCompute data shipping job (new version) show one of the following errors, the current data shipping concurrency may exceed the MaxCompute slot quota:

SlotExceeded: Your slot quota is exceeded. Region: cn-shanghai Project: xxx Slot Quota Exceeded. Total: xxx, Acquired: xxx
maxcompute error ClientError: io: read/write on closed pipe

To reduce the chance of this issue, use __receive_time__ as the partitioning field. This field is based on the server-side receive time. Logs in the same receive time window are written to the same partition, which helps avoid concurrent writes to multiple partitions and can reduce slot consumption. You can change the partitioning field when you create a MaxCompute data shipping job (new version).

Partition by log time

The __partition_time__ field is calculated based on the value of the __time__ field in Simple Log Service. A time string is generated based on the specified time zone and partition time format. To prevent exceeding the single-table partition limit in MaxCompute, MaxCompute aligns values for date-based partition key columns to 1,800-second (30-minute) intervals. For more information, see __partition_time__ field.

MaxCompute partition columns: __partition_time__ and __receive_time__

__partition_time__ is based on log time. The value of __partition_time__ is calculated from the __time__ field of a log. Because __time__ comes from the log, the target partition can be affected by collection latency and client clock skew.

If logs in the same receive time window are distributed to multiple partitions, the following issues may occur:

• Historical partitions continue to receive late-arriving data, which makes it difficult to determine whether a partition is complete.

• Future partitions may appear earlier than expected because logs with abnormal client timestamps are written to future partition ranges.

• Downstream task scheduling becomes harder because partition boundaries are unclear and data lifecycle management becomes more complex.

__receive_time__ is based on the server-side receive time. Simple Log Service writes this field when it receives logs, so the value is not affected by collection latency, client clock skew, or log time parsing exceptions. In the preceding examples, all three logs are received at 10:00 and are written to p=10:00.

Using __receive_time__ provides the following benefits:

• Logs in the same receive time window are written to the same partition, and partition boundaries remain clear.

• Partition progress is consistent with the actual data shipping progress, which makes partition completeness easier to determine.

• Downstream task scheduling is more predictable because historical partitions are less likely to be repeatedly backfilled.

It is recommended to use __receive_time__ as the partitioning field. In real-time data scenarios, __partition_time__ and __receive_time__ usually have no significant difference. In production environments with delayed data or client clock skew, __receive_time__ provides more stable partitioning behavior.

View MaxCompute table partitions

Run the show partitions <table-name>; command to view a table's partitions. In this command, replace <table-name> with the name of your table.

Ship log time data

When you create a MaxCompute data shipping job (new version), you can add the __time__ field in the MaxCompute Common Column section. The __time__ field represents the log time in UNIX timestamp format, accurate to the second.

For example, you can map the __time__ field to a destination column in your MaxCompute table, such as c14.

Null field values

This issue is typically caused by a data type mismatch, a type conversion failure, or a missing field in the source log.

• To handle data type mismatches or conversion failures, see Handle dirty data.

• For information about data type format requirements, see Stability and limits.

Job failure after enabling an IP whitelist

If you enable an IP address whitelist for a MaxCompute project (for example, a classic network IP address whitelist), a data shipping job may fail with an error message similar to the following:

error:RequestId=202211181752058538dc0b0018****, ErrorCode=NoPermission, ErrorMessage=Access denied by project ip white list: sourceIP:'192.168.x.x' is not in white list. project: slstest-1 from:
errorType:odps.OdpsError
jobId:****-****

To resolve failures caused by the IP address whitelist, run the following commands in MaxCompute:

1. Log on to the MaxCompute console .

2. In the upper-left corner of the page, select the region where your project is located.

3. In the left-side navigation pane of MaxCompute, choose DataWorks > Data Development.

4. In DataWorks, perform the following steps. For more information about how to use DataWorks, see Advanced: Analyze best-selling product categories.

   1. Query the current IP address whitelist configuration.

      setproject; 
      select 1;

   2. Append AliyunLogSLRService,AliyunLogDefaultService to the existing service whitelist by using the following command:

      setproject odps.security.ip.whitelist.services=AliyunLogSLRService,AliyunLogDefaultService;

   3. After the configuration is complete, run the following commands to verify the new whitelist settings.

      setproject; 
      select 1;

      After you run the setproject command, the system returns the project configuration properties. Check the values of the odps.security.ip.whitelist.services and odps.security.outbound.destination properties:

      odps.output.field.formatter=
      odps.project.longtimetmpfile.max.remain.days=30
      odps.project.longtimetmpfile.min.remain.days=3
      odps.project.maintenance.alarm={"storage.warning.ratio":"85","storage.warning.receiver":"","storage.critical.receiver":"","storage.critical.ratio":"95"}
      odps.project.maintenance.window=
      odps.schema.evolution.enable=false
      odps.schema.model.enabled=false
      odps.security.enabledownloadprivilege=false
      odps.security.ip.whitelist=
      odps.security.ip.whitelist.services=AliyunLogSLRService,AliyunLogDefaultService
      odps.security.outbound.destination=
      odps.security.vpc.whitelist=
      odps.session.flag.authority=false
      odps.spark.kube.mode=false
      odps.sql.acid.table.enable=true
      odps.sql.allow.cartesian=false
      odps.sql.allow.fullscan=false
      odps.sql.auto.mv.quota.gb=-1
      odps.sql.cfile2.field.maxsize=8192