How to troubleshoot MaxCompute data shipping jobs (new version) - Simple Log Service

"User does not exist" error

If the execution record of a MaxCompute data shipping job (new version) shows the error ODPS-0420095: Access Denied - Authorization Failed [4002], You don't exist in project ****. Context ID:11111-1111-111-1111-11111111. --->Tips: Pricipal:INVALID$v4_11111111111; You don't exist in project ****, it indicates that the target user was not added, or was added incorrectly, to the MaxCompute workspace. To resolve this issue, add the user again and complete the authorization. For more information, see Authorize the default role to write data to MaxCompute (for data shipping within the same account) or Authorize a custom role to write data to MaxCompute (for data shipping within the same account).

"RAM role has no permission" error

If the execution record of a MaxCompute data shipping job (new version) shows the error ODPS-0420095: Access Denied - Authorization Failed [4019], You have NO privilege 'odps:Describe' on {acs:odps:*:projects/****/tables/****}. Context ID:111111-1111-11111-1111-1111111. --->Tips: Pricipal:INVALID$v4_111111111, it indicates that the RAM role lacks the required permissions when it was added as a member of the MaxCompute workspace. To resolve this, grant the required permissions to the RAM role again. For more information, see Authorize the default role to write data to MaxCompute (for data shipping within the same account) or Authorize a custom role to write data to MaxCompute (for data shipping within the same account).

`FAILED: mismatched input` error

If you are a RAM user and encounter the FAILED: mismatched input error when adding a RAM role as a member of a MaxCompute workspace, it indicates that your account lacks the permissions to execute statements such as ADD USER. To resolve this, use an Alibaba Cloud account or a RAM user with the following policy to execute statements such as ADD USER. After this one-time operation, you can use the RAM user to execute statements like ADD USER without encountering this error.

{
  "Version": "1",
  "Statement": [
    {
      "Action": "ram:CreateServiceLinkedRole",
      "Resource": "*",
      "Effect": "Allow",
      "Condition": {
        "StringEquals": {
          "ram:ServiceName": "usermgmt.odps.aliyuncs.com"
        }
      }
    }
  ]
}

Insufficient user permissions

If the current user encounters the error FAILED: ODPS-0130013:Authorization exception - Authorization Failed [4003], You have NO privilege to do the PROJECT SECURITY OPERATION for {acs:odps:*:projects/****/authorization/users}. Context ID:1111-11111-1111-1111-11111. when adding a RAM role as a member of a MaxCompute workspace, it indicates that the user lacks the permissions required for project security operations in MaxCompute. To resolve this, assign the admin role to the user. For more information, see Relationship between MaxCompute and DataWorks permissions.

Follow these steps to assign the admin role to a user.

Log on to the MaxCompute console.
In the upper-left corner of the page, select a region.
On the Project Management page, find the target project and click Manage in the Actions column.
Click the Role Permissions tab.
Find the admin role and click Manage Members in the Actions column.
In the Manage Members panel, enter the account name in the Account name to add field.
In the Manage Members dialog box, click OK. Select the target account, click the right arrow (>) to move it to the Added Accounts list, and then click OK.

Partition by log time

When you create a MaxCompute data shipping job (new version), you can set the MaxCompute Partition Key Column to __partition_time__ to partition data by log time. In the MaxCompute partition key column configuration, __partition_time__ and __receive_time__ are system-defined partition fields calculated from the log time and receive time, respectively. You can also click the plus icon (+) to add custom partition key columns and map them to existing fields in Simple Log Service, such as http_method or operation.

The __partition_time__ field is calculated based on the value of the __time__ field in Simple Log Service. A time string is generated based on the specified time zone and partition time format. To prevent exceeding the single-table partition limit in MaxCompute, MaxCompute aligns values for date-based partition key columns to 1,800-second (30-minute) intervals. For more information, see __partition_time__ field.

View MaxCompute table partitions

Run the show partitions <table-name>; command to view a table's partitions. In this command, replace <table-name> with the name of your table.

Ship log time data

When you create a MaxCompute data shipping job (new version), you can add the __time__ field in the MaxCompute Common Column section. The __time__ field represents the log time in UNIX timestamp format, accurate to the second.

For example, you can map the __time__ field to a destination column in your MaxCompute table, such as c14.

Null field values

This issue is typically caused by a data type mismatch, a type conversion failure, or a missing field in the source log.

To handle data type mismatches or conversion failures, see Handle dirty data.
For information about data type format requirements, see Stability and limits.

Job failure after enabling an IP whitelist

If you enable an IP address whitelist for a MaxCompute project (for example, a classic network IP address whitelist), a data shipping job may fail with an error message similar to the following:

error:RequestId=202211181752058538dc0b0018****, ErrorCode=NoPermission, ErrorMessage=Access denied by project ip white list: sourceIP:'192.168.x.x' is not in white list. project: slstest-1 from:
errorType:odps.OdpsError
jobId:****-****

To resolve failures caused by the IP address whitelist, run the following commands in MaxCompute:

Log on to the MaxCompute console .
In the upper-left corner of the page, select the region where your project is located.
In the left-side navigation pane of MaxCompute, choose DataWorks > Data Development.

In DataWorks, perform the following steps. For more information about how to use DataWorks, see Advanced: Analyze best-selling product categories.

Query the current IP address whitelist configuration.
```
setproject; 
select 1;
```
Append AliyunLogSLRService,AliyunLogDefaultService to the existing service whitelist by using the following command:
```
setproject odps.security.ip.whitelist.services=AliyunLogSLRService,AliyunLogDefaultService;
```

After the configuration is complete, run the following commands to verify the new whitelist settings.

setproject; 
select 1;

After you run the setproject command, the system returns the project configuration properties. Check the values of the odps.security.ip.whitelist.services and odps.security.outbound.destination properties:

odps.output.field.formatter=
odps.project.longtimetmpfile.max.remain.days=30
odps.project.longtimetmpfile.min.remain.days=3
odps.project.maintenance.alarm={"storage.warning.ratio":"85","storage.warning.receiver":"","storage.critical.receiver":"","storage.critical.ratio":"95"}
odps.project.maintenance.window=
odps.schema.evolution.enable=false
odps.schema.model.enabled=false
odps.security.enabledownloadprivilege=false
odps.security.ip.whitelist=
odps.security.ip.whitelist.services=AliyunLogSLRService,AliyunLogDefaultService
odps.security.outbound.destination=
odps.security.vpc.whitelist=
odps.session.flag.authority=false
odps.spark.kube.mode=false
odps.sql.acid.table.enable=true
odps.sql.allow.cartesian=false
odps.sql.allow.fullscan=false
odps.sql.auto.mv.quota.gb=-1
odps.sql.cfile2.field.maxsize=8192