Troubleshoot Logstore dimension table lookup errors - Simple Log Service

Identify and resolve errors that occur when a data transformation rule loads data from another Logstore as a dimension table.

When an error occurs during Logstore dimension table loading:

Error impact:

Log events that conflict with the transformation rule are discarded. The transformed output does not include these events.
The transformation job skips failed events and continues processing the remaining events. It does not retry failed events.
If a single source event is split into multiple events and one split event triggers an error, all events derived from the same source event are also discarded.

Note
Before output, split events are linked in a tree structure and are not independent.

Missing required parameter

Transformation rule

e_table_map(res_log_logstore_pull(endpoint="cn-shenzhen.log.aliyuncs.com",ak_id="xxx",
        ak_secret="xxx",project="etl-test-shenzhen",
        fields=["__source__"]),field="processid",output_fields=["__source__"])

Error message

error when calling : res_log_logstore_pull\nDetail: res_log_logstore_pull() missing 1 required positional argument: 'logstore'", "requestId": "

Cause

The missing 1 required positional argument message indicates that a required parameter is missing from the function call. In this example, the logstore parameter is not specified in res_log_logstore_pull.
Resolution

Check the error message to identify the missing parameter. Add the missing parameter to the transformation rule, and then save and restart the data transformation job.

primary_keys specified without delete_data

Transformation rule

e_table_map(res_log_logstore_pull(endpoint="xx",ak_id="xxx",
        ak_secret="xxx",project="etl-test-shenzhen",logstore="rds-mysql-test",
        fields=["__source__"],primary_keys="cid"),field="processid",output_fields=["__source__"])

Error message

when setting parameter primart_keys,need set delete_data\nDetail: None

Cause

The primary_keys and delete_data parameters are interdependent. Setting primary_keys without delete_data is not allowed.
Resolution

Add the delete_data parameter alongside primary_keys in the transformation rule. Save and restart the data transformation job.

Logstore does not exist (LogStoreNotExist)

Transformation rule

e_table_map(res_log_logstore_pull(endpoint="xx",ak_id="xxx",
        ak_secret="xxx",project="etl-test-shenzhen",logstore="pull_logstore_test9900881",
        fields=["__source__"],primary_keys="cid"),field="processid",output_fields=["__source__"])

Error message

message:  fetch data get errors:{"errorCode": "LogStoreNotExist", "errorMessage": "logstore pull_logstore_test9900881 does not exist", "requestId": "5D7227AA269948500404B777"},retrytimes=2

Cause

The specified Logstore does not exist in the target project. The Logstore name may be misspelled or the Logstore may have been deleted.
Resolution

Verify that the target Logstore exists in the Simple Log Service console. Correct the Logstore name in the transformation rule, and then save and restart the data transformation job.

Incorrect AccessKey (SignatureNotMatch / Unauthorized)

Transformation rule

e_table_map(res_log_logstore_pull(
        endpoint="cn-hangzhou.log.aliyuncs.com",
        ak_id="xx",
        ak_secret="xx",
        project="sls-test",
        logstore="pull_logstore_test",
        fields=[("id", "new_id"), "name", "status"],
        from_time="begin"), "name", "new_id")

Error message

message:  fetch data get errors:{"errorCode": "SignatureNotMatch", "errorMessage": "signature gdaL/nWSRtve5FOB+QqHO/sBdnA= not match", "requestId": "5D760261ED35D40AA4AB1953"},retrytimes=1

message:  fetch data get errors:{"errorCode": "Unauthorized", "errorMessage": "AccessKeyId not found: xx", "requestId": "5D7602A01808F9EAA6EB0E2B"},retrytimes=3

Cause

The AccessKey pair configured in the transformation rule is invalid. Either the `ak_id` value does not exist, or the `ak_id` and `ak_secret` values do not match.
Resolution

Verify that the AccessKey ID exists in the RAM console and that the AccessKey Secret matches the AccessKey ID. Update the `ak_id` and `ak_secret` parameters in the transformation rule, and then save and restart the data transformation job.

Insufficient AccessKey permissions (Unauthorized)

Transformation rule

e_table_map(res_log_logstore_pull(
        endpoint="cn-hangzhou.log.aliyuncs.com",
        ak_id="xx",
        ak_secret="xx",
        project="sls-test",
        logstore="pull_logstore_test",
        fields=[("id", "new_id"), "name", "status"],
        from_time="begin"), "name", "new_id")

Error message

message:  fetch data get errors:{"errorCode": "Unauthorized", "errorMessage": "denied by sts or ram, action: log:ListShards, resource: acs:log:cn-hangzhou:1654218965343050:...}

Cause

The AccessKey pair lacks the required permissions to access the target Logstore. The error message specifies the denied action and the target resource ARN.
Resolution

Identify the required permission from the action field in the error message. In the RAM console, attach a policy that grants the required permissions on the target Logstore to the RAM user or role associated with the AccessKey pair, and then restart the data transformation job.

Incorrect endpoint (ProjectNotExist / LogRequestError)

Transformation rule

e_table_map(res_log_logstore_pull(
        endpoint="xxx",
        ak_id="xx",
        ak_secret="xx",
        project="sls-test",
        logstore="pull_logstore_test",
        fields=[("id", "new_id"), "name", "status"],
        from_time="begin"), "name", "new_id")

Error message

message:  fetch data get errors:{"errorCode": "ProjectNotExist", "errorMessage": "The Project does not exist : ali-sls-etl-regression-test", "requestId": "5D760AB12ECD0722AA1DD681"}

message:  fetch data get errors:{"errorCode": "LogRequestError", "errorMessage": "HTTPConnectionPool(host='ali-sls-etl-regression-test.xx', port=80): Max retries exceeded with url: /logstores/pull_logstore_test/shards (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f968a298ef0>: Failed to establish a new connection: [Errno -2] Name or service not known))

Cause

The ProjectNotExist or LogRequestError errors occur because the endpoint parameter value is incorrect. An incorrect endpoint causes one of two failures: the hostname cannot be resolved, resulting in a connection failure, or the connection succeeds but the project does not exist at the resolved endpoint.
Resolution

Verify the correct endpoint for the target region. For more information, see Endpoints. Update the endpoint parameter in the transformation rule, and then save and restart the data transformation job.