If the transformation rule of a Logstore also pulls data from another Logstore, data pull or update errors may occur. This topic describes these errors and the corresponding troubleshooting methods.
After reading data from a Logstore, the data transformation engine starts to transform log events in the Logstore. If the transformation rule involves data pulls from external resources such as Object Storage Service (OSS), ApsaraDB for RDS, and other Logstores, data pull or update errors may occur.
- Log events that conflict with the transformation rule will cause errors and be dropped. These log events will not be included in the transformation result.
- Dropped log events will not be further transformed. The transformation task will continue to transform other log events.
- If a log event is split into multiple log events and a single split is dropped, all
associated splits will be dropped.
Note These log event splits are associated in a tree structure before they are exported.
Error 1: A required parameter is missing
- Transformation rule
e_table_map(res_log_logstore_pull(endpoint="cn-shenzhen.log.aliyuncs.com",ak_id="xxx", ak_secret="xxx",project="etl-test-shenzhen", fields=["__source__"]),field="processid",output_fields=["__source__"])
- Error message
error when calling : res_log_logstore_pull\nDetail: res_log_logstore_pull() missing 1 required positional argument: 'logstore'", "requestId": "
- Error cause
The error message
missing 1 required positional argument
is displayed because the required parameterlogstore
is missing. - Troubleshooting method
Check the orchestration syntax and specify the missing parameter.
Error 2: The delete_data parameter is not specified
- Transformation rule
e_table_map(res_log_logstore_pull(endpoint="xx",ak_id="xxx", ak_secret="xxx",project="etl-test-shenzhen",logstore="rds-mysql-test", fields=["__source__"],primary_keys="cid"),field="processid",output_fields=["__source__"])
- Error message
errorMessage\": \"when setting parameter primart_keys,need set delete_data\\nDetail: None\
- Error cause
This error occurs because the
delete_data
parameter is not specified when theprimary_keys
parameter is specified. Theprimary_keys
anddelete_data
parameters must be set at the same time. - Troubleshooting method
Specify the primary_keys and delete_data parameters, and then restart the data transformation task.
Error 3: The target Logstore does not exist
- Transformation rule
e_table_map(res_log_logstore_pull(endpoint="xx",ak_id="xxx", ak_secret="xxx",project="etl-test-shenzhen",logstore="pull_logstore_test9900881", fields=["__source__"],primary_keys="cid"),field="processid",output_fields=["__source__"])
- Error message
message: fetch data get errors:{"errorCode": "LogStoreNotExist", "errorMessage": "logstore pull_logstore_test9900881 does not exist", "requestId": "5D7227AA269948500404B777"},retrytimes=2
- Error cause
This error occurs because the target Logstore does not exist.
- Troubleshooting method
Check the orchestration syntax and enter the correct Logstore name.
Error 4: The specified AccessKey pair is incorrect or invalid
- Transformation rule
e_table_map(res_log_logstore_pull( endpoint="cn-hangzhou.log.aliyuncs.com", ak_id="xx", ak_secret="xx", project="sls-test", logstore="pull_logstore_test", fields=[("id", "new_id"), "name", "status"], from_time="begin"), "name", "new_id")
- Error message
message: fetch data get errors:{"errorCode": "SignatureNotMatch", "errorMessage": "signature gdaL/nWSRtve5FOB+QqHO/sBdnA= not match", "requestId": "5D760261ED35D40AA4AB1953"},retrytimes=1
message: fetch data get errors:{"errorCode": "Unauthorized", "errorMessage": "AccessKeyId not found: xx", "requestId": "5D7602A01808F9EAA6EB0E2B"},retrytimes=3
- Error cause
This error occurs because the ak_secret and ak_id parameters in the orchestration syntax do not match or the ak_id does not exist.
- Troubleshooting method
Set the ak_secret and ak_id parameters correctly, and then restart the data transformation task.
Error 5: The current RAM user does not have sufficient permissions
- Transformation rule
e_table_map(res_log_logstore_pull( endpoint="cn-hangzhou.log.aliyuncs.com", ak_id="xx", ak_secret="xx", project="sls-test", logstore="pull_logstore_test", fields=[("id", "new_id"), "name", "status"], from_time="begin"), "name", "new_id")
- Error message
message: fetch data get errors:{"errorCode": "Unauthorized", "errorMessage": "denied by sts or ram, action: log:ListShards, resource: acs:log:cn-hangzhou:1654218965343050:...}
- Error cause
The
Unauthorized
error message is displayed because the current RAM user does not have sufficient read and write permissions on the target Logstore. - Troubleshooting method
Grant sufficient permissions to the RAM user.
Error 6: The specified endpoint parameter is incorrect or invalid
- Transformation rule
e_table_map(res_log_logstore_pull( endpoint="xxx", ak_id="xx", ak_secret="xx", project="sls-test", logstore="pull_logstore_test", fields=[("id", "new_id"), "name", "status"], from_time="begin"), "name", "new_id")
- Error message
message: fetch data get errors:{"errorCode": "ProjectNotExist", "errorMessage": "The Project does not exist : ali-sls-etl-regression-test", "requestId": "5D760AB12ECD0722AA1DD681"}
message: fetch data get errors:{"errorCode": "LogRequestError", "errorMessage": "HTTPConnectionPool(host='ali-sls-etl-regression-test.xx', port=80): Max retries exceeded with url: /logstores/pull_logstore_test/shards (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f968a298ef0>: Failed to establish a new connection: [Errno -2] Name or service not known
- Error cause
The
ProjectNotExist
orLogRequestError
error message is displayed because the specified endpoint is incorrect or invalid or the required project does not exist in the specified endpoint. - Troubleshooting method
Correct the endpoint settings. For more information, see Service endpoint.