If the transformation rule of a Logstore also pulls data from another Logstore, data pull or update errors may occur. This topic describes these errors and the corresponding troubleshooting methods.

After reading data from a Logstore, the data transformation engine starts to transform log events in the Logstore. If the transformation rule involves data pulls from external resources such as Object Storage Service (OSS), ApsaraDB for RDS, and other Logstores, data pull or update errors may occur.

Data pull or update errors may have the following impacts:
  • Log events that conflict with the transformation rule will cause errors and be dropped. These log events will not be included in the transformation result.
  • Dropped log events will not be further transformed. The transformation task will continue to transform other log events.
  • If a log event is split into multiple log events and a single split is dropped, all associated splits will be dropped.
    Note These log event splits are associated in a tree structure before they are exported.

Error 1: A required parameter is missing

  • Transformation rule 
    e_table_map(res_log_logstore_pull(endpoint="cn-shenzhen.log.aliyuncs.com",ak_id="xxx",
        ak_secret="xxx",project="etl-test-shenzhen",
        fields=["__source__"]),field="processid",output_fields=["__source__"])
  • Error message 
    error when calling : res_log_logstore_pull\nDetail: res_log_logstore_pull() missing 1 required positional argument: 'logstore'", "requestId": "
  • Error cause

    The error message missing 1 required positional argument is displayed because the required parameter logstore is missing.

  • Troubleshooting method

    Check the orchestration syntax and specify the missing parameter.

Error 2: The delete_data parameter is not specified

  • Transformation rule 
    e_table_map(res_log_logstore_pull(endpoint="xx",ak_id="xxx",
        ak_secret="xxx",project="etl-test-shenzhen",logstore="rds-mysql-test",
        fields=["__source__"],primary_keys="cid"),field="processid",output_fields=["__source__"])
  • Error message 
    errorMessage\": \"when setting parameter primart_keys,need set delete_data\\nDetail: None\
  • Error cause

    This error occurs because the delete_data parameter is not specified when the primary_keys parameter is specified. The primary_keys and delete_data parameters must be set at the same time.

  • Troubleshooting method

    Specify the primary_keys and delete_data parameters, and then restart the data transformation task.

Error 3: The target Logstore does not exist

  • Transformation rule 
    e_table_map(res_log_logstore_pull(endpoint="xx",ak_id="xxx",
        ak_secret="xxx",project="etl-test-shenzhen",logstore="pull_logstore_test9900881",
        fields=["__source__"],primary_keys="cid"),field="processid",output_fields=["__source__"])
  • Error message 
    message:  fetch data get errors:{"errorCode": "LogStoreNotExist", "errorMessage": "logstore pull_logstore_test9900881 does not exist", "requestId": "5D7227AA269948500404B777"},retrytimes=2
  • Error cause

    This error occurs because the target Logstore does not exist.

  • Troubleshooting method

    Check the orchestration syntax and enter the correct Logstore name.

Error 4: The specified AccessKey pair is incorrect or invalid

  • Transformation rule 
    e_table_map(res_log_logstore_pull(
        endpoint="cn-hangzhou.log.aliyuncs.com",
        ak_id="xx",
        ak_secret="xx",
        project="sls-test",
        logstore="pull_logstore_test",
        fields=[("id", "new_id"), "name", "status"],
        from_time="begin"), "name", "new_id")
  • Error message 
    message:  fetch data get errors:{"errorCode": "SignatureNotMatch", "errorMessage": "signature gdaL/nWSRtve5FOB+QqHO/sBdnA= not match", "requestId": "5D760261ED35D40AA4AB1953"},retrytimes=1
    message:  fetch data get errors:{"errorCode": "Unauthorized", "errorMessage": "AccessKeyId not found: xx", "requestId": "5D7602A01808F9EAA6EB0E2B"},retrytimes=3
  • Error cause

    This error occurs because the ak_secret and ak_id parameters in the orchestration syntax do not match or the ak_id does not exist.

  • Troubleshooting method

    Set the ak_secret and ak_id parameters correctly, and then restart the data transformation task.

Error 5: The current RAM user does not have sufficient permissions

  • Transformation rule 
    e_table_map(res_log_logstore_pull(
        endpoint="cn-hangzhou.log.aliyuncs.com",
        ak_id="xx",
        ak_secret="xx",
        project="sls-test",
        logstore="pull_logstore_test",
        fields=[("id", "new_id"), "name", "status"],
        from_time="begin"), "name", "new_id")
  • Error message 
    message:  fetch data get errors:{"errorCode": "Unauthorized", "errorMessage": "denied by sts or ram, action: log:ListShards, resource: acs:log:cn-hangzhou:1654218965343050:...}
  • Error cause

    The Unauthorized error message is displayed because the current RAM user does not have sufficient read and write permissions on the target Logstore.

  • Troubleshooting method

    Grant sufficient permissions to the RAM user.

Error 6: The specified endpoint parameter is incorrect or invalid

  • Transformation rule 
    e_table_map(res_log_logstore_pull(
        endpoint="xxx",
        ak_id="xx",
        ak_secret="xx",
        project="sls-test",
        logstore="pull_logstore_test",
        fields=[("id", "new_id"), "name", "status"],
        from_time="begin"), "name", "new_id")
  • Error message 
    message:  fetch data get errors:{"errorCode": "ProjectNotExist", "errorMessage": "The Project does not exist : ali-sls-etl-regression-test", "requestId": "5D760AB12ECD0722AA1DD681"}
    message:  fetch data get errors:{"errorCode": "LogRequestError", "errorMessage": "HTTPConnectionPool(host='ali-sls-etl-regression-test.xx', port=80): Max retries exceeded with url: /logstores/pull_logstore_test/shards (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f968a298ef0>: Failed to establish a new connection: [Errno -2] Name or service not known
  • Error cause

    The ProjectNotExist or LogRequestError error message is displayed because the specified endpoint is incorrect or invalid or the required project does not exist in the specified endpoint.

  • Troubleshooting method

    Correct the endpoint settings. For more information, see Service endpoint.