This topic describes the causes of data pull errors during data transformation and their corresponding resolutions.

If the transformation rule involves data pulls from external resources such as ApsaraDB for RDS and other Logstores, data pull or update errors may occur.

Error impact:
  • If the data of an external resource fails to be pulled or refreshed, the corresponding data transformation task retries the data pull until a retry succeeds or is manually stopped. After a retry succeeds, the data transformation task continues.
  • The data transformation task saves the breakpoint and retries the data pull. After a retry succeeds, the data pull restarts from the breakpoint to avoid data loss or redundancy.

Error impact

  • If the data of an external resource fails to be pulled or refreshed, the corresponding data transformation task retries the data pull until a retry succeeds or is manually stopped. After a retry succeeds, the data transformation task continues.
  • The data transformation task saves the breakpoint and retries the data pull. After a retry succeeds, the data pull restarts from the breakpoint to avoid data loss or redundancy.

Error handling

For more information about error handling, see res_log_logstore_pull, res_rds_mysql, and res_oss_file.

Solutions

The following example demonstrates how to fix an error caused by incorrect use of a resource function in the console.
  • Transformation rule 
    res_log_logstore_pull(endpoint="cn-shenzhen.log.aliyuncs.com",ak_id="xxx",
        ak_secret="xxx",project="etl-test-shenzhen",
        fields=["__source__"]),field="processid",output_fields=["xx"]
    res_rds_mysql(address="xx",username="xx",password="xx",database="xx")
    res_oss_file(endpoint='xx',ak_id="xx",ak_key="xx",bucket='xx', file='xx',format='xx',change_detect_interval=0)
  • Error message 
    aliyun.log.logexception.LogException: {"errorCode": "InvalidEtlConfig", "errorMessage": "ETL config doesn't pass security check, detail: invalid type detected: <class '_ast.Expr'>", "requestId": ""}
  • Cause

    This error occurs because the syntax is incorrect. The error may occur when you use the res_log_logstore_pull, res_rds_mysql, or res_oss_file function alone in the console.

  • Resolution

    Use the resource function together with the e_table_map or e_search_table_map function.

Reference