This topic describes the causes of data pull errors during data transformation and their corresponding resolutions.
If the transformation rule involves data pulls from external resources such as ApsaraDB for RDS and other Logstores, data pull or update errors may occur.
- If the data of an external resource fails to be pulled or refreshed, the corresponding data transformation task retries the data pull until a retry succeeds or is manually stopped. After a retry succeeds, the data transformation task continues.
- The data transformation task saves the breakpoint and retries the data pull. After a retry succeeds, the data pull restarts from the breakpoint to avoid data loss or redundancy.
Error impact
- If the data of an external resource fails to be pulled or refreshed, the corresponding data transformation task retries the data pull until a retry succeeds or is manually stopped. After a retry succeeds, the data transformation task continues.
- The data transformation task saves the breakpoint and retries the data pull. After a retry succeeds, the data pull restarts from the breakpoint to avoid data loss or redundancy.
Error handling
For more information about error handling, see res_log_logstore_pull, res_rds_mysql, and res_oss_file.
Solutions
- Transformation rule
res_log_logstore_pull(endpoint="cn-shenzhen.log.aliyuncs.com",ak_id="xxx", ak_secret="xxx",project="etl-test-shenzhen", fields=["__source__"]),field="processid",output_fields=["xx"]
res_rds_mysql(address="xx",username="xx",password="xx",database="xx")
res_oss_file(endpoint='xx',ak_id="xx",ak_key="xx",bucket='xx', file='xx',format='xx',change_detect_interval=0)
- Error message
aliyun.log.logexception.LogException: {"errorCode": "InvalidEtlConfig", "errorMessage": "ETL config doesn't pass security check, detail: invalid type detected: <class '_ast.Expr'>", "requestId": ""}
- Cause
This error occurs because the syntax is incorrect. The error may occur when you use the
res_log_logstore_pull
,res_rds_mysql
, orres_oss_file
function alone in the console. - Resolution
Use the resource function together with the
e_table_map
ore_search_table_map
function.
Reference
- For information about how to fix errors that may occur in data pulls from other Logstores, see How can I fix errors that occur when I pull Logstore data (dimension table)?
- For information about how to fix errors that may occur in data pulls from OSS, see How can I fix errors that occur during data pulls from OSS?
- For information about how to fix errors that may occur in data pulls from ApsaraDB for RDS, see How can I fix errors of the syntax used to pull data from ApsaraDB RDS for MySQL?