Data transformation engine startup errors

When the data transformation engine fails to start, it typically means your SLS DSL rules contain errors that did not pass the engine's security check. This topic describes the most common startup errors and how to fix them.

The data transformation engine runs a security check on your SLS DSL rules at the start of each transformation task. If the check fails, the engine does not start. 加工引擎启动

Error log

When the engine detects an error during startup, it returns a message in the following format.

{
  "errorMessage": "ETL config doesn't pass security check, detail: XXXXXX"
}

Note

View the error log in the exception details section of the data transformation diagnostic report or in the internal-etl-log Logstore.

If a startup error occurs, the task retries continuously until it succeeds or you stop it manually.
After you fix the processing rule, the task succeeds on its next retry and resumes normal operation. No logs are lost or duplicated.

Troubleshoot common errors

Basic syntax error.

This error occurs when a processing rule violates SLS DSL grammar — for example, mismatched parentheses or a colon (:) used instead of a comma (,).

Error log

{
  "errorMessage": "ETL config doesn't pass security check, detail: invalid syntax"
}
{
  "errorMessage": "ETL config doesn't pass security check, detail: unexpected EOF while parsing"
}

Troubleshooting

Use the traceback information in the error log to find the exact location of the syntax error. For example, the following error occurs when e_set("test": v("status")) is used instead of e_set("test", v("status")).

__source__:  172.20.3.124
__tag__:__job_name__:  etl-preview-1563260078-834525
__tag__:__schedule_type__:  DryRun
__topic__:
event_id : transform_data:init:fail
logging : {"processName": "MainProcess", "threadName": "MainThread", "thread": "140462572971840", "funcName": "transform_data", "process": "1", "module": "logclient_operator", "levelname": "ERROR"}
message :  ETL config doesn't pass security check, detail: invalid syntax (<unknown>, line 5)
Traceback (most recent call last):
File "/usr/local/python-etl/aliyunlogetl_common/core/restrict_config_parser.py", line 100, in security_check_config
RestrictConfigParser().parse(code)
File "/usr/local/python-etl/aliyunlogetl_common/core/restrict_config_parser.py", line 88, in parse
self.visit(ast.parse(code))
File "/usr/local/lib/python3.7/ast.py", line 35, in parse
return compile(source, filename, mode, PyCF_ONLY_AST)
File "<unknown>", line 5
TRANSFORM_ANY_1563260645 = e_set("test": v("status"))
SyntaxError: invalid syntax
 reason :  ETL config doesn't pass security check, detail: invalid syntax (<unknown>, line 5)

Invalid use of an operator.

SLS DSL requires built-in functions — such as those prefixed with op_* — for all operations, including numerical calculations and comparisons. Direct use of standard operators is not allowed.
- Error log
```
{
  "errorMessage": "ETL config doesn't pass security check, detail: invalid type detected: <class `_ast.BinOp`> "
}
```
- Troubleshooting
  
  Review your SLS DSL processing rules and replace any direct operator usage with the equivalent SLS DSL built-in functions.
- Examples
```
e_set("b", v("a") - 10) # Incorrect example
e_set("b", op_sub(v("a"), 10)) # Correct example
e_set("b", v("a") >= v("c")) # Incorrect example
e_set("b", op_ge(v("a"), v("c"))) # Correct example
```
Incorrect parameter type or call to a non-existent function.
This error occurs when an argument's data type does not match the function's expected parameter type, or when you call a function that does not exist.
- Error log
```
{
  "errorMessage": "ETL config doesn't pass security check, detail: invalid call in detected: function_name"
}
```
- Troubleshooting
  - Verify that the function exists and that its name is spelled correctly. If the name is correct, check that its arguments have the correct data types.
  - Use the traceback information from the error log to identify the function that caused the error. For example, if the error originates from the dt_totimestamp function, first confirm that the function exists. Then, review the rule where dt_totimestamp is called to make sure you are passing the correct parameter types.
```
File "/usr/local/lib/python3.7/ast.py", line 262, in visit
return visitor(node)
File "/usr/local/python-etl/aliyunlogetl_common/core/restrict_config_parser.py", line 50, in visit_Call
raise InvalidETLConfig("invalid call id detected: {}".format(node.func.id))
aliyunlogetl_common.core.restrict_config_parser.InvalidETLConfig: invalid call id detected: dt_totimestamp
reason : ETL config doesn't pass security check, detail: invalid call id detected: dt_totimestamp
```
- Examples
  
  The dt_totimestamp function expects a datetime object. Here, v("time1") returns a string, causing a type mismatch error.
  
  To fix this, use dt_parse to convert the string to a datetime object before passing it to dt_totimestamp. Alternatively, use dt_parsetimestamp, which accepts string arguments directly.
```
# Incorrect example
e_set("time1", "2019-06-03 2:41:26")
e_set("time2", dt_totimestamp(v("time1")))
# Correct example
e_set("time1", "2019-06-03 2:41:26")
e_set("time2", dt_totimestamp(dt_parse(v("time1"))))
# Correct example
e_set("time1", "2019-06-03 2:41:26")
e_set("time2", dt_parsetimestamp(v("time1")))
```
Calling an expression function as a global operation.
SLS DSL provides two types of functions: global operation functions and expression functions. Only global operation functions can be called as standalone steps in a processing rule. Calling an expression function directly as a global operation causes an error.
- Error log
```
{
  "errorMessage": "ETL config doesn't pass security check, detail:  invalid type detected: <class '_ast.Expr'>"
}
```
- Troubleshooting
  
  Check your processing rules to make sure no expression functions are called as standalone global operations. Wrap them inside a global operation function such as e_set.
- Examples
```
# Incorrect example
op_add(v("a"), v("b"))
str_lower(v("name"))
# Correct example
e_set("add", op_add(v("a"), v("b")))
e_set("lower", str_lower(v("name")))
```
Invalid variable assignment.
SLS DSL does not support variable assignment. Pass values between functions statelessly.
- Error log
```
{
  "errorMessage": "ETL config doesn't pass security check, detail: invalid assign detected: variable_name"
}
```
- Troubleshooting
  - Check your SLS DSL processing rules for any variable assignments.
  - Use the traceback information in the error log to locate the line where the error occurs.
- Examples
```
# Incorrect example
sum_value = op_add(v("a"), v("b"))
e_set("sum", sum_value)
# Correct example
e_set("sum", op_add(v("a"), v("b")))
```