Diagnose and govern node issues

Configuration settings

Configure the issue diagnosis feature based on your preferences. All configuration settings apply only to the current logged-in account.

1. Go to the DataWorks workspace list page, find the target workspace, and click Actions > Quick Access > Data Studio.

2. Click > Setup at the bottom of the left-side navigation pane, and go to the User tab of the Setup page.

Built-in rule library management

The check behavior of the Data Asset Governance plugin is driven by rule libraries, which are divided into built-in rules and custom rules. You can enable or disable specific rules as needed.

1. In the left-side navigation pane of the Data Studio page, click the governance icon to open the Data Asset Governance plugin configuration.

2. In the Built-in Rule Library section of the Data Asset Governance plugin panel, find the target governance rule and click the toggle icon next to it to enable or disable the rule.

   • Enabled (default): The rule is checked when you save a node.

   • Disabled: The rule is not checked when you save a node.

Custom rule library

When built-in rules cannot meet the specific business logic or coding standards of your team, the Data Asset Governance plugin provides a custom rule library. You can define new check rules by using natural language descriptions and positive/negative examples.

Deep scan

In addition to the single-file scan mentioned in the Quick start section, batch deep scans are also supported.

1. In the left-side navigation pane of the Data Studio page, click the governance icon to open the Data Asset Governance plugin configuration.

2. In the In-depth Inspection section of the Data Asset Governance plugin panel, select the files to scan in batch and the rules to apply, and then initiate a deep scan.

   Important

   • The batch scan scope is limited to the file tabs currently open in the editor. You can select up to 5 files. Files that are not open cannot be selected.

   • Only one deep scan can be executed at a time during a batch scan.

   

3. In the In-depth Inspection results pane, click View Log in the upper-right corner to view the detailed judgment process of the large language model for the check rules.

Partitioned table queries must include partition filters

• Risk description: Querying a MaxCompute partitioned table without specifying a partition triggers a full table scan, which consumes significant compute resources and incurs high costs.

• Incorrect code example:

  SELECT user_id, order_amount
  FROM user_orders
  WHERE status = 'paid';

• Correct code example:

  SELECT user_id, order_amount
  FROM user_orders
  WHERE status = 'paid'
  AND pt = '${bizdate}'; -- Add partition filter condition

• Auto-fix logic: Quick fix is supported. The system automatically adds a partition condition to the WHERE clause, such as AND pt = '${bizdate}'.

INSERT INTO with rerun enabled is not allowed

• Risk description: If an SQL task contains only INSERT INTO logic and reruns are allowed in the schedule settings, each rerun appends data to the target table, which can easily cause data duplication and compromise data accuracy.

• Incorrect code example:

  -- Set the task property to "Rerunnable"
  INSERT INTO target_table SELECT * FROM source_table;

• Correct code example:

  -- Set the task property to "Rerunnable"
  INSERT OVERWRITE TABLE target_table SELECT * FROM source_table;

• Auto-fix logic: Quick fix is supported. The system changes INSERT INTO to INSERT OVERWRITE to ensure that data is overwritten rather than appended during reruns.

JOIN column types must be consistent

• Risk description: In MaxCompute SQL, inconsistent column types in JOIN operations cause implicit type conversions, which may lead to computation errors, performance degradation, and data quality issues.

• Incorrect code example:

  -- a.user_id is BIGINT, b.uid is STRING
  SELECT * FROM table_a a JOIN table_b b ON a.user_id = b.uid;

• Correct code example:

  -- a.user_id is BIGINT, b.uid is STRING
  SELECT * FROM table_a a JOIN table_b b ON a.user_id = CAST(b.uid AS BIGINT);

• Auto-fix logic: Quick fix is supported. The system automatically detects and applies an explicit type conversion using the CAST function on one of the columns to match the type of the other column.

INSERT into tables outside the current project

• Risk description: Writing data from a task in project A to a table in project B is a high-risk operation. This breaks project isolation and may lead to unauthorized data access and data leakage.

• Incorrect code example:

  -- Execute in a task within project_A
  INSERT INTO project_B.some_table SELECT * FROM my_table;

• Correct code example:

  -- Recommended approach: Data should be written by a task belonging to the table's owning project (project_B).
  -- Avoid this operation in project_A.

• Auto-fix logic: Auto-fix is not supported. You must adjust the data synchronization pipeline based on your business requirements to ensure that data is written by the target project's own tasks.

Scheduled tasks must not write to development environment tables

• Risk description: When scheduled tasks in the production environment write data to tables in the development environment, the data protection level is reduced, creating data security risks.

• Incorrect code example:

  -- Execute in a task in the production environment (PROD)
  INSERT OVERWRITE TABLE user_dev.temp_data SELECT * FROM user_prod.source_data;

• Correct code example:

  -- Production tasks should write to tables in the production environment
  INSERT OVERWRITE TABLE user_prod.result_data SELECT * FROM user_prod.source_data;

• Auto-fix logic: Auto-fix is not supported. You must manually modify the target table to ensure that data flows comply with environment isolation standards.

CREATE TABLE is not allowed in SQL

• Risk description: Using CREATE TABLE directly in a scheduled SQL task can cause unclear table ownership (tables are typically owned by the primary account or scheduling account), increase management costs, and create risks of accidental data deletion.

• Incorrect code example:

  CREATE TABLE my_temp_table (id INT);
  INSERT INTO my_temp_table VALUES (1);

• Correct code example:

  -- Tables should be pre-created in the DataWorks table management module, then used directly in SQL tasks.
  INSERT INTO my_temp_table VALUES (1);

• Auto-fix logic: Auto-fix is not supported. We recommend that you create tables in DataWorks metadata management, or use CREATE TABLE IF NOT EXISTS to avoid errors in certain scenarios.

Missing scheduling parameter checker

• Risk description: The core purpose of periodic scheduling tasks is to process data incrementally by time dimension. If a scheduling parameter (such as ${bizdate}) is missing from the WHERE condition, the task may process full data every day or process the wrong date range, resulting in data omission, inconsistency, and significant resource waste.

• Incorrect code example:

  -- Error: Missing scheduling parameter, unable to perform daily incremental processing
  INSERT OVERWRITE TABLE users_active_today PARTITION (pt = '${bizdate}')
  SELECT    user_id  FROM    login_log; -- Missing WHERE pt = '...'

• Correct code example:

  -- Correct: Use scheduling parameter as a filter condition to achieve daily incremental processing
  INSERT OVERWRITE TABLE users_active_today PARTITION (pt = '${bizdate}')
  SELECT    user_id FROM    login_log
  WHERE    pt = '${bizdate}';

• Auto-fix logic: Auto-fix is not fully supported. The plugin highlights the potential missing scheduling parameter filter, but due to the complexity of business logic, we recommend that you manually add the correct filter condition based on your actual requirements.