All Products
Search
Document Center

DataWorks:Create and manage custom rule templates

Last Updated:Jan 25, 2024

DataWorks Data Quality allows you to manage a set of custom rules in a centralized manner and create a self-managed rule template library to configure rules in a more efficient manner. This topic describes how to create a rule template on the Rule Templates and Monitoring Rules pages in the DataWorks console, and how to perform operations on a rule template.

Prerequisites

DataWorks Enterprise Edition or a more advanced edition is activated.

Background information

You can create a rule template on the Rule Templates and Monitoring Rules pages. After the rule template is created, you can manage and use it.

Precautions

The changes that are made to a custom rule take effect only for tasks that reference the changed custom rule. Existing tasks for which the original custom rule is applied are not affected.

Create a rule template on the Rule Templates page

  1. Go to the Data Quality page.

    Log on to the DataWorks console. In the left-side navigation pane, choose Data Modeling and Development > Data Quality. On the page that appears, select the desired workspace from the drop-down list and click Go to Data Quality.

  2. In the left-side navigation pane of the Data Quality page, choose Configuration > Rule Templates.

  3. Click the 新建 icon and select Create Folder.

  4. In the Create Folder dialog box, configure the Name and Location parameters and click OK.

  5. In the left-side directory tree, right-click the created folder and select Create Rule Template.

    You can also rename or delete a folder.

  6. In the Create Rule Template dialog box, configure the parameters.

    image.png

    Parameter

    Description

    Template Name

    The name of the custom rule template.

    Field

    The fields that you want to monitor. You can set this parameter only to Custom SQL.

    Sampling Method

    The sampling method for the custom rule template. You can set this parameter only to Custom SQL.

    Set Flag

    The SET clause of the SQL statement that is used to query the fields that you want to monitor.

    Note

    Separate multiple statements with commas (,). You do not need to add a semicolon (;) at the end of each statement.

    Check type

    The threshold type for the custom rule template. Valid values: Numeric type and Fluctuation.

    Verification Method

    The verification method for the custom rule template. The verification methods that can be selected vary based on the threshold type.

    • If you set the Check type parameter to Numeric type, you can set this parameter only to Compare with a specified value.

      Only the value that is calculated by the COUNT or SUM function can be returned. The return value is compared with a specified value.

    • If you set the Check type parameter to Fluctuation, the valid values of this parameter are Compare the current value with the average value of the last 7 days, Compare the current value with the average value of the last 30 days, Compare the current value with the value 1 day before, Compare the current value with the value 7 days before, Compare the current value with the value 30 days before, The variance between the current value and the value 7 days before, The variance between the current value and the value 30 days before, Compare with the value 1, 7, and 30 days before, and Compare with the value of the previous cycle.

    Custom SQL

    The SQL statement that is used to query the fields to be monitored. You can use ${tableName} to specify a table name.

    Note

    Make sure that the return value is the value in a row of a column and can be compared with the specified threshold.

    Location

    The name of the folder in which you want to store the custom rule template.

  7. Click OK.

Create a rule template on the Monitoring Rules page

  1. Go to the Data Quality page.

    Log on to the DataWorks console. In the left-side navigation pane, choose Data Modeling and Development > Data Quality. On the page that appears, select the desired workspace from the drop-down list and click Go to Data Quality.

  2. In the left-side navigation pane, choose Rule Management > Rules.

  3. On the Rules page, specify filter conditions such as Engine/Data Source and Engine/Database Instance to search for the desired rule. Find the desired rule in the search result and click the rule ID. The details page of the rule appears.

    Note

    A MaxCompute data source is used in this example.

  4. On the page that appears, select a partition filter expression and click the Custom Rules tab. On the Custom Rules tab, find a custom rule based on which you want to create a rule template and click Generate Template in the Actions column.

    自定义规则

  5. In the Generate Template dialog box, configure the parameters.

    新建规则模板

    Parameter

    Description

    Template Name

    The name of the custom rule template.

    Field

    The fields that you want to monitor. You can set this parameter only to Custom SQL.

    Sampling Method

    The sampling method for the custom rule template. You can set this parameter only to Custom SQL.

    Set Flag

    The SET clause of the SQL statement that is used to query the fields that you want to monitor.

    Note

    Separate multiple statements with commas (,). You do not need to add a semicolon (;) at the end of each statement.

    Check type

    The threshold type for the custom rule template. Valid values: Numeric type and Fluctuation.

    Verification Method

    The verification method for the custom rule template. The verification methods that can be selected vary based on the threshold type.

    • If you set the Check type parameter to Numeric type, you can set this parameter only to Compare with a specified value.

      Only the value that is calculated by the COUNT or SUM function can be returned. The return value is compared with a specified value.

    • If you set the Check type parameter to Fluctuation, the valid values of this parameter are Compare the current value with the average value of the last 7 days, Compare the current value with the average value of the last 30 days, Compare the current value with the value 1 day before, Compare the current value with the value 7 days before, Compare the current value with the value 30 days before, The variance between the current value and the value 7 days before, The variance between the current value and the value 30 days before, Compare with the value 1, 7, and 30 days before, and Compare with the value of the previous cycle.

    Custom SQL

    The SQL statement that is used to query the fields to be monitored. You can use ${tableName} to specify a table name.

    Note

    Make sure that the return value is the value in a row of a column and can be compared with the specified threshold.

    Location

    The name of the folder in which you want to store the custom rule template.

  6. Click OK.

  7. In the left-side navigation pane of the Data Quality page, choose Configuration > Rule Templates to view the created rule template.

Manage a rule template

On the Rule Templates page, you can click the name of a rule template to go to the template details page. On this page, you can view, modify, delete, or copy the rule template.操作规则模板

Action

Description

View

You can view the parameter settings and logs of the rule template and the rules that use the rule template.

  • The Application List tab displays the rules that use the rule template.

  • The View Log tab displays the operations logs of the rule template, including the user who performed each operation, the time when each operation was performed, and the operation details.

Edit

Click Edit in the upper-right corner. In the Edit Rule Template dialog box, modify the parameters as required and click OK.

Delete

Click Delete in the upper-right corner. In the Delete Template message, click OK.

Copy

Click Copy in the upper-right corner. In the Clone Rule Template dialog box, configure the Template Name and Location parameters and click OK.

Use a rule template

When you create a monitoring rule, you can select a custom rule template to create the rule based on the rule template.

  1. Go to the Data Quality page.

    Log on to the DataWorks console. In the left-side navigation pane, choose Data Modeling and Development > Data Quality. On the page that appears, select the desired workspace from the drop-down list and click Go to Data Quality.

  2. In the left-side navigation pane, choose Rule Management > Rules.

  3. On the Rules page, specify filter conditions such as Engine/Data Source and Engine/Database Instance to search for the desired rule. Find the desired rule in the search result and click the rule ID. The details page of the rule appears.

    Note

    A MaxCompute data source is used in this example.

  4. On the page that appears, select a partition filter expression and click Create Rule.

  5. On the Template Rules tab of the Create Rule panel, click Add Monitoring Rule.

  6. Configure the parameters for the rule. In this example, set the Rule Source parameter to Rule Templates and select a rule template. For more information about the parameter description, see Configure monitoring rules.

    创建规则

  7. Click Batch Create.