The Monitoring Rules page is the most important part of Data Quality. On this page, you can configure rules to monitor data in E-MapReduce, Hologres, AnalyticDB for PostgreSQL, MaxCompute, and DataHub. This topic describes how to configure monitoring rules for MaxCompute.
Create a MaxCompute connection
- Log on to the DataWorks console.
- In the left-side navigation pane, click Workspaces.
- On the Workspaces page, find the workspace in which you want to create a connection and click Data Integration in the Actions column.
- On the Data Integration page, click Connection in the left-side navigation pane. The page appears.
- Click New data source in the upper-right corner. In the Add data source dialog box, set the parameters as required to create a MaxCompute connection. For more information, see Configure a MaxCompute connection.
Select the MaxCompute connection
Configure a partition filter expression
- To configure rules for a non-partitioned table, you can specify NOTAPARTITIONTABLE as the partition filter expression.
- To configure rules for a partitioned table, you can specify a data timestamp expression, such as $[yyyymmdd], or a regular expression as the partition filter expression.

- Create a partition filter expression
In the Add Partition dialog box, enter a partition filter expression that conforms to the syntax as required. For a non-partitioned table, select NOTAPARTITIONTABLE from the recommended partition filter expressions.
- For a table with only one partition, follow the format: Partition key=Partition value. The partition value can be a constant or a system parameter. You must configure partition filter expressions by using the last partition.
- For a table with multiple partitions, follow the format: Partition key 1=Partition value/Partition key 2=Partition value/Partition key N=Partition value. Each partition value can be a constant or a system parameter. You must enclose a parameter in brackets [], such as $[yyyymmdd-N].
The data timestamp that is configured in a partition filter expression also determines the recurrence of the partition filter expression. For example, if the data timestamp is the date of five days ago, the partition filter expression is triggered every five days. The following table describes the supported partition filter expressions.Partition filter expression Description dt=$[yyyymmdd-N] Indicates N days before. dt=$[yyyymm01-1] Indicates the first day of each month. dt=$[yyyymm01-Nm] Indicates the first day of the month that is N months before the current month. dt=$[yyyymmld-1] Indicates the last day of each month. dt=$[yyyymmld-1m] Indicates the last day of the month that is N months before the current month. dt=$[hh24miss-1/24] Indicates one hour before the hour that is specified by the data timestamp. dt=$[hh24miss-30/24/60] Indicates half an hour before the hour that is specified by the data timestamp. $[yyyymmdd] Indicates the data timestamp. $[yyyymmdd-1] Indicates one day before the data timestamp of the current instance. $[yyyymmddhh24miss] Indicates the data timestamp of the current instance. Follow the yyyymmddhh24miss
format, where:- yyyy indicates a four-digit year.
- mm indicates a two-digit month.
- dd indicates a two-digit day.
- hh24 indicates a two-digit hour (24-hour clock).
- mi indicates two-digit minutes.
- ss indicates two-digit seconds.
NOTAPARTITIONTABLE Indicates the partition filter expression of a non-partitioned table. - Select a recommended partition filter expression
This section describes how to select a recommended partition filter expression. In this example, the dt partition is used. We recommend that you specify a regular expression as the partition filter expression for a dynamic partitioned table.
- In the Add Partition dialog box, click the Partition Expression field. A drop-down list appears to show
you the partition filter expressions that are recommended by Data Quality.
- Select a recommended partition filter expression that meets your expectation.
- Specify a custom partition filter expression if no recommended partition filter expressions meet your expectation.
- After you enter a partition filter expression, click Verify. Data Quality uses the current time as the data timestamp to calculate data and verify
the partition filter expression.
- Click OK.
- In the Add Partition dialog box, click the Partition Expression field. A drop-down list appears to show
you the partition filter expressions that are recommended by Data Quality.
If you need to delete a configured partition filter expression, move the pointer over the partition filter expression and click the Delete icon. After you delete a partition filter expression, all rules that are configured based on the partition filter expression are also deleted.
Link a partition filter expression to a node
- The Manage Linked Nodes dialog box lists all committed nodes. Data Quality allows you to link a partition filter expression to a node in another workspace.
- Before you link a partition filter expression to a node in another workspace, make sure that you are an administrator, a developer, or an administration expert in the two workspaces.
Create a rule
The Monitoring Rules page is the most important part of Data Quality, where you can create rules for your tables.
Data Quality allows you to create template rules and custom rules as needed. If you want to create a template rule or a custom rule, you can click Add Monitoring Rule or Quick Create. For more information, see Configure monitoring rules.
Creation method | Parameter | Description |
---|---|---|
Add Monitoring Rule | Rule Name | The name of the rule. |
Rule Type | The type of the rule. Valid values:
|
|
Auto-Generated Threshold | Specifies whether to use dynamic thresholds. You can use the dynamic threshold feature only in DataWorks Enterprise Edition or more advanced editions. | |
Rule Source | The source of the rule. Valid values: Built-in Template and Rule Templates. | |
Field | The fields to be monitored. You can select All Fields in Table or a specific field. If you select a field, you can apply the rule to the specified
field in the table.
Note In this example, select All Fields in Table and set other parameters for the table-specific
rule.
|
|
Template |
|
|
Comparison Method | The comparison method of the rule. Valid values: Absolute Value, Raise, and Drop. | |
Thresholds | The warning threshold and error threshold of the fluctuation. You can enter thresholds or adjust the slider to specify thresholds. | |
Description | The description of the rule. | |
Quick Create | Rule Name | The name of the rule. |
Field | The fields to be monitored. You can select All Fields in Table or a specific field. If you select a field, you can apply the rule to the specified field in the table. | |
Trigger |
|
Test rules
Manage subscriptions
By default, Data Quality sends notifications to the user who created a partition filter expression. You can add other users so that Data Quality sends notifications to them.
View operations logs
On the rule configuration page of a table, click View Operation Log. In the Operations Logs panel, you can view the information about each operation, including the user who performed the operation, the time when the operation was performed, and the operation details.
The Details column displays the details of each operation that is performed on the current partition filter expression, including the rule configuration details.
View check results
On the rule configuration page of a table, click View Check Results to go to the Node Query page. On this page, you can view the check results for all rules under the current partition filter expression.
Clone rules
- On the rule configuration page of a table, click Clone Rules.
- In the Clone Rules dialog box, set the Target Expression parameter.
- Select Clone Subscribers and Change Table Names in Custom Rules as required.
- Click Clone.