Data Management (DMS) lets you define quality rules for tables and columns, then automatically validates your data on a schedule using audit tasks in a task flow. This keeps data consistent, unique, and available — so downstream analysis and development can rely on it.
Prerequisites
Before you begin, make sure that:
The database instance is managed in Security Collaboration mode. For more information, see Control modes.
You meet one of the following permission requirements:
You are a DMS administrator, a database administrator (DBA), or a user with the schema read-only role. For more information, see System roles.
You are the instance owner or the DBA of the target database instance. For more information, see Modify database instances.
You have query, change, or export permissions on the target database instance or a table in it. For more information, see View owned permissions.
Key concepts
Quality rule: A condition you define for a table or column. DMS evaluates data against this condition and generates a report. For example, you can require that a column's maximum value never exceeds 100, or that the row count stays below 100,000.
Audit task: A scheduled task that checks your data against a specific quality rule and generates a report. Once you attach an audit task to a quality rule and publish a task flow with a scheduling cycle, DMS runs the check automatically.
Create a quality rule
Log in to the DMS console V5.0.
Open the details page of the target table. Use one of the following methods: From the SQL Console: From the search bar:
In the top navigation bar, click Home, then enter the table name in the search box.
In the search results, click Table, find your table, and click Details.
Click the Data quality tab.
Click Create Rule and set the following parameters: For example, the settings above mean: *check that the maximum value in the
columns_testcolumn is less than or equal to 100.*Parameter Description Example Rule name A name for the quality rule The maximum column value is 100 Rule type The scope of the rule. Table Rule checks table-level metrics (such as row count). Column Rule checks column-level metrics (such as null values or value ranges). Column Rule Rule template The preset check to apply. See Rule templates for all available options. Maximum Value in a Column Column name The column to check. Appears only when Rule type is set to Column Rule. columns_test Comparison method How to compare the actual metric against the expected value. Options: Greater, Greater than or equal, Equal, Is less than or equal, Less, Not equal. Is less than or equal Expected value The threshold value for the rule 100 Click OK.
The rule is created in the Enabled state.
To manage an existing rule, hover over More in the Actions column and choose from the following options:
Edit — modify the rule.
Logs — view the version history of the rule.
Disable — disable the rule. A rule cannot be disabled if it is used by an audit task. Delete the audit task first, then disable the rule.
Schedule automated checks
After creating a quality rule, set up a task flow to run automated checks on a schedule.
In the Actions column of the quality rule, hover over More and select New Task Flow.
On the Task Orchestration page, add an Audit Task node and configure it to reference your quality rule.
Set a scheduling cycle and publish the task flow.
DMS starts checking your data automatically based on the scheduling cycle and generates a report after each run. For details on configuring the task flow, see Overview.
Rule templates
Use this table to pick the right template for your data quality goal.
| Rule type | Rule template | What it checks | Example use |
|---|---|---|---|
| Table Rule | Number of Rows in a Table. The Number is a Fixed Value | Whether the total row count meets your requirement | Row count must be less than 100,000 |
| Column Rule | Maximum Value in a Column | Whether the column's maximum value is within bounds | Primary key cannot exceed 2,147,483,647 (2³¹ − 1) |
| Column Rule | Minimum Value in a Column | Whether the column's minimum value is within bounds | Minimum value cannot be less than 3 |
| Column Rule | Number of Strings That Match the Expression Containing Wildcards. The Number is a Fixed Value | Whether the count of values matching a wildcard pattern meets your requirement | Values starting with a must be fewer than 10 |
| Column Rule | Number of Strings that Match the Regular Expression. The Number is a Fixed Value | Whether the count of values matching a regular expression meets your requirement | Regex-matching values must be greater than 100 |
| Column Rule | Number of Null Values in a Column. The Number is a Fixed Value | Whether the null value count meets your requirement | Null values must be less than or equal to 0 |
| Column Rule | Number of Unique Values in a Column. The Number is a Fixed Value | Whether the count of distinct values meets your requirement | At least 100 distinct values must exist |
| Column Rule | Number of Duplicate Values in a Column. The Number is a Fixed Value | Whether the count of duplicate values meets your requirement | Duplicate values must be fewer than 100 |
How to choose a template:
To prevent bad data from entering analysis pipelines, start with null value and duplicate value checks.
To enforce value ranges (for example, ID boundaries or score limits), use Maximum Value or Minimum Value checks.
To validate data formats (for example, phone numbers or codes), use wildcard or regular expression checks.
To monitor table size or detect unexpected data loss, use the row count check.