ListDataQualityRules - DataWorks - Alibaba Cloud ドキュメントセンター

Queries quality monitoring rules by paging.

Note

This API has been deprecated. We recommend migrating to dataworks-public(2024-05-18) - ListDataQualityScans for enhanced functionality and continued support. While the deprecated API remains temporarily accessible, no further updates or bug fixes will be provided.

Operation description

You must purchase DataWorks Basic Edition or a higher edition to use this feature.

Try it now

Try this API in OpenAPI Explorer, no manual signing needed. Successful calls auto-generate SDK code matching your parameters. Download it with built-in credential security for local usage.

Test

RAM authorization

The table below describes the authorization required to call this API. You can define it in a Resource Access Management (RAM) policy. The table's columns are detailed below:

Action: The actions can be used in the Action element of RAM permission policy statements to grant permissions to perform the operation.
API: The API that you can call to perform the action.
Access level: The predefined level of access granted for each API. Valid values: create, list, get, update, and delete.
Resource type: The type of the resource that supports authorization to perform the action. It indicates if the action supports resource-level permission. The specified resource must be compatible with the action. Otherwise, the policy will be ineffective.
- For APIs with resource-level permissions, required resource types are marked with an asterisk (*). Specify the corresponding Alibaba Cloud Resource Name (ARN) in the Resource element of the policy.
- For APIs without resource-level permissions, it is shown as All Resources. Use an asterisk (*) in the Resource element of the policy.
Condition key: The condition keys defined by the service. The key allows for granular control, applying to either actions alone or actions associated with specific resources. In addition to service-specific condition keys, Alibaba Cloud provides a set of common condition keys applicable across all RAM-supported services.
Dependent action: The dependent actions required to run the action. To complete the action, the RAM user or the RAM role must have the permissions to perform all dependent actions.

Action

Access level

Resource type

Condition key

Dependent action

dataworks:*

list

*All Resource

*

None

Request parameters

Parameter	Type	Required	Description	Example
ProjectId	integer	Yes	The DataWorks workspace ID.	10002
DataQualityEvaluationTaskId	integer	No	The ID of the associated quality check task.	10000
TableGuid	string	No	The unique identifier of the table that the rule applies to in DataWorks Data Map.	odps.unit_test.tb_unit_test
Name	string	No	The rule name for fuzzy matching.	unit_test
PageSize	integer	No	The number of entries per page. Default value: 10. Maximum value: 200.	10
PageNumber	integer	Yes	The page number for paging. Default value: 1.	1

Response elements

Element	Type	Description	Example
	object	The response parameters.
RequestId	string	The request ID.	691CA452-D37A-4ED0-9441
PagingInfo	object	The paginated query result of the rule list.
PageNumber	integer	The page number.	1
PageSize	integer	The page size.	10
TotalCount	integer	The total number of entries.	294
DataQualityRules	array<object>	The list of rules.
	array<object>
Id	integer	Rule ID.	22130
Name	string	Rule name.	The table cannot be empty.
ProjectId	integer	DataWorks workspace ID.	100001
Enabled	boolean	Whether the data quality rule is enabled.	true
Severity	string	Severity level of the rule for the business (corresponding to strong/weak rules on the page). Valid enumerated values: Normal High	High
Description	string	Rule description. Maximum length: 500 characters.	this is a odps _sql task
Target	object	Object monitored by the rule.
Type	string	Type of the monitored object. Table	Table
DatabaseType	string	For table-type datasets, the database type to which the table belongs. maxcompute emr cdh hologres analyticdb_for_postgresql analyticdb_for_mysql starrocks	maxcompute
TableGuid	string	Unique ID of the table to which the rule applies in Data Map.	odps.unit_test.tb_unit_test
TemplateCode	string	Unique identifier of the rule template referenced by the rule.	SYSTEM:user_defined_sql
SamplingConfig	object	Settings required for sample collection.
Metric	string	Sampling metric name. Count: number of table rows. Min: minimum value of the field. Max: maximum value of the field. Avg: average value of the field. DistinctCount: number of distinct values of the field. DistinctPercent: ratio of the number of distinct values of the field to the number of data rows. DuplicatedCount: number of duplicate values of the field. DuplicatedPercent: ratio of the number of duplicate values of the field to the number of data rows. TableSize: table size. NullValueCount: number of rows where the field value is null. NullValuePercent: percentage of rows where the field value is null. GroupCount: each value and its corresponding number of data rows after aggregation by field value. CountNotIn: number of rows whose enumerated values do not match. CountDistinctNotIn: number of distinct values whose enumerated values do not match. UserDefinedSql: sample collection via custom SQL.	Max
MetricParameters	string	Parameters required during sample collection.	{ "Columns": [ "id", "name" ] , "SQL": "select count(1) from table;"}
SettingConfig	string	Runtime parameter setting statements inserted and executed before the sampling statement is actually executed. Maximum length: 1,000 characters. Currently only MaxCompute is supported.	SET odps.sql.udf.timeout=600s; SET odps.sql.python.version=cp27;
SamplingFilter	string	Condition for the secondary filtering of data that is not of concern during sampling. Maximum length: 16,777,215 characters.	id IS NULL
CheckingConfig	object	Sample validation settings.
Type	string	Threshold calculation method. Fixed Fluctation FluctationDiscreate Auto Average Variance	Fixed
ReferencedSamplesFilter	string	Some types of thresholds require querying some reference samples, then aggregating the values of the reference samples to obtain the threshold for comparison. An expression is used here to represent the query method for the reference samples.	{ "bizdate": [ "-1", "-7", "-1m" ] }
Thresholds	object	Threshold settings.
Expected	object	Expected threshold settings.
Operator	string	Comparison operator. > >= < <= != =	>
Value	string	Threshold value.	100.0
Expression	string	Threshold expression. Fluctuation rate type rules must use the expression method to indicate the fluctuation threshold. Examples: Fluctuation increase greater than 0.01: $checkValue > 0.01 Fluctuation decrease greater than 0.01: $checkValue < -0.01 Absolute value of the fluctuation rate: abs($checkValue) > 0.01 Fixed value type rules can also configure thresholds using expressions. If both are configured, the expression takes precedence over Operator and Value.	$checkValue > 0.01
Warned	object	Threshold settings for normal warnings.
Operator	string	Comparison operator. > >= < <= != =	>
Value	string	Threshold value.	100.0
Expression	string	Threshold expression. Fluctuation rate type rules must use the expression method to indicate the fluctuation threshold. Examples: Fluctuation increase greater than 0.01: $checkValue > 0.01 Fluctuation decrease greater than 0.01: $checkValue < -0.01 Absolute value of the fluctuation rate: abs($checkValue) > 0.01 Fixed value type rules can also configure thresholds using expressions. If both are configured, the expression takes precedence over Operator and Value.	$checkValue > 0.01
Critical	object	Threshold settings for critical warnings.
Operator	string	Comparison operator. > >= < <= != =	>
Value	string	Threshold value.	100.0
Expression	string	Threshold expression. Fluctuation rate type rules must use the expression method to indicate the fluctuation threshold. Examples: Fluctuation increase greater than 0.01: $checkValue > 0.01 Fluctuation decrease greater than 0.01: $checkValue < -0.01 Absolute value of the fluctuation rate: abs($checkValue) > 0.01 Fixed value type rules can also configure thresholds using expressions. If both are configured, the expression takes precedence over Operator and Value.	$checkValue > 0.01
ErrorHandlers	array<object>	List of issue handlers for data quality rule validation.
	object	Issue handler for data quality rule validation.
Type	string	Handler type: SaveErrorData	SaveErrorData
ErrorDataFilter	string	For custom SQL rules, the user needs to specify SQL to filter problem data.	SELECT * FROM tb_api_log WHERE id IS NULL

Examples

Success response

JSON format

{
  "RequestId": "691CA452-D37A-4ED0-9441",
  "PagingInfo": {
    "PageNumber": 1,
    "PageSize": 10,
    "TotalCount": 294,
    "DataQualityRules": [
      {
        "Id": 22130,
        "Name": "The table cannot be empty.",
        "ProjectId": 100001,
        "Enabled": true,
        "Severity": "High",
        "Description": "this is a odps _sql task",
        "Target": {
          "Type": "Table",
          "DatabaseType": "maxcompute",
          "TableGuid": "odps.unit_test.tb_unit_test"
        },
        "TemplateCode": "SYSTEM:user_defined_sql",
        "SamplingConfig": {
          "Metric": "Max",
          "MetricParameters": "{ \"Columns\": [ \"id\", \"name\" ] , \"SQL\": \"select count(1) from table;\"}",
          "SettingConfig": "SET odps.sql.udf.timeout=600s; \nSET odps.sql.python.version=cp27;",
          "SamplingFilter": "id IS NULL"
        },
        "CheckingConfig": {
          "Type": "Fixed",
          "ReferencedSamplesFilter": "{ \"bizdate\": [ \"-1\", \"-7\", \"-1m\" ] }",
          "Thresholds": {
            "Expected": {
              "Operator": ">",
              "Value": "100.0",
              "Expression": "$checkValue > 0.01"
            },
            "Warned": {
              "Operator": ">",
              "Value": "100.0",
              "Expression": "$checkValue > 0.01"
            },
            "Critical": {
              "Operator": ">",
              "Value": "100.0",
              "Expression": "$checkValue > 0.01"
            }
          }
        },
        "ErrorHandlers": [
          {
            "Type": "SaveErrorData\n",
            "ErrorDataFilter": "SELECT * FROM tb_api_log WHERE id IS NULL"
          }
        ]
      }
    ]
  }
}

Error codes

See Error Codes for a complete list.

Release notes

See Release Notes for a complete list.