This topic describes the structure and fields of an event message in DataWorks and provides examples of bodies of event messages for different types of events to help you understand event messages and the status changes of event messages in DataWorks.
Message structure
The following sample code provides an example of a complete event message:
{
"datacontenttype": "application/json;charset=utf-8",
"data": {
"beginWaitTimeTime": 165270057****,
"dagId": 44655****,
"dagType": 0,
"taskType": 0,
"modifyTime": 165270057****,
"createTime": 165254323****,
"appId": 3*****2,
"tenantId": 235454***432001,
"opCode": 31,
"flowId": 1,
"nodeId": 100269****,
"beginWaitResTime": 165270057****,
"taskId": 453268****,
"status": 3
},
"id": "539fd8f4-4ea1-4625-aa8b-6c906674****",
"source": "acs.dataworks",
"specversion": "1.0",
"subject": "",
"time": "2020-11-19T21:04:41+08:00",
"type": "dataworks:InstanceStatusChanges:InstanceStatusChanges",
"aliyunaccountid": "123456789098****",
"aliyunpublishtime": "2020-11-19T21:04:42.179PRC",
"aliyuneventbusname": "default",
"aliyunregionid": "cn-hangzhou",
"aliyunpublishaddr": "172.25.XX.XX"
}
A complete event message contains information such as the body, ID, source, and generation time of the event message. The following table describes the key fields in the preceding sample code. Note For more information about the fields in an event message, see Overview.
Event types
The type of an event is indicated by the value of the
type
field in the message for the event. The message body varies based on the type of the event. The following table describes the event types. Event type | Description | Message body (indicated by the data field) |
---|---|---|
dataworks:InstanceStatusChanges:InstanceStatusChanges | Change the status of an auto triggered node. | Message body for a status change event for an auto triggered node |
dataworks:InstanceChange:UnfreezeInstance | Unfreeze an instance. | Message body for an instance operation event (freeze, unfreeze, terminate, rerun, and set the instance status to Succeeded) |
dataworks:InstanceChange:FreezeInstance | Freeze an instance. | |
dataworks:InstanceChange:KillInstance | Terminate an instance. | |
dataworks:InstanceChange:RerunInstance | Rerun an instance. | |
dataworks:InstanceChange:SetInstanceSuccess | Set the status of an instance to Succeeded. | |
dataworks:FileChange:CommitFile | Commit a file. | Message body for a file change event (commit, deploy, execute, and delete) |
dataworks:FileChange:DeployFile | Deploy a file. | |
dataworks:FileChange:RunFile | Run a file. | |
dataworks:FileChange:DeleteFile | Delete a file. | |
dataworks:TableChange:CommitTable | Commit a table to the development environment. | Message body for a table change event (commit a table to the development environment and deploy a table to the production environment) |
dataworks:TableChange:DeployTable | Deploy a table to the production environment. | |
dataworks:NodeChange:NodeChangeCreated | Create a node. | Message body for a node change event (create, modify, delete, freeze, unfreeze, and undeploy) |
dataworks:NodeChange:NodeChangeUpdated | Modify a node. | |
dataworks:NodeChange:NodeChangeDeleted | Delete a node. | |
dataworks:NodeChange:FreezeNode | Freeze a node. | |
dataworks:NodeChange:UnFreezeNode | Unfreeze a node. | |
dataworks:NodeChange:UndeployNode | Undeploy a node. | |
dataworks:BackfillDataOperate:BackfillData | Backfill data for a node. | Message body for a data backfill event |
dataworks:ApprovalChange:ApprovalChangeCreated | Create a request order. | Message body for an Approval Center event (request task creation and completion) |
dataworks:ApprovalChange:ApprovalChangeFinished | Complete a request order. | |
dataworks:MonitorAlert:WorkbenchMonitorAlert | Report an alert in Operation Center. | Message body for an alert event in Operation Center (baseline alert, event alert, and rule-based alert for an object such as node, baseline, workspace, workflow, or resource group) |
dataworks:DagStatusChanges:DagStatusChanges | Change the status of a workflow. | Message body for a status change event for a workflow |
dataworks:DqcCheck:DqcCheckFeedbackEvent | Send feedback on data quality check results. | Message body for a check result feedback event |
dataworks:DqcCheck:DqcCheckFinishedEvent | Report that a data quality check is complete. | Message body for a check completion event |
Message body for a status change event for an auto triggered node
Sample code of the message body for a status change event for an auto triggered node:{
"datacontenttype": "application/json;charset=utf-8",
"data": {
"beginWaitTimeTime": 1652700576000,
"dagId": 446555330,
"dagType": 0,
"taskType": 0,
"modifyTime": 1652700577000,
"createTime": 1652543233000,
"appId": 3*****2,
"tenantId": 235454***432001,
"opCode": 31,
"flowId": 1,
"nodeId": 1002698219,
"beginWaitResTime": 1652700577000,
"taskId": 4532688169,
"status": 3
}
The following table describes the key fields in the preceding sample code. Note The value of the
data
field in the preceding sample code is only for your reference. The following table describes the common fields related to the data
field for a status change event of an auto triggered node. Field | Type | Description |
---|---|---|
finishTime | Long | The time when the auto triggered node instance finished running. |
beginWaitTimeTime | Long | The time when the auto triggered node instance started to wait for the scheduling time. |
beginRunningTime | Long | The time when the auto triggered node instance started to run. |
dagId | Long | The ID of the directed acyclic graph (DAG). You can query the details of the DAG based on the dagId field. |
dagType | Integer | The type of the DAG. Valid values:
|
taskType | Integer | The type of the node. Valid values:
|
modifyTime | Long | The last time when the node instance was modified. |
createTime | Long | The time when the node instance was created. |
appId | Long | The ID of the workspace. You can call the ListProjects operation to query the ID of a workspace. |
tenantId | Long | The ID of the tenant of the workspace to which the auto triggered node instance belongs. |
opCode | Integer | The operation code of the auto triggered node instance. You can ignore the field value. |
flowId | Long | The ID of the workflow.
|
nodeId | Long | The ID of the node for which the auto triggered node instance was generated. |
beginWaitResTime | Long | The time when the auto triggered node instance started to wait for resources. |
taskId | Long | The ID of the auto triggered node instance. |
status | Integer | The status of the node. Valid values:
|
Message body for an instance operation event (freeze, unfreeze, terminate, rerun, and set the instance status to Succeeded)
Sample code of the message body for an instance operation event:
{
"datacontenttype": "application/json;charset=utf-8",
"data": {
"operator": "182293110403****",
"projectId": 123488,
"projectType": "PROD",
"taskIds":
[
523536569736
],
"tenantId": 280749521950784
}
The following table describes the fields in the preceding sample code. Field | Type | Description |
---|---|---|
operator | String | The UID of the user who performs the operations such as freezing, unfreezing, terminating, or rerunning an instance, or setting the status of an instance to Succeeded. |
projectType | String | The runtime environment.
|
taskIds | List<Long> | The IDs of the instances. |
projectId | Long | The ID of the workspace. |
tenantId | Long | The ID of the tenant of the workspace to which the instance belongs. |
Message body for a file change event (commit, deploy, execute, and delete)
- Sample code of the message body for a file commit or deployment event:
The following table describes the fields in the preceding sample code.{ "datacontenttype": "application/json;charset=utf-8", "data": { "operator": "**************", // The operator. "projectId": 12*****56, "tenantId": 12*****56, "nodeId": 123456, // The node ID. "fileType": 123456, // The type of the file. "fileName": "********", "fileOwner": "****************", // The owner of the file. "extensionBizId": "***************", // "changeType": "UPDATE", //NEW UPDATE DELETE "fileCreateTime": "2021-01-15 14:03:02", "fileId": 123456, "fileVersion": 3, "dataSourceName":"odps_first" }
Field Type Description operator String The UID of the user who commits or deploys the file. projectId Long The ID of the workspace to which the file belongs. tenantId Long The ID of the tenant. nodeId Long The ID of the auto triggered node. fileType Long The code type of the file. Valid values: - 6 (Shell)
- 10 (ODPS SQL)
- 11 (ODPS MR)
- 23 (Data Integration)
- 24 (ODPS Script)
- 99 (zero load)
- 221 (PyODPS 2)
- 225 (ODPS Spark)
- 227 (EMR Hive)
- 228 (EMR Spark)
- 229 (EMR Spark SQL)
- 230 (EMR MR)
- 239 (OSS object inspection)
- 257 (EMR Shell)
- 258 (EMR Spark Shell)
- 259 (EMR Presto)
- 260 (EMR Impala)
- 900 (real-time synchronization)
- 1089 (cross-tenant collaboration)
- 1091 (Hologres development)
- 1093 (Hologres SQL)
- 1100 (assignment)
- 1221 (PyODPS 3)
fileName String The name of the file. fileOwner String The owner of the file. extensionBizId String The ID of the check process in an extension. changeType String The type of the change to the file. Valid values: - UPDATE: updates the file.
- DELETE: deletes the file.
- CREATE: creates the file.
fileCreateTime String The time when the file was created. The time is in the yyyy-MM-dd HH:mm:ss
format.fileId Long The ID of the file. fileVersion Long The version of the file. dataSourceName String The name of the data source. - Sample code of the message body for a file deletion or execution event:
The following table describes the fields in the preceding sample code.{ "datacontenttype": "application/json;charset=utf-8", "data": { "operator": "**************", // The operator. "projectId": 123456, "tenantId": 123456, "nodeId": 123456, // The node ID. "fileType": 123456, // The type of the file. "fileName": "********", "fileOwner": "****************", // The owner of the file. "extensionBizId": "***************", // "fileCreateTime": "2021-01-15 14:03:02", "fileId": 123456 }
Field Type Description operator String The UID of the user who deletes or runs the file. projectId Long The ID of the workspace to which the file belongs. tenantId Long The ID of the tenant. nodeId Long The ID of the auto triggered node. fileType Long The code type of the file. Valid values: - 6 (Shell)
- 10 (ODPS SQL)
- 11 (ODPS MR)
- 23 (Data Integration)
- 24 (ODPS Script)
- 99 (zero load)
- 221 (PyODPS 2)
- 225 (ODPS Spark)
- 227 (EMR Hive)
- 228 (EMR Spark)
- 229 (EMR Spark SQL)
- 230 (EMR MR)
- 239 (OSS object inspection)
- 257 (EMR Shell)
- 258 (EMR Spark Shell)
- 259 (EMR Presto)
- 260 (EMR Impala)
- 900 (real-time synchronization)
- 1089 (cross-tenant collaboration)
- 1091 (Hologres development)
- 1093 (Hologres SQL)
- 1100 (assignment)
- 1221 (PyODPS 3)
fileName String The name of the file. fileOwner String The owner of the file. extensionBizId String The ID of the check process in an extension. fileCreateTime String The time when the file was created. The time is in the yyyy-MM-dd HH:mm:ss
format.fileId Long The ID of the file.
Message body for a table change event (commit a table to the development environment and deploy a table to the production environment)
Sample code of the message body for a table change event:{
"datacontenttype": "application/json;charset=utf-8",
"data": {
"operator": "**************",
"projectId": 12*****56,
"tenantId": 12******56,
"extensionBizId": "123456",
"tableName":"table1",
"tableType":"ODPS", // The type of the table. Valid values: ODPS and EMR.
"maxComputeProject":"project1"
}
The following table describes the fields in the preceding sample code. Field | Type | Description |
---|---|---|
operator | String | The UID of the user who commits or deploys the table. |
projectId | Long | The ID of the workspace. |
tenantId | Long | The ID of the tenant. |
extensionBizId | String | The ID of the check process in an extension. |
tableName | String | The name of the table. |
tableType | String | The type of the table. Valid values:
|
maxComputeProject | String | The name of the MaxCompute project. |
Message body for a node change event (create, modify, delete, freeze, unfreeze, and undeploy)
- Sample code of the message body for a node creation, modification, or deletion event:
The following table describes the fields in the preceding sample code.{ "datacontenttype": "application/json;charset=utf-8", "data": { "nodeName": "", "programType": "ODPS_SQL", "cronExpress": "00 20 00 * * ?", "schedulerType": "NORMAL", "ownerId": "1912232488744735", "priority": 1, "baselineId": 117801853, "repeatability": true, "modifyTime": 1646364549642, "createTime": 1646364549642, "datasource": "odps_first", "tenantId": 28378****10656, "nodeId": 100***150, "projectId": 30**95, "operator": "**************" // The operator. }
Field Type Description nodeName String The name of the node. programType String The code type of the file. Valid values: - 6 (Shell)
- 10 (ODPS SQL)
- 11 (ODPS MR)
- 24 (ODPS Script)
- 99 (zero load)
- 221 (PyODPS 2)
- 225 (ODPS Spark)
- 227 (EMR Hive)
- 228 (EMR Spark)
- 229 (EMR Spark SQL)
- 230 (EMR MR)
- 239 (OSS object inspection)
- 257 (EMR Shell)
- 258 (EMR Spark Shell)
- 259 (EMR Presto)
- 260 (EMR Impala)
- 900 (real-time synchronization)
- 1089 (cross-tenant collaboration)
- 1091 (Hologres development)
- 1093 (Hologres SQL)
- 1100 (assignment)
- 1221 (PyODPS 3)
cronExpress String The CRON expression that represents the periodic scheduling policy of the node. This parameter is equivalent to the Cron Expression parameter in the Schedule section of the Properties tab on the DataStudio page in the DataWorks console.
After you configure the Scheduling Cycle and Run At parameters in the DataWorks console, DataWorks automatically generates the value of the Cron Expression parameter. Examples:- CRON expression for a node that is scheduled to run at 05:30 every day:
00 30 05 * * ?
- CRON expression for a node that is scheduled to run at the fifteenth minute of each hour:
00 15 * * * ?
- CRON expression for a node that is scheduled to run every 10 minutes:
00 00/10 * * * ?
- CRON expression for a node that is scheduled to run every 10 minutes from 08:00 to 17:00 every day:
00 00-59/10 8-17 * * * ?
- CRON expression for a node that is scheduled to run at 00:20 on the first day of each month:
00 20 00 1 * ?
- CRON expression for a node that is scheduled to run every three months from 00:10 on January 1:
00 10 00 1 1-12/3 ?
- CRON expression for a node that is scheduled to run at 00:05 every Tuesday and Friday:
00 05 00 * * 2,5
Note Limits on CRON expressions:- A node can be scheduled to run at a minimum interval of 5 minutes.
- A node can be scheduled to run at 00:05 every day at the earliest.
schedulerType String The type of the node. Valid values: - 0: NORMAL. The node is a normal auto triggered node. The scheduling system regularly runs the node.
- 1: MANUAL. The node is a manually triggered node. The scheduling system does not regularly run the node.
- 2: PAUSE. The node is a frozen node. The scheduling system regularly runs the node but sets the node status to Failed when the scheduling system starts to run the node.
- 3: SKIP. The node is a dry-run node. The scheduling system regularly runs the node but sets the node status to Succeeded when the scheduling system starts to run the node.
- 4: SKIP_UNCHOOSE. The node is an unselected node in a temporary workflow. This type of node exists only in temporary workflows. The scheduling system sets the node status to Succeeded when the scheduling system starts to run the node.
- 5: SKIP_CYCLE. The node is a node that is scheduled by the week or month and is waiting for the scheduling time to arrive. The scheduling system regularly runs the node but sets the node status to Succeeded when the scheduling system starts to run the node.
- 6: CONDITION_UNCHOOSE. The node is not selected by its ancestor branch node and is run as a dry-run node.
- 7: REALTIME_DEPRECATED. The node has instances that are generated in real time but are deprecated. The scheduling system sets the node status to Succeeded.
ownerId String The ID of the Alibaba Cloud account used by the owner of the node. If this field is not configured, the ID of the Alibaba Cloud account of the user who schedules the node is used. priority Integer The priority of the node. Valid values: 1, 3, 5, 7, and 8. A greater value specifies a higher priority. baselineId Long The ID of the baseline. repeatability Boolean Specifies whether the node can be rerun. Valid values: - true: The node can be rerun.
- false: The node cannot be rerun.
modifyTime Long The last time when the node was modified. createTime Long The time when the node was created. nodeId Long The ID of the node. projectId Long The ID of the workspace to which the node belongs. tenantId Long The ID of the tenant to which the node belongs. operator String The UID of the user who adds, modifies, or deletes the node. - Sample code of the message body for a node freezing, unfreezing, or undeployment event:
The following table describes the fields in the preceding sample code.{ "datacontenttype": "application/json;charset=utf-8", "data": { "operator": "**************", "projectId": 123456, "tenantId": 123456, "nodeIds":[1,2,3], "extensionBizId": "123456" }
Field Type Description operator String The UID of the user who freezes, unfreezes, or undeploys the node. projectId Long The ID of the workspace to which the node belongs. tenantId Long The ID of the tenant. nodeIds Array The IDs of the nodes on which you performed the operation. extensionBizId String The ID of the check process in an extension.
Message body for a data backfill event
Sample code of the message body for a data backfill event:{
"datacontenttype": "application/json;charset=utf-8",
"data": {
"excludeNodeIds":[],
"rootNodeId": 10000838271,
"startFutureInstanceImmediately": false,
"useMultipleTimePeriods": true,
"operator": "1320876710332592",
"multipleTimePeriods": "[{\"bizBeginTime\":\"2022-04-17\",\"bizEndTime\":\"2022-04-17\"}]",
"parallelGroup": 1,
"rootNodeProjectId": 12*****8,
"isParallel": false,
"name": "P_fff_20220418_215404",
"tenantId": 16935*****3377,
"includeNodeIds":
[
10000838271
],
"projectId": 12958,
"order": "asc",
"extensionBizId": "123456"
}
The following table describes the key fields in the preceding sample code. Note The value of the
data
field in the preceding sample code is only for your reference. The following table describes the common fields related to the data
field for a data backfill event. Field | Type | Description |
---|---|---|
name | String | The name of the workflow that contains the node for which you backfill data. |
rootNodeId | Long | The ID of the root node in the workflow that contains the node for which you backfill data. |
rootNodeProjectId | Long | The ID of the workspace to which the root node belongs. The root node is in the workflow that contains the node for which you backfill data. |
includeNodeIds | Array | The IDs of the nodes for which you backfill data. |
excludeNodeIds | Array | The IDs of the nodes for which no data needs to be backfilled. The system generates dry-run instances for all these nodes. After the dry-run instances are scheduled, the statuses of the instances are directly set to Succeeded but the script is not run. |
bizBeginTime | String | The time when the node started to run. This field is required only for auto triggered nodes that are scheduled by hour. Specify the value in the HH:mm:ss format. Valid values: 00:00:00 to 23:59:59. |
bizEndTime | String | The time when the node finished running. This field is required only for auto triggered nodes that are scheduled by hour. Specify the value in the HH:mm:ss format. Valid values: 00:00:00 to 23:59:59. |
isParallel | Boolean | Specifies whether data backfill instances that are scheduled to run at different points in time can be run in parallel.
|
parallelGroup | Integer | The number of data backfill instance groups that are run in parallel. The value 1 indicates that data backfill instances are not grouped. |
startFutureInstanceImmediately | Boolean | Specifies whether to immediately run node instances that are scheduled to run at future points in time.
|
order | String | The node execution sequence based on data timestamps.
|
multipleTimePeriods | String | Multiple data timestamp time periods. The field value is determined by the bizBeginTime and bizEndTime fields. Example: [{\"bizBeginTime\":\"2022-04-17\",\"bizEndTime\":\"2022-04-17\"}] . |
tenantId | Long | The ID of the tenant. |
projectId | Long | The ID of the workspace in which you perform the data backfill operation. |
operator | String | The user who performs the data backfill operation. |
extensionBizId | String | The ID of the check process in an extension. |
Message body for an Approval Center event (request task creation and completion)
- Sample code of the message body for a request task creation event:
The following table describes the key fields in the preceding sample code.{ "datacontenttype": "application/json;charset=utf-8", "data": { "appId":194209, "assignee":"286098539641742899", "assigneeName":"yupeng.sunyp", "createTime":1652094363000, "eventType":"approval", "process":{ "applicant":"286098539641742899", "applicantName":"yupeng.sunyp", "approvalContent":{ "applyPeriod":"2997964800000", "applyReason":"test", "arrayData":[ { "ownerBaseId":"1822931104031845", "columns":[ { "columnType":"string", "columnGuid":"odps.b_mc1.loghub_070103.id", "columnComment":"", "columnName":"id" }, { "columnType":"string", "columnGuid":"odps.b_mc1.loghub_070103.name", "columnComment":"", "columnName":"name" }, { "columnType":"string", "columnGuid":"odps.b_mc1.loghub_070103.times", "columnComment":"", "columnName":"times" } ], "columnMetaList":[ { "columnType":"string", "columnGuid":"odps.b_mc1.loghub_070103.id", "columnComment":"", "columnName":"id" }, { "columnType":"string", "columnGuid":"odps.b_mc1.loghub_070103.name", "columnComment":"", "columnName":"name" }, { "columnType":"string", "columnGuid":"odps.b_mc1.loghub_070103.times", "columnComment":"", "columnName":"times" } ], "objectType":"TABLE", "odpsTable":"loghub_070103", "envType":1, "projectGuid":"odps.b_mc1", "objectGuid":"odps.b_mc1.loghub_070103", "tenantId":0, "objectName":"loghub_070103", "ownerAccountName":"ALIYUN$dataworks_3h1_1(dataworks_3h1_1)", "odpsProject":"B_MC1", "projectName":"B_MC1", "actions":[ "Select", "Describe" ], "projectId":194209, "workspaceId":"194209" } ], "contentType":"application/json", "granteeAccounts":[ { "granteeId":"286098539641742899", "granteeTypeSub":103, "granteeType":1, "granteeName":"RAM$dataworks_3h1_1:yupeng.sunyp" }, { "granteeId":"237857631119109360", "granteeTypeSub":105, "granteeType":1, "granteeName":"RAM$dataworks_3h1_1:dev" } ], "odpsProjectName":"B_MC1", "projectEnv":"1", "resourceSummary":"loghub_070103", "tenantId":280749521950784, "workspaceId":194209 }, "assignmentCategory":"MaxCompute", "createTime":1652094363000, "processDefinitionId":"definition-3dcc9ce7-d29d-435d-a908-60d4355ff5e2", "processId":"528535869984706", "status":"Pending", "title":"MaxComputeTable", "updateTime":1652094363000 }, "processId":"528535869984706", "status":"Submit", "taskId":"528535870015424", "tenantId":280749521950784, "updateTime":1652094364000 }
Field Type Description appId Long The ID of the workspace. assignee String The base ID of the Alibaba Cloud account used by the request approver. assigneeName String The name of the Alibaba Cloud account used by the request approver. comments String The comment. createTime Long The timestamp at which the request order is created. processId String The ID of the request processing order. status String The status of the request processing order. taskId String The ID of the request task. tenantId String The ID of the tenant. updateTime String The timestamp at which the request order is updated. eventType String The type of the event. process Object The process for the request task. applicant String The base ID of the applicant. applicantName String The name of the applicant. assignmentCategory String The type of the content on which you request permissions. createTime String The timestamp at which the request order is created. processDefinitionId String The unique ID of the defined approval workflow. processId String The ID of the workflow. status String The status of the workflow. title String The title of the workflow. updateTime Long The timestamp at which the request order is updated. approvalContent Object The content of the request order. applyPeriod String The validity period of the requested permissions. applyReason String The reason for the request. contentType String The data type of the content. odpsProjectName String The name of the MaxCompute project. resourceSummary String The description of the resource. tenantId Long The ID of the tenant. workspaceId Long The ID of the workspace. projectEnv String The environment of the MaxCompute project. granteeAccounts Array The accounts of the requesters. .granteeId String The ID of the requester. granteeType String The authorization type. granteeTypeSub String The account type of the requester. granteeName String The name of the requester. arrayData Array The authorized content. For more information, see Appendix 1: Authorized content (MaxCompute). - Sample code of the message body for a request task completion event:
{ "datacontenttype": "application/json;charset=utf-8", "data": { "appId":227859, "assignee":"286098539641742899", "assigneeName":"yupeng.sunyp", "comments":"ces", "createTime":1652095981000, "eventType":"approval", "process":{ "applicant":"286098539641742899", "applicantName":"yupeng.sunyp", "approvalContent":{ "applyPeriod":"2997964800000", "applyReason":"test", "arrayData":[ { "ownerBaseId":"238266539737108884", "columns":[ { "columnType":"string", "columnGuid":"odps.da_simple_202112.cdd.id", "columnComment":"", "columnName":"id" } ], "columnMetaList":[ { "columnType":"string", "columnGuid":"odps.da_simple_202112.cdd.id", "columnComment":"", "columnName":"id" } ], "objectType":"TABLE", "odpsTable":"cdd", "objectNameCn":"a big tree with deep roots", "envType":1, "projectGuid":"odps.da_simple_202112", "objectGuid":"odps.da_simple_202112.cdd", "tenantId":0, "objectName":"cdd", "ownerAccountName":"RAM$dataworks_3h1_1:wb-wjh926877(wb-wjh926877)", "odpsProject":"da_simple_202112", "projectName":"da_simple_202112", "actions":[ "Select", "Describe" ], "projectId":227859, "workspaceId":"227859" } ], "contentType":"application/json", "granteeAccounts":[ { "granteeId":"286098539641742899", "granteeTypeSub":103, "granteeType":1, "granteeName":"RAM$dataworks_3h1_1:yupeng.sunyp" } ], "odpsProjectName":"da_simple_202112", "projectEnv":"1", "resourceSummary":"cdd", "tenantId":280749521950784, "workspaceId":227859 }, "assignmentCategory":"MaxCompute", "createTime":1652095981000, "processDefinitionId":"definition-6e6418e6-c65f-4f26-a673-88576b1c1e4a", "processId":"528539182432192", "status":"Pending", "title":"MaxComputeTable", "updateTime":1652095981000 }, "processId":"528539182432192", "status":"Completed", "taskId":"528539182780353", "tenantId":280749521950784, "updateTime":1652096017000 }
Field | Type | Description |
---|---|---|
appId | Long | The ID of the workspace. |
assignee | String | The base ID of the Alibaba Cloud account used by the request approver. |
assigneeName | String | The name of the Alibaba Cloud account used by the request approver. |
comments | String | The comment. |
createTime | Long | The timestamp at which the request order is created. |
processId | String | The ID of the request processing order. |
status | String | The status of the request processing order. |
taskId | String | The ID of the request task. |
tenantId | String | The ID of the tenant. |
updateTime | String | The timestamp at which the request order is last updated. |
eventType | String | The type of the event. |
process | Object | The process for the request task. |
applicant | String | The base ID of the applicant. |
applicantName | String | The name of the applicant. |
assignmentCategory | String | The type of the content on which you request permissions. |
createTime | String | The timestamp at which the request order is created. |
processDefinitionId | String | The unique ID of the defined approval workflow. |
processId | String | The ID of the workflow. |
status | String | The status of the workflow. |
title | String | The title of the workflow. |
updateTime | Long | The timestamp at which the request order is updated. |
approvalContent | Object | The content of the request order. |
applyPeriod | String | The validity period of the requested permissions. |
applyReason | String | The reason for the request. |
contentType | String | The data type of the content. |
odpsProjectName | String | The name of the MaxCompute project. |
resourceSummary | String | The description of the resource. |
tenantId | Long | The ID of the tenant. |
workspaceId | Long | The ID of the workspace. |
projectEnv | String | The environment of the MaxCompute project. |
granteeAccounts | Array | The accounts of the requesters. |
granteeId | String | The ID of the requester. |
granteeType | String | The authorization type. |
granteeTypeSub | String | The account type of the requester. Valid values:
|
granteeName | String | The name of the requester. |
arrayData | Array | The authorized content. For more information, see Appendix 2: Authorized content (DataService). |
Message body for an alert event in Operation Center (baseline alert, event alert, and rule-based alert for an object such as node, baseline, workspace, workflow, or resource group)
- Sample code of the message body for a baseline alert:
The following table describes the fields in the preceding sample code.{ "datacontenttype": "application/json;charset=utf-8", "data": { "alarmType": "SLA_ALERT", "baselineId": 137000723, "baselineName": "Dedicated to the check for hour-level nodes at the ODS layer", "baselineStatus": 3, "bizDate": 1654444800000, "inGroupId": 14, "nodeId": 10000398734, "projectId": 7634, "taskId": 3072893778, "tenantId": 260953503125729 }
Field Type Description alarmType String The type of the alert. Valid values: - SLA_ALERT
- REMIND_ALERT
- TOPIC_ALERT
baselineId Long The ID of the baseline. baselineName String The name of the baseline. baselineStatus Integer The status of the baseline. Valid values: - -1: indicates that the baseline is abnormal.
- 1: indicates that the baseline is safe.
- 2: indicates that an alert is triggered for the baseline.
- 3: indicates that the baseline is in the Overtime state.
bizDate Long The data timestamp of the baseline instance. inGroupId Integer The ID of the scheduling cycle of the baseline instance. For a baseline instance that is scheduled by day, the field value is 1
. For a baseline instance that is scheduled by hour, the field value ranges from1 to 24
.nodeId Long The ID of the node that triggers an exception on the baseline to which the node belongs. projectId Long The ID of the workspace to which the baseline belongs. taskId Long The ID of the instance that triggers an exception on the baseline to which the instance belongs. tenantId Long The ID of the tenant. - Sample code of the message body for an event alert:
The following table describes the fields in the preceding sample code.{ "datacontenttype": "application/json;charset=utf-8", "data": { "alarmType": "TOPIC_ALERT", "nodeId": 10000316315, "projectId": 9109, "taskId": 3072750357, "taskStatus": 5, "tenantId": 280749521950784, "topicId": 1084769 }
Field Type Description alarmType String The type of the alert. Valid values: - SLA_ALERT
- REMIND_ALERT
- TOPIC_ALERT
topicId Long The ID of the event. taskStatus String The status of the instance that is generated for the node in which the event occurred. nodeId Integer The ID of the node in which the event occurred. projectId Long The ID of the workspace to which the node in which the event occurred belongs. taskId Long The ID of the instance that is generated for the node in which the event occurred. tenantId Long The ID of the tenant. - Sample code of the message body for a rule-based alert:Note In most cases, the objects for which you can configure a rule include node, baseline, workspace, and workflow.
The following table describes the fields in the preceding sample code.{ "datacontenttype": "application/json;charset=utf-8", "data": { "alarmType": "REMIND_ALERT", "nodeIds": "10000405472,10000405473,10000405474", "projectId": 9876, "remindId": 7605, "remindName": "Error alert", "remindType": "ERROR", "remindUnit": "NODE", "taskIds": "3072900896,3072900870,3072900855", "tenantId": 259025816027648 }
Field Type Description alarmType String The type of the alert. Valid values: - SLA_ALERT
- REMIND_ALERT
- TOPIC_ALERT
nodeIds String The IDs of the nodes for which the rule-based alert is triggered. remindId Long The ID of the rule. remindType Stirng The trigger condition of the rule. Valid values: - FINISHED: The object finishes running.
- UNFINISHED: The object does not finish running.
- ERROR: An error occurs when the object runs.
- CYCLE_UNFINISHED: The object does not finish running in the current scheduling cycle.
- TIMEOUT: The execution of the object times out.
projectId Long The ID of the workspace. The workspace contains the node for which the rule-based alert is triggered. remindUnit String The type of the object for which the rule-based alert is triggered. Valid values: - NODE: a node
- GATEWAY_RES: an exclusive resource group for scheduling
- DI_RES: a resource group for Data Integration
tenantId Long The ID of the tenant. taskId String The IDs of the instances for which the rule-based alert is triggered. remindName String The name of the rule. - Sample code of the message body for a rule-based alert for a resource group:
The following table describes the fields in the preceding sample code.{ "datacontenttype": "application/json;charset=utf-8", "data": { "alarmType": "REMIND_ALERT", "projectId": 2991, "remindId": 200000186, "remindName": "Alert for an exclusive resource group", "remindType": "RES_GROUP_THRESHOLD", "remindUnit": "GATEWAY_RES", "resourceGroupIdentifier": "S_res_group_195820716552192_1650965857744", "resourceGroupName": "emr_exclusive_scheduld", "resourceGroupType": "GATEWAY", "tenantId": 195820716552192 }
Field Type Description alarmType String The type of the alert. Valid values: - SLA_ALERT
- REMIND_ALERT
- TOPIC_ALERT
remindId Long The ID of the rule. remindType Stirng The trigger condition of the rule. Valid values: - FINISHED: The object finishes running.
- UNFINISHED: The object does not finish running.
- ERROR: An error occurs when the object runs.
- CYCLE_UNFINISHED: The object does not finish running in the current scheduling cycle.
- TIMEOUT: The execution of the object times out.
- RES_GROUP_THRESHOL: The resource usage of a resource group reaches the upper limit of the threshold.
- RES_GROUP_WAIT_AMOUNT: The number of instances that are run on a resource group and are waiting for the resources in the resource group reaches the upper limit of the threshold.
projectId Long The ID of the workspace. The workspace contains the node for which the rule-based alert is triggered. remindUnit String The type of the object for which the rule-based alert is triggered. Valid values: - NODE: a node
- GATEWAY_RES: an exclusive resource group for scheduling
- DI_RES: a resource group for Data Integration
tenantId Long The ID of the tenant. remindName String The name of the rule. resourceGroupIdentifier String The ID of the resource group. resourceGroupName String The name of the resource group. resourceGroupType String The type of the resource group. Valid values: - GATEWAY: a resource group for scheduling
- DI: a resource group for Data Integration
Message body for a status change event for a workflow
Sample code of the message body for a status change event for a workflow:{
"datacontenttype": "application/json;charset=utf-8",
"data": {
"bizDate": "2022-11-07 00:00:00",
"createTime": "2022-11-08 10:56:32",
"dagId": 500358972116,
"dagName": "P_test_spark_true_copy_20221108_105631",
"dagType": 3,
"flowId": 1,
"flowName": "ATCLOUD_FLOW",
"operator": "1107550004253538",
"projectEnv": "PROD",
"projectId": 258463,
"status": 6,
"tenantId": 524257424564736
}
The following table describes the fields in the preceding sample code. Field | Type | Description |
---|---|---|
bizDate | String | The data timestamp of the workflow. The data timestamp is in the yyyy-mm-dd hh24:mi:ss format. |
createTime | String | The time when the workflow was created. The time is in the yyyy-mm-dd hh24:mi:ss format. |
dagId | Long | The ID of the DAG. You can query the details of the DAG based on the dagId field. |
dagName | String | The name of the workflow. |
dagType | Integer | The type of the DAG. Valid values:
|
flowId | Integer | The ID of the workflow. |
flowName | String | The name of the workflow. |
operator | String | The UID of the user who created the workflow. |
projectEnv | String | The environment to which the workflow belongs. Valid values:
|
tenantId | Long | The ID of the tenant of the workspace to which the auto triggered node instance belongs. |
projectId | Long | The ID of the workspace. |
status | Integer | The status of the node in the workflow. Valid values:
|
Message body for a check result feedback event
Sample code of the message body for a check result feedback event:
{
"datacontenttype": "application/json;charset=utf-8",
"data": {
"ruleCheckId": 521771452,
"feedbackContent": "Skip",
"ruleId": 28610334,
"createUser": "110755000425****",
"taskId": "1676448144650974278b99ee64ca9a26ecf4063a88797",
"beginTime": "1676448145000",
"envType": "ODPS",
"projectName": "test_mc_2303_kongjian",
"projectId": 242601,
"tenantId": 524257424564736
}
Field | Type | Description |
---|---|---|
ruleId | Long | The ID of the monitoring rule. |
ruleCheckId | Long | The auto-increment ID of the data quality check result. |
feedbackContent | String | The feedback content. |
createUser | String | The ID of the user who provides the feedback. |
taskid | String | The ID of the node that is associated with the monitoring rule. |
beginTime | String | The time when the feedback is provided. |
envType | String | The type of the data source to which the table associated with the monitoring rule belongs. Valid values: ODPS, EMR, and HOLO. |
projectName | String | The name of the data source to which the table associated with the monitoring rule belongs. |
projectId | Long | The ID of the workspace. |
tenantId | Long | The ID of the DataWorks tenant. |
Message body for a check completion event
Sample code of the message body for a check completion event:
{
"datacontenttype": "application/json;charset=utf-8",
"data": {
"projectId": 242601,
"tenantId": 52425742456****,
"id": 52177****,
"taskId": "1676463288449d1b2102ac40048aeb7a628753363****",
"entityId": 1562***,
"ruleId": 28610334,
"property": "-",
"bizdate": "2023-02-09 00:00:00",
"dateType": "YMD",
"actualExpression": "ds\u003d20230210",
"matchExpression": "ds\u003d$[yyyymmdd]",
"blockType": 1,
"checkResult": 0,
"checkResultStatus": 0,
"methodName": "table_count",
"beginTime": "2023-02-15 20:14:48",
"endTime": "2023-02-15 20:14:55",
"timeConsuming": "7s",
"externalType": "CWF2",
"externalId": "triggerByManual",
"discrete": false,
"fixedCheck": true,
"referenceValue": [
{
"bizDate": "3000-12-31 00:00:00",
"discreteProperty": "Number of rows, one day difference",
"value": 0.0,
"singleCheckResult": 0
}
],
"sampleValue": [
{
"bizDate": "2023-02-09 00:00:00",
"value": 3.0
}
],
"trend": "\u003e\u003d",
"expectValue": 0.0,
"op": "\u003e\u003d",
"projectName": "test_mc_2303_kongjian",
"tableName": "sx_dim_1209_001",
"templateId": 47,
"checkerType": 0,
"ruleName": "Difference compared with the previous day",
"isPrediction": false,
"feedbackStatus": 0,
"whetherToFilterDirtyData": false
}
Field | Type | Description | |
---|---|---|---|
id | Long | The ID of the primary key for the check process. A primary key ID record is added each time the rule is triggered. | |
projectId | Long | The ID of the workspace. | |
tenantId | Long | The ID of the DataWorks tenant. | |
taskId | String | The ID of the monitoring task. | |
entityId | Long | The ID of the partition filter expression. | |
ruleId | Long | The ID of the rule. | |
property | String | The field whose data quality is checked based on the monitoring rule. This field is a column in the data source table that is monitored. | |
bizdate | Long | The data timestamp. If the quality of offline data needs to be checked, the value is one day earlier than the time when you perform the data quality check in most cases. | |
dateType | String | The scheduling frequency. In most cases, the value is YMD, which indicates that the node is scheduled by year, month, or day. | |
actualExpression | String | The partition in the monitored data source table. | |
matchExpression | String | The partition filter expression. | |
blockType | Integer | The strength of the monitoring rule. The strength of a monitoring rule indicates the importance of the rule. Valid values:
| |
checkResult | Integer | The status of the check result. Valid values:
| |
methodName | String | The method used to collect sample data, such as avg, count, sum, min, max, count_distinct, user_defined, table_count, table_size, table_dt_load_count, table_dt_refuseload_count, null_value, null_value/table_count, (table_count-count_distinct)/table_count, or table_count-count_distinct. | |
beginTime | Long | The time when the check process started. | |
endTime | Long | The time when the check process ended. | |
timeConsuming | String | The time that was taken to run the monitoring task. | |
externalType | String | The type of the scheduling system. Only CWF2 is supported. If the externalType field is left empty, a manual test node is used. | |
externalId | String |
| |
discrete | Boolean | Indicates whether discrete monitoring is performed. Valid values:
| |
fixedCheck | Boolean | Indicates whether the monitoring is performed based on a fixed value. Valid values:
| |
referenceValue | The historical sample values. | ||
bizDate | Long | The data timestamp. If the quality of offline data needs to be checked, the value is one day earlier than the time when you perform the data quality check in most cases. | |
discreteProperty | String | The values of the sample field that are grouped by using the GROUP BY clause. For example, the values of the Gender field are grouped by using the GROUP BY clause. In this case, the values of DiscreteProperty are Male, Female, and null. | |
value | Decimal | The sample value. | |
singleCheckResult | Integer | The string of the check result. | |
sampleValue | The current sample values. | ||
bizDate | Long | The data timestamp. If the quality of offline data needs to be checked, the value is one day earlier than the time when you perform the data quality check in most cases. | |
value | Decimal | The sample value. | |
trend | String | The trend of the check result. | |
expectValue | Double | The expected value. | |
op | String | The comparison operator. | |
projectName | String | The name of the compute engine or data source whose data quality you want to monitor. | |
tableName | String | The name of the table that is monitored. | |
templateId | Integer | The ID of the data quality monitoring template you use. | |
checkerType | Integer | The type of the checker. | |
ruleName | String | The name of the rule. | |
isPrediction | Boolean | Indicates whether the check result meets your business requirements. Valid values:
| |
comment | String | The description of the monitoring rule. |
Appendix 1: Authorized content (MaxCompute)
The following code shows
the data in arrays
when the value of the assignmentCategory
field is MaxCompute: [
{
"ownerBaseId":"1822931104031845",
"columns":[
{
"columnType":"decimal(38,18)",
"columnGuid":"odps.px_wokebench_1.oracle_dss_a1214_1136_1_invalid_column.i__d",
"columnComment":"",
"columnName":"i__d"
}
],
"columnMetaList":[
{
"columnType":"decimal(38,18)",
"columnGuid":"odps.px_wokebench_1.oracle_dss_a1214_1136_1_invalid_column.i__d",
"columnComment":"",
"columnName":"i__d"
}
],
"objectType":"TABLE",
"odpsTable":"oracle_dss_a1214_1136_1_invalid_column",
"envType":1,
"projectGuid":"odps.px_wokebench_1",
"objectGuid":"odps.px_wokebench_1.oracle_dss_a1214_1136_1_invalid_column",
"tenantId":0,
"objectName":"oracle_dss_a1214_1136_1_invalid_column",
"ownerAccountName":"ALIYUN$dataworks_3h1_1(dataworks_3h1_1)",
"odpsProject":"dataworks_pref_test1113",
"projectName":"dataworks_pref_test1113",
"actions":[
"Select",
"Describe"
],
"projectId":107100,
"workspaceId":"107100"
}
]
The following table describes the key fields in the preceding sample code. Field | Type | Description |
---|---|---|
ownerBaseId | String | The base ID of the table owner. |
ownerAccountName | String | The name of the table owner. |
objectType | String | The type of the object. |
odpsTable | String | The name of the table. |
envType | String | The type of the environment in which the table is used. |
projectGuid | String | The Globally Unique Identifier (GUID) of the MaxCompute project. |
objectGuid | String | The GUID of the object. |
objectName | String | The name of the object. |
odpsProject | String | The name of the MaxCompute project. |
projectName | String | The code of the MaxCompute project. |
projectId | Long | The ID of the project. |
workspaceId | String | The ID of the workspace. |
actions | Array | The actions. |
columns | Array | The columns. |
columnType | String | The type of the column. |
columnGuid | String | The GUID of the column. |
columnComment | String | The comment for the column. |
columnName | String | The name of the column. |
columnMetaList | Array | The metadata for the column. |
Appendix 2: Authorized content (DataService)
The following code shows
the data in arrays
when the value of the assignmentCategory
field is DataService: [
{
"resourceId":"DsApiDeploy/1822931104031845/workspaceId/227859/dsDeployId/1|417381955947846|1|280749521950784|227859",
"ownerName":"ysf01115618",
"resourceVersion":1,
"name":"api_api",
"dsDeployId":"1|417381955947846|1|280749521950784|227859",
"workspaceName":"da_simple_202112",
"id":"417381955947846",
"type":1,
"ownerId":"258882139554032242",
"url":"https://ds-cn-shanghai.data.aliyun.com/?projectId=227859&type=api&id=417381955947846&version=1&defaultProjectId=227859",
"workspaceId":"227859"
}
]
The following table describes the fields in the preceding sample code. Field | Type | Description |
---|---|---|
resourceId | String | The ID of the resource. |
ownerName | String | The name of the resource owner. |
resourceVersion | Long | The version of the resource. |
name | String | The name of the resource. |
dsDeployId | String | The ID of the API published in DataService Studio. |
workspaceName | String | The name of the workspace. |
id | String | The unique identifier of the resource. |
type | String | The type of the resource. Valid values:
|
ownerId | String | The base ID of the resource owner. |
url | String | The URL for DataService Studio. |
workspaceId | String | The ID of the workspace. |