Document Center
DataWorks
Update Records
Release notes of key features
Version history
DataWorks V2.0
DataWorks V3.0
Change history
Product Introduction
What is DataWorks?
Basic concepts
Scenarios
Data development process
Basic mode and standard mode
Pricing
Purchase guide
Billing
View spending details
Pay-as-you-go
Subscription (pre-payment)
DataWorks advanced editions
DataWorks resource plans
DataWorks exclusive resources
Billing standards of exclusive resource groups for scheduling
Performance metrics and billing standards of exclusive resource groups for Data Integration
Appendix
Usage notes of DataWorks resource groups
Feature comparison among DataWorks editions
Feature downgrades of DataWorks advanced editions upon expiration
DataWorks (pay-as-you-go) billing details
Preparation
Administrator Operations
Prepare an Alibaba Cloud account
Prepare a RAM user
Create a workspace
Add workspace members
Work as a workspace administrator
Use a RAM user
Quick Start
Overview
Create tables and import data
Create a workflow
Create a sync node
Configure recurrence and dependencies for a node
Run a node and troubleshoot errors
Optional: Use an ad hoc query node to run SQL statements
Tutorials
DataWorks for MaxCompute Workshop
Workshop introduction
Prepare the environment
Collect data
Process data
Configure rules to monitor data quality
Visualize data on a dashboard
Use Function Studio to develop a UDF
DataWorks for EMR Workshop
Prepare the environment
Collect data
Process data
Collect and view metadata
Configure rules to monitor data quality
Build an online operation analysis platform
Business scenarios and development process
Prepare the environment
Prepare data
Build data models and analyze data
Create tables
Design workflows
Configure nodes
Commit and test nodes
Display data on dashboards
Automatically identify users who steal electricity
Overview
Prepare the environment
Prepare data
Process data
Build a data model
Workbench
Overview of the DataWorks console
Workspace list
Resource group list
Create and use an exclusive resource group for scheduling
View a resource group list
View the resource usage of an exclusive resource group
O&M Assistant
Add a route
Create custom resource groups for scheduling
Exclusive resource group mode
Compute engine list
MaxCompute Management
Workspace management
Configure a workspace
Manage workspace members
Permission list
Configure MaxCompute
Manage connections
Upgrade the workspace mode
Data Integration
Overview
Homepage
Supported data stores
Common configurations
Test data store connectivity
Configure whitelists
Configure a security group
Batch data synchronization
Connections
Supported data stores, readers, and writers
Isolate connections between the development and production environments
Manage connection permissions
Node configuration
Create a sync node by using the codeless UI
Create a sync node by using the code editor
Synchronize incremental data
Synchronize data from tables in multiple databases to the same destination table
Migrate databases to MaxCompute
Rules and restrictions
Migrate all data in a MySQL database
Migrate all data in an Oracle database
Migrate multiple tables to the cloud at a time
Migrate multiple tables to the cloud at a time
Add multiple connections at a time
Add a prefix to the names of the destination tables
Resources for batch sync nodes
Real-time synchronization
Overview
Create, configure, commit, and manage real-time sync nodes
Supported data stores for real-time synchronization
Fields used for real-time synchronization
Reader
Configure MySQL Binlog Reader
Configure SQLServer CDC Reader
Configure DataHub Reader
Configure LogHub Reader
Configure Kafka Reader
Configure PolarDB Reader
Writer
Configure MaxCompute Writer
Configure Hologres Writer
Configure DataHub Writer
Configure Kafka Writer
Transformation
Configure Data Filtering
Configure String Replace
Sync solutions
Go to the Sync Solutions page
Synchronize data to MaxCompute
Configure a source Oracle data source
Configure a source MySQL data source
Add a data source
Node optimization
Optimize performance
Optimize a sync node
Resource groups
Overview
Use the default resource group
Exclusive resource group for Data Integration
Create a custom resource group for Data Integration
RAM authorization mode
Service linked role of DataWorks Data Integration
Use the RAM authorization mode to configure connections to data stores
Appendixes
Connection configuration
Reader configuration
Writer configuration
Data Development
Create a solution
SQL coding guidelines and specifications
GUI elements
GUI elements
Use workflow parameters
Lineage
Versions
Code Structure
Workflow
Manage workflows
Node group
Node types
Data Integration
Create a batch sync node
MaxCompute
Create an ODPS SQL node
Create an SQL component node
Create an ODPS Spark node
Create a PyODPS 2 node
Create a PyODPS 3 node
Create an ODPS Script node
Create an ODPS MR node
Create a MaxCompute table
Create MaxCompute resources
Create a MaxCompute function
AnalyticDB for PostgreSQL
Create an ADB for PostgreSQL node
Create an AnalyticDB for PostgreSQL table
E-MapReduce
E-MapReduce access modes
EMR Presto node
Create an EMR Hive node
Create an EMR MR node
Create an EMR Spark SQL node
Create an EMR Spark node
Create an EMR JAR resource
Create an E-MapReduce table
Create an E-MapReduce function
Hologres
Create a Hologres SQL node
Algorithm
Create a Machine Learning (PAI) node
General
OSS Object Inspection node
Create and configure a for-each node
Configure a do-while node
Configure a merge node
Configure a branch node
Configure an assignment node
Create a Shell node
Create a zero-load node
Create an HTTP trigger node
Cross-tenant collaboration node
Custom nodes
Create a Data Lake Analytics node
Create a Hologres development node
Create an AnalyticDB for MySQL node
MaxCompute
MaxCompute modules
MaxCompute functions
MaxCompute resources
Schedule
Basic properties
Scheduling parameters
Configure time properties
Dependencies
Configure the resource group
Immediate instance generation
Cross-cycle dependencies
Context-based parameters
Logic of scheduling dependencies
Script template management
Create a script template
Use a script template
Node configuration
Overview
Create a wrapper
Create a custom node type
Create a data quality wrapper
Create an ad hoc query
View tenant tables
Runtime logs
Manage manually triggered workflows
Manage tables
External table
Functions
Recycle bin
Setup
Setup
Configuration center
Workspace settings
Template management
Folder management
Level management
Workspace backup and restoration
Code review
Editor shortcuts
DataService Studio
Overview
Features
Create an API
Manage API tags
Use service orchestration
Use filters
Use prefilters
Use post filters
View API versions
Use business processes
Manage business processes
Manage APIs
Manage functions
Manage workflows
Generate API
Configure a connection
Create an API in the codeless UI
Create an API in the code editor
Register an API
View API statistics
Test an API
Publish an API
Authorize access to an API
Call an API
Appendix: DataService Studio error codes
DataAnalysis
Overview
Workbook
Create and manage a workbook
Import data to a workbook
Analyze data
Export, share, and download a workbook
Manage a template
MaxCompute table
Create and manage a MaxCompute table
Edit a MaxCompute table
Import data to a MaxCompute table
Share a MaxCompute table
Report
Create and manage a report
Edit a report
Save a report as a template
Share a report
Charts
Area charts
Pie charts
Horizontal bar charts
Scatter charts
Column charts
Stock charts
Line charts
Analyze data
Task Operation
Operation Center
Overview
View the dashboard
Auto triggered node O&M
View auto triggered nodes
Manage retroactive instances
View auto triggered node instances
Instance diagnosis
Test instances
Manually triggered node O&M
Manually triggered nodes
Manually triggered node instances
Monitor
Overview
Features
Instructions
Publish nodes
Deploy nodes
Delete a node
Cross-workspace cloning
Overview
Clone nodes across workspaces
Data governance
Security Center
Overview
Quick start
My Permissions
Authorizations
Approval Center
Data Quality
Overview
Go to the Overview page
View my subscriptions
Configure monitoring rules
View monitoring results
Manage templates
Create and manage report templates
Create, manage, and use rule templates
Instructions
Configure monitoring rules for DataHub
Configure monitoring rules for MaxCompute
Built-in rule templates for offline data
Data security guard
Overview
Discover data
View data activities
Data risk identification
Audit data
Set data identification rules
Customize de-identification rules
Manage data security levels
Manually correct data
Manage risk identification rules
Data Map
Overview
View overall data
View and manage data and data permissions
Manage categories and permissions on MaxCompute tables
Table details
View the details of a table
Apply for table permissions
Add a table to favorites
Go to DataService Studio to create API operations
Data discovery
Collect metadata from an E-MapReduce data store
Collect Tablestore metadata
Collect metadata from a MySQL data store
Collect metadata from an SQL Server data store
Collect metadata from a PostgreSQL data store
Collect metadata from an Oracle data store
Collect metadata from an AnalyticDB for PostgreSQL data store
Collect metadata from an AnalyticDB for MySQL 2.0 data store
Collect metadata from an AnalyticDB for MySQL 3.0 data store
Collect metadata from an OSS data store
Requirements Management
Overview
Create a requirement
Search for a requirement
Manage a requirement
Best practice of Requirements Management
Migration Assistant
Overview
Cloud tasks
Export tasks from open source engines
Import tasks of open source engines
Migration of data objects in DataWorks
Create and view export tasks
Create and view import tasks
Best Practices
Data migration
Migrate data across DataWorks workspaces
Migrate data from Hadoop to MaxCompute
Best practices for migrating data from Kafka to MaxCompute
Migrate JSON data from OSS to MaxCompute
Migrate JSON-formatted data from MongoDB to MaxCompute
Migrate data from Elasticsearch to MaxCompute
Connect an exclusive resource group for Data Integration to a data store that is deployed in a VPC
Add a relational database driver for RDBMS
Use Data Integration to import data to Elasticsearch
Configure an OTSStream sync node
Migrate data from a user-created MySQL database on an ECS instance to MaxCompute
Use Data Integration to import data to DataHub
Use Data Integration to ship data collected by LogHub to destinations
Best practice to migrate data from Oracle to MaxCompute
Automatically synchronize IoT data to the cloud
Data development
Best practice to configure scheduling dependencies
Use MaxCompute to query geolocations of IP addresses
Reference a third-party package in a PyODPS node
Run nodes at a specific time by using branch nodes
Connect DataV to DataWorks DataService Studio
Use a PyODPS node to segment Chinese text based on Jieba
Build a data warehouse for an enterprise based on AnalyticDB for MySQL
Use a PyODPS node to send emails
Data security
Grant access to a specific UDF to a specified user
Allow a RAM user to log on to DataWorks only from a specific IP address
Data analysis
Use MaxCompute to analyze data and Quick BI to present the analysis results
Analyze offline data in the Internet and e-commerce industries
Intelligently recommend items on e-commerce websites
Videos
DataWorks for MaxCompute Workshop
FAQ
Data Integration
Batch synchronization
Management of permissions on data sources
Custom resource groups
Exclusive resource groups
Scheduling parameters
Scheduling properties
Dependencies
Data Analytics
Monitor
Operation Center
DataService Studio
Security Center
App Studio
Stream Studio
Troubleshooting for connections
API
Overview
Tenant
ListCalcEngines
CreateConnection
DeleteConnection
ListConnections
UpdateConnection
ListResourceGroups
GetProjectDetail
AddProjectMemberToRole
CreateProjectMember
DeleteProjectMember
ListProjectRoles
RemoveProjectMemberFromRole
ListProjectMembers
ListProjects
Metadata
GetMetaDBInfo
GetMetaDBTableList
CheckMetaTable
CheckMetaPartition
SearchMetaTables
GetMetaTableBasicInfo
GetMetaTableColumn
GetMetaTableFullInfo
GetMetaTablePartition
GetMetaTableOutput
GetMetaTableChangeLog
GetMetaTableIntroWiki
GetMetaTableLineage
GetMetaColumnLineage
CreateMetaCategory
DeleteMetaCategory
UpdateMetaCategory
GetMetaCategory
GetMetaTableListByCategory
AddToMetaCategory
DeleteFromMetaCategory
CreateTable
UpdateTable
DeleteTable
GetDDLJobStatus
UpdateMetaTable
UpdateMetaTableIntroWiki
ListMetaDB
UpdateTableModelInfo
CreateTableTheme
DeleteTableTheme
UpdateTableTheme
ListTableTheme
CreateTableLevel
DeleteTableLevel
UpdateTableLevel
ListTableLevel
Data Development
CreateFile
CreateUdfFile
DeleteBusiness
CreateFolder
UpdateBusiness
ListFiles
EstablishRelationTableToBusiness
SubmitFile
ListBusiness
GetFileVersion
DeleteFolder
DeleteFile
GetDeployment
UpdateFile
GetFolder
ListFileVersions
GetFile
ListFolders
DeployFile
UpdateUdfFile
GetBusiness
UpdateFolder
Operation Center
ListAlertMessages
GetBaselineConfig
GetNodeOnBaseline
ListBaselineConfigs
GetBaselineKeyPath
GetBaselineStatus
ListBaselineStatuses
DeleteRemind
UpdateRemind
CreateRemind
GetRemind
ListReminds
ListTopics
GetTopic
GetTopicInfluence
GetNode
ListNodeIO
ListNodes
GetNodeCode
GetInstance
ListInstances
GetInstanceLog
StopInstance
RestartInstance
SetSuccessInstance
SuspendInstance
ResumeInstance
CreateDagComplement
CreateDagTest
GetSuccessInstanceTrend
GetInstanceCountTrend
GetInstanceConsumeTimeRank
GetInstanceErrorRank
ListProgramTypeCount
GetInstanceStatusCount
GetNodeTypeListInfo
CreateManualDag
GetManualDagInstances
GetDag
SearchNodesByOutput
RunTriggerNode
Data Quality
UpdateQualityFollower
UpdateQualityRule
GetQualityRule
ListQualityRules
CreateQualityRule
DeleteQualityFollower
DeleteQualityRule
GetQualityFollower
CreateQualityFollower
DeleteQualityEntity
GetQualityEntity
CreateQualityEntity
CreateQualityRelativeNode
DeleteQualityRelativeNode
ListQualityResultsByEntity
ListQualityResultsByRule
DataService Studio
CreateDataServiceApi
PublishDataServiceApi
DeleteDataServiceApi
ListDataServiceApis
GetDataServiceApi
UpdateDataServiceApi
GetDataServicePublishedApi
ListDataServicePublishedApis
ListDataServiceApiAuthorities
ListDataServiceAuthorizedApis
AbolishDataServiceApi
GetDataServiceApplication
ListDataServiceApplications
CreateDataServiceFolder
GetDataServiceFolder
ListDataServiceFolders
CreateDataServiceGroup
GetDataServiceGroup
ListDataServiceGroups
CreateDataServiceApiAuthority
DeleteDataServiceApiAuthority
SDK
Install the Alibaba Cloud SDK for Java
All Products
Search
Document Center
DataWorks
Data Integration
Appendixes
Reader configuration
This Product
This Product
All Products
DataWorks
DRDS Reader
HBase Reader
HDFS Reader
MongoDB Reader
Db2 Reader
MySQL Reader
Oracle Reader
OSS Reader
FTP Reader
Tablestore Reader
AnalyticDB for MySQL 3.0 Reader
SQL Server Reader
Configure LogHub Reader
OTSReader-Internal
RDBMS Reader
Stream Reader
HybridDB for MySQL Reader
AnalyticDB for PostgreSQL Reader
PolarDB Reader
Elasticsearch Reader
Kafka Reader
MaxCompute Reader
InfluxDB Reader
OpenTSDB Reader
Prometheus Reader
PostgreSQL Reader
OTSStream Reader
MetaQ Reader
Hive Reader
Vertica Reader
Gbase8a Reader
DataHub Reader
ApsaraDB For OceanBase Reader
Hologres Reader
TSDB Reader
GDB Reader
RestAPI Reader
Free Trial
Free Trial