DataWorks:MaxCompute data sources

The following table lists the data type mappings based on which MaxCompute Reader converts data types.

You can use the account of a project owner to run the following command on the MaxCompute client to enable the atomicity, consistency, isolation, and durability (ACID) semantics. For more information about the ACID semantics, see ACID semantics.

setproject odps.sql.acid.table.enable=true;

If you want to use the TIMESTAMP data type in MaxCompute V2.0, use the account of a project owner to run the following command to enable the MaxCompute V2.0 data type edition:

setproject odps.sql.type.system.odps2=true;

After you associate a MaxCompute compute engine with a workspace as a compute engine instance, DataWorks generates a default MaxCompute data source in the workspace. You can use the default MaxCompute data source for data synchronization in the current workspace. If you want to access data in the default MaxCompute data source of the current workspace from another workspace, you must create an AccessKey pair. This way, you can access data in the compute engine instance by using the AccessKey pair when you create a MaxCompute data source or use the default MaxCompute data source in another workspace.

• Create an AccessKey pair. For more information, see Create an Alibaba Cloud account.

• Create a MaxCompute data source. For more information, see Add a MaxCompute data source.

Before you configure a synchronization task to synchronize data from or to MaxCompute, you must associate a MaxCompute compute engine with a DataWorks workspace and add a MaxCompute data source to DataWorks. When you associate a MaxCompute compute engine with a workspace, DataWorks automatically generates a default MaxCompute data source based on the compute engine.

• When you associate a MaxCompute compute engine with a workspace, DataWorks automatically generates a MaxCompute data source on the Data Source page in the DataWorks console based on the configurations of the compute engine. You can also add another MaxCompute project to DataWorks as a data source. For more information about the configuration procedure, see Add and manage data sources.

• When you associate a MaxCompute compute engine with a workspace for the first time, DataWorks generates a default MaxCompute data source named odps_first on the Data Source page in the DataWorks console. If you associate other MaxCompute compute engines with the workspace subsequently, DataWorks generates MaxCompute data sources named in the format of 0_Region ID_Compute engine instance name.

• The names of the MaxCompute projects based on which the default MaxCompute data sources are generated are the same as the names of the MaxCompute projects that are associated with the workspace as compute engine instances. You can go to the Workspace page in SettingCenter in the DataWorks console to view the information of the MaxCompute compute engine instances that are associated with a workspace. If you want to modify the information of a MaxCompute compute engine instance that is associated with a workspace, you must make sure that no tasks are running on the compute engine instance before you perform the operation. The tasks include synchronization tasks of Data Integration, tasks of DataStudio, and other tasks that are related to DataWorks. For information about how to view the information of a MaxCompute compute engine instance, see Associate a MaxCompute compute engine with a workspace.

{
    "type":"job",
    "version":"2.0",
    "steps":[
        {
            "stepType":"odps",// The plug-in name. 
            "parameter":{
                "partition":[],// The names of the partitions from which you want to read data. 
                "isCompress":false,// Specifies whether to enable compression. 
                "datasource":"",// The name of the data source. 
                "column":[// The names of the columns. 
                    "id"
                ],
                "emptyAsNull":true,
                "table":""// The name of the table. 
            },
            "name":"Reader",
            "category":"reader"
        },
        { 
            "stepType":"stream",
            "parameter":{
            },
            "name":"Writer",
            "category":"writer"
        }
    ],
    "setting":{
        "errorLimit":{
            "record":"0"// The maximum number of dirty data records allowed. 
        },
        "speed":{
            "throttle":true,// Specifies whether to enable throttling. The value false indicates that throttling is disabled, and the value true indicates that throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true. 
            "concurrent":1, // The maximum number of parallel threads. 
            "mbps":"12"// The maximum transmission rate. Unit: MB/s. 
        }
    },
    "order":{
        "hops":[
            {
                "from":"Reader",
                "to":"Writer"
            }
        ]
    }
}

If you want to specify the Tunnel endpoint of MaxCompute, you can use the code editor to configure the data source. To configure the data source, replace "datasource":"", in the preceding code with the parameters of the data source. The following code provides an example:

"accessId":"*******************",
"accessKey":"*******************",
"endpoint":"http://service.eu-central-1.maxcompute.aliyun-inc.com/api",
"odpsServer":"http://service.eu-central-1.maxcompute.aliyun-inc.com/api", 
"tunnelServer":"http://dt.eu-central-1.maxcompute.aliyun.com", 
"project":"*****", 

Sample code:

{
    "type":"job",
    "version":"2.0",// The version number. 
    "steps":[
        {
            "stepType":"stream",
            "parameter":{},
            "name":"Reader",
            "category":"reader"
        },
        {
            "stepType":"odps",// The plug-in name. 
            "parameter":{
                "partition":"",// The name of the partitions to which you want to write data. 
                "truncate":true,// The write rule. 
                "compress":false,// Specifies whether to enable compression. 
                "datasource":"odps_first",// The name of the data source. 
            "column": [// The names of the columns. 
                "id",
                "name",
                "age",
                "sex",
                "salary",
                "interest"
                ],
                "emptyAsNull":false,// Specifies whether to convert empty strings to null. 
                "table":""// The name of the table. 
            },
            "name":"Writer",
            "category":"writer"
        }
    ],
    "setting":{
        "errorLimit":{
            "record":"0"// The maximum number of dirty data records allowed. 
        },
        "speed":{
            "throttle":true,// Specifies whether to enable throttling. The value false indicates that throttling is disabled, and the value true indicates that throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true. 
            "concurrent":1, // The maximum number of parallel threads. 
            "mbps":"12"// The maximum transmission rate. Unit: MB/s. 
        }
    },
    "order":{
        "hops":[
            {
                "from":"Reader",
                "to":"Writer"
            }
        ]
    }
}

If you want to specify the Tunnel endpoint, you can configure the data source in the code editor. To configure the data source, replace "datasource":"", in the preceding code with detailed parameters of the data source. Example:

"accessId":"<yourAccessKeyId>",
 "accessKey":"<yourAccessKeySecret>",
 "endpoint":"http://service.eu-central-1.maxcompute.aliyun-inc.com/api",
 "odpsServer":"http://service.eu-central-1.maxcompute.aliyun-inc.com/api", 
"tunnelServer":"http://dt.eu-central-1.maxcompute.aliyun.com", 
"project":"**********",