MaxCompute Reader

MaxCompute Reader cannot read views. It can read only partitioned and non-partitioned tables. DataWorks cannot map the fields in partitioned MaxCompute tables. If you want to read data from a partitioned MaxCompute table, you must specify each desired partition when you configure MaxCompute Reader. For example, if you want to read data from the partition pt=1,ds=hangzhou in the t0 table, you must specify pt=1,ds=hangzhou when you configure MaxCompute Reader. In addition, you can select some or all of the table fields, change the order in which the fields are arranged, or add constant fields and partition key columns. Partition key columns are not table fields.

The following table lists the data types that are supported by MaxCompute Reader.

Category	Data Integration data type	MaxCompute data type
Integer	LONG	BIGINT, INT, TINYINT, and SMALLINT
Boolean	BOOLEAN	BOOLEAN
Date and time	DATE	DATETIME, TIMESTAMP, and DATE
Floating point	DOUBLE	FLOAT, DOUBLE, and DECIMAL
Binary	BYTES	BINARY
Complex	STRING	ARRAY, MAP, and STRUCT

Perform the following steps on the configuration tab of the synchronization node:

1. Configure data sources.
   Configure Source and Target for the synchronization node.

   Parameter	Description
   Connection	The name of the data source from which you want to read data. This parameter is equivalent to the datasource parameter that is described in the preceding section.
   Development Project Name	The name of the project in the development environment. You cannot change the value. Note This parameter is displayed only if the workspace is in standard mode.
   Production Project Name	The name of the project in the production environment. You cannot change the value.
   Table	The name of the table from which you want to read data. This parameter is equivalent to the table parameter that is described in the preceding section.
   Partition Key Column	If your daily incremental data is stored in the partitions of a specific date, you can specify the partition information to synchronize the daily incremental data. For example, set pt to ${bizdate}. Note DataWorks cannot map the fields in partitioned MaxCompute tables. If you want to read data from a partitioned MaxCompute table, you must specify each desired partition when you configure MaxCompute Reader.

   Note In the code editor, if you want to synchronize all the columns in the source table, specify "column": [""]. You can directly specify the partitions from which you want to read data. You can also use wildcards to specify the partitions.

   • "partition":"pt=20140501/ds=*" indicates all the ds partitions in the pt=20140501 partition.
   • "partition":"pt=top?" indicates partitions pt=top and pt=to.

   You can specify the partitions from which you want to synchronize data. For example, if you want to synchronize data from the partition pt=${bdp.system.bizdate} in a MaxCompute table, you can add the pt column to the source table in the Mappings section. If the value of Type for pt is Unidentified, you can ignore the value and proceed to the next step.

   • To synchronize data in all partitions, specify pt=* for the Partition Key Column parameter.
   • To synchronize data in specific partitions, specify the required dates.
2. Configure field mappings. This operation is equivalent to setting the column parameter that is described in the preceding section.
   Fields in the source on the left have a one-to-one mapping with fields in the destination on the right. You can click Add to add a field. To remove an added field, move the pointer over the field and click the Remove icon.

   Operation	Description
   Map Fields with the Same Name	Click Map Fields with the Same Name to establish mappings between fields with the same name. The data types of the fields must match.
   Map Fields in the Same Line	Click Map Fields in the Same Line to establish mappings between fields in the same row. The data types of the fields must match.
   Delete All Mappings	Click Delete All Mappings to remove the mappings that are established.
   Auto Layout	Click Auto Layout. Then, the system automatically sorts the fields based on specified rules.
   Change Fields	Click the Change Fields icon. In the Change Fields dialog box, you can manually edit the fields in the source table. Each field occupies a row. The first and the last blank rows are included, whereas other blank rows are ignored.
   Add	Click Add to add a field. Take note of the following rules when you add a field: You can enter constants. Each constant must be enclosed in single quotation marks ('), such as 'abc' and '123'. You can use scheduling parameters, such as ${bizdate}. If the field that you entered cannot be parsed, the value of Type for the field is Unidentified.

3. Configure channel control policies.

   Parameter	Description
   Expected Maximum Concurrency	The maximum number of parallel threads that the synchronization node uses to read data from the source or write data to the destination. You can configure the parallelism for the synchronization node on the codeless UI.
   Bandwidth Throttling	Specifies whether to enable bandwidth throttling. You can enable bandwidth throttling and specify a maximum transmission rate to avoid heavy read workloads on the source. We recommend that you enable bandwidth throttling and set the maximum transmission rate to an appropriate value based on the configurations of the source.
   Dirty Data Records Allowed	The maximum number of dirty data records allowed.
   Distributed Execution	The distributed execution mode that allows you to split your node into pieces and distribute them to multiple Elastic Compute Service (ECS) instances for parallel execution. This speeds up synchronization. If you use a large number of parallel threads to run your synchronization node in distributed execution mode, excessive access requests are sent to the data sources. Therefore, before you use the distributed execution mode, you must evaluate the access load on the data sources. You can enable this mode only if you use an exclusive resource group for Data Integration. For more information about exclusive resource groups for Data Integration, see Exclusive resource groups for Data Integration and Create and use an exclusive resource group for Data Integration.

In the following code, a synchronization node is configured to read data from MaxCompute. For more information about how to configure a synchronization node by using the code editor, see Create a synchronization node by using the code editor.

{
    "type":"job",
    "version":"2.0",
    "steps":[
        {
            "stepType":"odps",// The reader type. 
            "parameter":{
                "partition":[],// The partitions from which you want to read data. 
                "isCompress":false,// Specifies whether to enable compression. 
                "datasource":"",// The name of the data source. 
                "column":[// The names of the columns from which you want to read data. 
                    "id"
                ],
                "emptyAsNull":true,
                "table":""// The name of the table from which you want to read data. 
            },
            "name":"Reader",
            "category":"reader"
        },
        { 
            "stepType":"stream",
            "parameter":{
            },
            "name":"Writer",
            "category":"writer"
        }
    ],
    "setting":{
        "errorLimit":{
            "record":"0"// The maximum number of dirty data records allowed. 
        },
        "speed":{
            "throttle": true,// Specifies whether to enable bandwidth throttling. The value false indicates that bandwidth throttling is disabled, and the value true indicates that bandwidth throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true. 
            "concurrent":1, // The maximum number of parallel threads. 
            "mbps":"12"// The maximum transmission rate.
        }
    },
    "order":{
        "hops":[
            {
                "from":"Reader",
                "to":"Writer"
            }
        ]
    }
}

If you want to specify the Tunnel endpoint of MaxCompute, you can use the code editor to configure the data source. To configure the data source, replace "datasource":"", in the preceding code with the parameters of the data source. The following code provides an example:

"accessId":"*******************",
"accessKey":"*******************",
"endpoint":"http://service.eu-central-1.maxcompute.aliyun-inc.com/api",
"odpsServer":"http://service.eu-central-1.maxcompute.aliyun-inc.com/api", 
"tunnelServer":"http://dt.eu-central-1.maxcompute.aliyun.com", 
"project":"*****",