The MaxCompute input widget enables data reading from MaxCompute sources. When syncing data from MaxCompute to other sources, it's essential to configure the MaxCompute input widget's source and the target data source for synchronization. This topic guides you through the configuration process.
Prerequisites
A MaxCompute data source is created. For more information, see Create a MaxCompute data source.
To configure the properties of the MaxCompute input widget, the account must possess read-through permission for the data source. If you lack this permission, you must obtain it from the data source. For more information, see Request data source permission.
Procedure
Select Development > Data Integration from the top menu bar on the Dataphin home page.
In the integration page's top menu bar, select Project (Dev-Prod mode requires selecting an environment).
In the left-side navigation pane, click Batch Pipeline. Then, from the Batch Pipeline list, select the offline pipeline you want to develop to access its configuration page.
To open the Component Library panel, click Component Library located in the upper right corner of the page.
In the Component Library panel's left-side navigation pane, select Input. Then locate the MaxCompute component within the right-side input widget list and drag it onto the canvas.
Click the
icon on the MaxCompute input widget card to open the MaxCompute Input Configuration dialog box.In the MaxCompute Input Configuration dialog box, you can configure the necessary parameters.
Parameter
Description
Step Name
This is the name of the MaxCompute input widget. Dataphin automatically generates the step name, but you can modify it based on the business scenario. The naming convention is as follows:
Can only contain Chinese characters, letters, underscores (_), and numbers.
Cannot exceed 64 characters.
Datasource
The data source drop-down list displays all MaxCompute-type data sources, including those for which you have read-through permission and those for which you do not. Click the
icon to copy the current data source name.NoteWhen the compute engine is MaxCompute, you can select a logical table and switch to the logical table input widget for integration.
For data sources without read-through permission, you can click the request button after the data source to request read-through permission for the data source. For more information, see Request data source permission.
If you do not yet have a MaxCompute-type data source, click Create to create a data source. For more information, see Create a MaxCompute data source.
Table
Select the source table for data synchronization. You can enter a table name keyword to search or enter the exact table name and then click Exact Search. After selecting a table, the system will automatically perform table status detection. Click the
icon to copy the name of the currently selected table.Partition
If the table you selected is a partitioned table, you need to enter partition information. Supports reading static partitions or range partitions. Static partitions such as
ds=20230101ords1=2023,ds2=01; range partitions such as/*query*/ds >=20230101 and ds <= 20230107.When Partition Does Not Exist
You can choose the following policies to handle scenarios where the specified partition does not exist:
Set Task To Failed: Terminates the task and sets it to failed.
Set Task To Successful, No Data Written: The task runs successfully with no data written.
Use The Latest Non-empty Partition: Uses the current latest non-empty partition (max_pt) of the table as the partition to be synchronized. If the table does not have any partitions with data, the task reports an error and is set to failed.
Output Fields
The output fields area displays all field information of the selected table. If you do not need to output certain fields to downstream widgets, you can delete the corresponding fields:
NoteWhen the compute engine is MaxCompute, the output fields of the MaxCompute input widget support viewing the classification and grading of fields. Non-MaxCompute compute engines do not support this.
Single field deletion scenario: If you need to delete a small number of fields, you can click the
icon in the operation column to delete the extra fields.Batch field deletion scenario: If you need to delete many fields, you can click Field Management, select multiple fields in the Field Management dialog box, then click the
left shift icon to move the selected input fields to the unselected input fields and click Confirm to complete the batch deletion of fields.
Click Confirm to finalize the MaxCompute input widget property configuration.