Dataphin:Configure MaxCompute output component

Procedure

1. On the Dataphin home page, navigate to the top menu bar and select Development > Data Integration.

2. At the top menu bar of the integration page, select Project (select the environment in Dev-Prod mode).

3. In the left-side navigation pane, click Batch Pipeline. From the Batch Pipeline list, select the offline pipeline you want to develop to access its configuration page.

4. To open the Component Library panel, click Component Library at the top right corner of the page.

5. In the Component Library panel's left-side navigation, select Output. Find the MaxCompute component in the list and drag it onto the canvas.

6. Drag the target input component's icon to connect it with the MaxCompute output component.

7. Click the icon on the MaxCompute output component card to open the MaxCompute Output Configuration dialog box.

8. Within the MaxCompute Output Configuration dialog box, set the necessary parameters.

   Parameter		Description
   Basic Settings	Step Name	This is the name of the MaxCompute output component. Dataphin automatically generates the step name, and you can also modify it according to the business scenario. The naming convention is as follows: Can only contain Chinese characters, letters, underscores (_), and numbers. Cannot exceed 64 characters.
   	Datasource	In the data source drop-down list, all MaxCompute type data sources are displayed, including data sources for which you have write-through permissions and those for which you do not. Click the icon to copy the current data source name. For data sources without write-through permissions, you can click Request after the data source to request write-through permissions for the data source. For specific operations, see Request, Renew, and Return Data Source Permissions. If you do not have a MaxCompute type data source yet, click Create Data Source to create a data source. For specific operations, see Create MaxCompute Data Source.
   	Table	Select the target table for output data in data synchronization. If there is no target table for data synchronization in the MaxCompute data source, you can use the One-click Generate Target Table feature to quickly and easily generate a target table. The detailed procedure is as follows: Click One-click Create Table. You can choose Standard Table or Delta Table, with the default being a standard table. Switching the table format will change the DDL in the code area below. Dataphin will automatically match the code for creating the target table for you, including the target table name (default is the source table name), field types (initially transformed based on Dataphin fields), and other information. You can modify the SQL script for creating the target table according to the business situation, and then click Create. After the target table is successfully created, Dataphin automatically sets the newly created target table as the target table for output data. Note One-click Create Table is used to create target tables for data synchronization in development and production environments. Dataphin selects Create Table In Production Environment by default. If there is a table with the same name and structure in the production environment, you do not need to select Create Table In Production Environment. If there is a table with the same name in the development or production environment, after clicking Create, Dataphin will report an error indicating that the table already exists.
   	Loading Policy	When writing data to the target data source (MaxCompute data source), the strategy for writing data into the table. Loading strategies include overwriting data and appending data. The applicable scenarios are described as follows: Overwrite Data: Overwrite the historical data in the target table based on the current source table. Append Data: Append data to the existing data in the target table without modifying historical data. Only available for standard tables. Update Data: When there is a primary key conflict, update the mapped field data on existing records. Only available for Delta tables.
   	Partition	If a partitioned table is selected, you need to fill in the partition of the data table. For example, ds=20230101 or use the parameter ds=${bizdate}. It supports adding the /*dynamic*/ keyword before the partition to dynamically specify the write partition for each row of data using the source field. For example, /*dynamic*/ds=$date, where date is the source field name; or /*dynamic*/ds=${bizdate},hh = $hour, where bizdate is a parameter, and hour is the source table field name. Important The upper limit for dynamic partitions is 10,000. It is recommended not to exceed 1,000, otherwise, it may result in excessively long execution times or even failure.
   Field Mapping	Input Field	The input fields are displayed for you based on the output of the upstream component.
   	Output Field	The output field area displays all fields of the selected table. If certain fields do not need to be output to downstream components, you can delete the corresponding fields: If you need to delete a small number of fields, you can click the Actions column's icon to delete the extra fields. If you need to delete many fields, you can click Field Management, select multiple fields on the Field Management page, and then click the icon to move the Selected Input Fields to Unselected Input Fields.
   	Mapping	The mapping relationship is used to map the input fields of the source table to the output fields of the target table. Mapping relationships include same-name mapping and same-row mapping. The applicable scenarios are described as follows: Same-name Mapping: Map fields with the same field name. Same-row Mapping: The field names of the source table and the target table are inconsistent, but the data in the corresponding rows of the fields need to be mapped. Only map fields in the same row.

9. Click Confirm to finalize the property settings for the MaxCompute output component.