All Products
Search
Document Center

DataWorks:Configure MaxCompute Writer

Last Updated:Aug 01, 2023

MaxCompute provides a comprehensive data import scheme that supports the fast computing of large amounts of data.

Prerequisites

A reader or conversion node is configured. For more information, see Overview of the real-time synchronization feature.

Procedure

  1. Go to the DataStudio page.

    1. Log on to the DataWorks console.

    2. In the left-side navigation pane, click Workspaces.

    3. In the top navigation bar, select the region in which the workspace that you want to manage resides. On the Workspaces page, find the workspace and click Shortcuts > Data Development in the Actions column.

  2. In the Scheduled Workflow pane, move the pointer over the Create a table icon and choose Create Node > Data Integration > Real-time synchronization.

    Alternatively, right-click the required workflow, and then choose Create Node > Data Integration > Real-time synchronizationReal-time synchronization.

  3. In the Create Node dialog box, set the Sync Method parameter to End-to-end ETL and configure the Name and Path parameters.

    Important

    The node name cannot exceed 128 characters in length and can contain letters, digits, underscores (_), and periods (.).

  4. Click Confirm.

  5. On the configuration tab of the real-time sync node, drag MaxCompute in the Output section to the canvas on the right. Connect the MaxCompute node to the configured reader or conversion node.
  6. Click the MaxCompute node. In the Node configuration panel, set the parameters. The following table describes the parameters.
    MaxCompute
    ParameterDescription
    Data sourceThe MaxCompute data source that you configured. You can select only a MaxCompute data source.

    If no data source is available, click New data source on the right to add a data source on the Data Source page. For more information, see Add a MaxCompute data source.

    TableThe name of the MaxCompute table to which you want to write data.
    You can click Create Table on the right to create a table, or click Data preview to preview the selected table.
    Important Before you create a table, connect the MaxCompute node to a reader node and make sure that the output fields are specified for the reader node.
    ModeThe mode in which data is written to the destination partitions of the MaxCompute table. Valid values: Partitioning by Time and Dynamic Partitioning by Field Value. If you select Partitioning by Time, data is written to the destination partitions of the MaxCompute table based on the value of the _execute_time_ field. For more information, see Fields used for real-time synchronization. If you select Dynamic Partitioning by Field Value, data is dynamically written to the destination partitions of the MaxCompute table based on the value of a specified field in the source table after the mapping between the specified field in the source table and the specified partition field in the MaxCompute table is defined.
    Partition messageThe information about the partitioned MaxCompute table.
    Field MappingThe field mappings between the source and destination. Click Field Mapping to configure field mappings. The real-time sync node synchronizes data based on the field mappings.
    If you want to create a table, click Create Table next to Table. In the New data table dialog box, set the parameters. The following table describes the parameters. Create Table
    ParameterDescription
    Table nameThe name of the MaxCompute table to which you want to write data in real time.
    Life cycleThe lifecycle of the MaxCompute table. For more information, see Lifecycle.
    Data field structureThe fields of the MaxCompute table. To add a field, click New field.
    Partition settingsThe partition information of the MaxCompute table. You can select Partitioning by Time or Dynamic Partitioning by Field Value as the partitioning mode.
    • Partitioning by Time: Data is written to the destination partitions of the MaxCompute table based on the value of the _execute_time_ field. For more information, see Fields used for real-time synchronization.Partitioning by Time
      Important
      • You must configure at least two levels of partitions, which are yearly and monthly partitions. You can configure a maximum of five levels of partitions, which are yearly, monthly, daily, hourly, and minutely partitions.
      • For more information about MaxCompute tables, see Partition.
    • Dynamic Partitioning by Field Value: Data is dynamically written to the destination partitions of the MaxCompute table based on the value of a specified field in the source table after the mapping between the specified field in the source table and the specified partition field in the MaxCompute table is defined. Dynamic Partitioning by Field ValueFor example, the value of Field A in the source table is defined as the value of the partition field in the MaxCompute table. If the value of Field A in a record is aa, this record is written to the aa partition of the MaxCompute table. If the value of Field A in a record is bb, this record is written to the bb partition of the MaxCompute table.
  7. Click the Save icon in the toolbar.