MaxCompute:Migrate data from MaxCompute to OSS

Procedure

1. Activate services

   1. Activate MaxCompute

   2. Activate DataWorks

2. Create a table in DataWorks

   1. Log on to the DataWorks console and select a region in the upper-left corner.

   2. In the left navigation pane, click Workspace.

   3. On the Workspaces page, find the target workspace and in the Actions column, choose Shortcuts > DataStudio.

   4. On the DataStudio page, create a MaxCompute SQL node. Use the following statement to create a table named transs.

      CREATE TABLE transs
      (name    STRING,
      id    STRING,
      gender    STRING);

3. Import data into the transs table

   1. Save the following sample data to your computer as a CSV file.

      qwe,145,F
      asd,256,F
      xzc,345,M
      rgth,234,F
      ert,456,F
      dfg,12,M
      tyj,4,M
      bfg,245,M
      nrtjeryj,15,F
      rwh,2344,M
      trh,387,F
      srjeyj,67,M
      saerh,567,M

   2. In the left navigation pane, choose Data Integration > Data Upload and Download.

   3. Click Go to Data Upload and Download.

   4. In the left navigation pane, click the upload icon and then click Upload Data.

4. Create a file in OSS

   1. Log on to the Object Storage Service (OSS) console.

   2. In the navigation pane on the left, click Buckets.

   3. On the Buckets page, click Create Bucket.

   4. Create a file named qwee.csv on your computer and upload it to OSS. For more information, see Upload objects. Make sure that the field names in the qwee.csv file match the field names in the transs table.

5. Add a MaxCompute data source in DataWorks

   1. Log on to the DataWorks console and select a region in the upper-left corner.

   2. In the left navigation pane, click Workspace.

   3. On the Workspaces page, click the name of the target workspace.

   4. On the Workspace Details page, click Data Sources in the left navigation pane.

   5. On the Data Source tab, click Add Data Source, and select MaxCompute.

   6. On the Add MaxCompute Data Source page, configure the Basic Information.

      For more information, see Configure a MaxCompute data source.

6. Add an OSS data source in DataWorks

   For more information, see Configure an OSS data source.

7. Configure a MaxCompute (ODPS) reader and an OSS writer

   1. Log on to the DataWorks console and select a region in the upper-left corner.

   2. In the left navigation pane, choose Data Development and O&M > Data Development.

   3. In the Select Workspace section, click Go To Data Studio.

   4. In the left pane of Data Studio, click the icon and select Create Node > Data Integration > Batch Synchronization.

      • For Data source, select OSS.

      • For Data destination, select MaxCompute (ODPS).

   5. On the node configuration page, configure the parameters.

      • Data Source > Source: Select the newly added OSS data source.

      • Data Source-Destination: Select the destination MaxCompute data source.

      • For Data Source - Text Type: Select CSV.

      • Data Source > File Path: The path of the input file.

      • Destination-Table: The destination MaxCompute table.

      • Runtime Resource: Select the dedicated resource group.

      • Keep the default values for other parameters.

      You can also click the icon above the configuration section to switch to the code editor and configure the node using a script. The following code provides an example:

      {
          "order":{
              "hops":[
                  {
                      "from":"Reader",
                      "to":"Writer"
                  }
              ]
          },
          "setting":{
              "errorLimit":{
                  "record":"0"
              },
              "speed":{
                  "concurrent":1,
                  "dmu":1,
                  "throttle":false
              }
          },
          "steps":[
              {
                  "category":"reader",
                  "name":"Reader",
                  "parameter":{
                      "column":[
                          "name",
                          "id",
                          "gender"
                      ],
                      "datasource":"odps_first",
                      "partition":[],
                      "table":"Transs"
                  },
                  "stepType":"odps"
              },
              {
                  "category":"writer",
                  "name":"Writer",
                  "parameter":{
                      "datasource":"Trans",
                      "dateFormat":"yyyy-MM-dd HH:mm:ss",
                      "encoding":"UTF-8",
                      "fieldDelimiter":",",
                      "fileFormat":"csv",
                      "nullFormat":"null",
                      "object":"qwee.csv",
                      "writeMode":"truncate"
                  },
                  "stepType":"oss"
              }
          ],
          "type":"job",
          "version":"2.0"
      }                           

   6. Click Run.

8. View the synchronized data in the OSS console. For more information, see Download objects.