DataWorks:Hive

Configuration DLF

Si votre source de données Hive provient d'EMR et utilise DLF (Data Lake Formation) pour la gestion des métadonnées, ajoutez les éléments suivants au champ Extension Parameters :

{"dlf.catalog.id" : "my_catalog_xxxx"}

Remplacez my_catalog_xxxx par la valeur de dlf.catalog.id issue de votre configuration EMR Hive.

Configuration haute disponibilité (HA)

Si le cluster EMR Hive a la haute disponibilité (HA) activée, activez le High-availability Mode et configurez les paramètres HA dans Extension Parameters . Obtenez les valeurs de configuration depuis la console EMR en accédant au cluster cible et en cliquant sur Cluster Services dans la colonne Actions .

{
  "dfs.nameservices": "testDfs",
  "dfs.ha.namenodes.testDfs": "namenode1,namenode2",
  "dfs.namenode.rpc-address.testDfs.namenode1": "",
  "dfs.namenode.rpc-address.testDfs.namenode2": "",
  "dfs.client.failover.proxy.provider.testDfs": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
  "fs.oss.accessKeyId": "<yourAccessKeyId>",
  "fs.oss.accessKeySecret": "<yourAccessKeySecret>",
  "fs.oss.endpoint": "oss-cn-<yourRegion>-internal.aliyuncs.com"
}

Les trois derniers champs OSS ( fs.oss.* ) sont requis uniquement si le stockage sous-jacent est OSS.

Configuration de table externe OSS

Si le stockage sous-jacent est OSS :

• Définissez defaultFS avec le préfixe oss:// , par exemple oss://bucketName .

• Pour les tables externes OSS, ajoutez les éléments suivants à Extension Parameters :

  {
    "fs.oss.accessKeyId": "<yourAccessKeyId>",
    "fs.oss.accessKeySecret": "<yourAccessKeySecret>",
    "fs.oss.endpoint": "oss-cn-<yourRegion>-internal.aliyuncs.com"
  }

• Pour les tables externes OSS-HDFS, utilisez plutôt le format de point de terminaison OSS-HDFS :

  {
    "fs.oss.accessKeyId": "<yourAccessKeyId>",
    "fs.oss.accessKeySecret": "<yourAccessKeySecret>",
    "fs.oss.endpoint": "cn-<yourRegion>.oss-dls.aliyuncs.com"
  }

{
  "type": "job",
  "steps": [
    {
      "stepType": "hive",
      "parameter": {
        "datasource": "hive_not_ha_****",
        "table": "part_table_1",
        "readMode": "hdfs",
        "partition": "pt1=a,pt2=b,pt3=c",
        "column": ["id", "pt2", "pt1"],
        "fileSystemUsername": "hdfs",
        "hivePartitionColumn": [
          {"type": "string", "value": "partition name 1"},
          {"type": "string", "value": "partition name 2"}
        ],
        "successOnNoFile": true
      },
      "name": "Reader",
      "category": "reader"
    },
    {
      "stepType": "hive",
      "parameter": {},
      "name": "Writer",
      "category": "writer"
    }
  ],
  "version": "2.0",
  "order": {"hops": [{"from": "Reader", "to": "Writer"}]},
  "setting": {
    "errorLimit": {"record": ""},
    "speed": {
      "concurrent": 2,
      "throttle": true,
      "mbps": "12"
    }
  }
}

{
  "type": "job",
  "steps": [
    {
      "stepType": "hive",
      "parameter": {
        "datasource": "hive_not_ha_****",
        "table": "part_table_1",
        "readMode": "jdbc",
        "querySql": "select id,name,age from part_table_1 where pt2='B'",
        "column": ["id", "name", "age"],
        "where": "",
        "session": ["mapred.task.timeout=600000"]
      },
      "name": "Reader",
      "category": "reader"
    },
    {
      "stepType": "hive",
      "parameter": {},
      "name": "Writer",
      "category": "writer"
    }
  ],
  "version": "2.0",
  "order": {"hops": [{"from": "Reader", "to": "Writer"}]},
  "setting": {
    "errorLimit": {"record": ""},
    "speed": {
      "concurrent": 2,
      "throttle": true,
      "mbps": "12"
    }
  }
}

{
  "type": "job",
  "steps": [
    {
      "stepType": "hive",
      "parameter": {},
      "name": "Reader",
      "category": "reader"
    },
    {
      "stepType": "hive",
      "parameter": {
        "datasource": "hive_ha_shanghai",
        "table": "partitiontable2",
        "partition": "year=a,month=b,day=c",
        "column": ["id", "name", "age"],
        "writeMode": "append",
        "fileSystemUsername": "hdfs"
      },
      "name": "Writer",
      "category": "writer"
    }
  ],
  "version": "2.0",
  "order": {"hops": [{"from": "Reader", "to": "Writer"}]},
  "setting": {
    "errorLimit": {"record": ""},
    "speed": {
      "concurrent": 2,
      "throttle": true,
      "mbps": "12"
    }
  }
}