DataWorks:Graph Database

Configurer une tâche de synchronisation par lots pour synchroniser les données d'une table unique

• Pour plus d'informations sur la procédure de configuration, consultez les sections Configurer une tâche de synchronisation par lots à l'aide de l'interface sans code et Configurer une tâche de synchronisation par lots à l'aide de l'éditeur de code.

• Pour obtenir des informations sur tous les paramètres configurés et le code exécuté lors de l'utilisation de l'éditeur de code pour configurer une tâche de synchronisation par lots, consultez la section Annexe : Code et paramètres.

Configurer une tâche de synchronisation par lots à l'aide de l'éditeur de code

Pour configurer une tâche de synchronisation par lots à l'aide de l'éditeur de code, définissez les paramètres associés dans le script conformément aux exigences de format de script unifié. Pour plus d'informations, consultez la section Configuration en mode script. Les informations suivantes décrivent les paramètres à configurer pour les sources de données lors de la configuration d'une tâche de synchronisation par lots à l'aide de l'éditeur de code.

Exemple de script Reader

Lorsque vous configurez une tâche de synchronisation des données pour écrire des données dans Graph Database (GDB), configurez les sommets et les arêtes séparément :

• Configurer une tâche de synchronisation pour lire les données des sommets depuis une instance GDB

  {
      "order":{
          "hops":[
              {
                  "from":"Reader",
                  "to":"Writer"
              }
          ]
      },
      "setting":{
          "errorLimit":{
              "record":"100"  // Maximum number of dirty data records to tolerate.
          },
          "jvmOption":"",
          "speed":{
              "concurrent":3,
              "throttle":true, // If true, throttling is enabled. If false, throttling is disabled and the mbps parameter is ignored.
              "mbps":"12"// The throttling limit. 1 mbps is equal to 1 MB/s.
          }
      },
      "steps":[
          {
              "category":"reader",
              "name":"Reader",
              "parameter":{
                  "host": "gdb-xxxxxx.aliyuncs.com", // The endpoint of the GDB instance.
                  "port": 8182, // The port of the GDB instance.
                  "username": "gdb", // The username for accessing the GDB instance.
                  "password": "gdb", // The password for the username.
                  "labelType": "VERTEX", // The type of the label. VERTEX specifies a vertex.
                  "labels": ["label1", "label2"],  // A list of labels. If left empty, all vertices are exported.
                  "column": [
                      {
                          "name": "id",               // The field name.
                          "type": "string",           // The field type.
                          "columnType": "primaryKey"  // Field category. Specifies the vertex primary key, which must be of the STRING type in GDB.
                      },
                      {
                          "name": "label",              // The field name.
                          "type": "string",             // The field type.
                          "columnType": "primaryLabel"  // Field category. Specifies the vertex label, which must be of the STRING type in GDB.
                      },
                      {
                          "name": "age",                   // The property name.
                          "type": "int",                   // The property type.
                          "columnType": "vertexProperty"   // Field category. Specifies a basic property of the vertex in GDB.
                      }
                  ]
              },
              "stepType":"gdb"
          },
          {
              "category":"writer",
              "name":"Writer",
              "parameter":{
                  "print": true
              },
              "stepType":"stream"
          }
      ]
  }

• Configurer une tâche de synchronisation pour lire les données des arêtes depuis une instance GDB

  {
      "order":{
          "hops":[
              {
                  "from":"Reader",
                  "to":"Writer"
              }
          ]
      },
      "setting":{
          "errorLimit":{
              "record":"100"  // Maximum number of dirty data records to tolerate.
          },
          "jvmOption":"",
          "speed":{
              "concurrent":3,
              "throttle":true,// If true, throttling is enabled. If false, throttling is disabled and the mbps parameter is ignored.
              "mbps":"12"// The throttling limit. 1 mbps is equal to 1 MB/s.
          }
      },
      "steps":[
          {
              "category":"reader",
              "name":"Reader",
              "parameter":{
                  "host": "gdb-xxxxxx.aliyuncs.com", // The endpoint of the GDB instance.
                  "port": 8182, // The port of the GDB instance.
                  "username": "gdb", // The username for accessing the GDB instance.
                  "password": "gdb", // The password for the username.
                  "labelType": "EDGE", // The type of the label. EDGE specifies an edge.
                  "labels": ["label1", "label2"],  // A list of labels. If left empty, all edges are exported.
                  "column": [
                      {
                          "name": "id",               // The field name.
                          "type": "string",           // The field type.
                          "columnType": "primaryKey"  // Field category. Specifies the edge primary key, which must be of the STRING type in GDB.
                      },
                      {
                          "name": "label",              // The field name.
                          "type": "string",             // The field type.
                          "columnType": "primaryLabel"  // Field category. Specifies the edge label, which must be of the STRING type in GDB.
                      },
                      {
                          "name": "srcId",               // The field name.
                          "type": "string",              // The field type.
                          "columnType": "srcPrimaryKey"  // Field category. Specifies the start vertex primary key, which must be of the STRING type in GDB.
                      },
                      {
                          "name": "srcLabel",               // The field name.
                          "type": "string",                 // The field type.
                          "columnType": "srcPrimaryLabel"   // Field category. Specifies the start vertex label, which must be of the STRING type in GDB.
                      },
                      {
                          "name": "dstId",                    // The field name.
                          "type": "string",                   // The field type.
                          "columnType": "dstPrimaryKey"       // Field category. Specifies the end vertex primary key, which must be of the STRING type in GDB.
                      },
                      {
                          "name": "dstLabel",                 // The field name.
                          "type": "string",                   // The field type.
                          "columnType": "dstPrimaryLabel"     // Field category. Specifies the end vertex label, which must be of the STRING type in GDB.
                      },
                      {
                          "name": "weight",               // The property name.
                          "type": "double",               // The property type.
                          "columnType": "edgeProperty"    // Field category. Specifies an edge property.
                      }
                  ]
              },
              "stepType":"gdb"
          },
          {
              "category":"writer",
              "name":"Writer",
              "parameter":{
                  "print": true
              },
              "stepType":"stream"
          }
      ]
  }

Paramètres du script Reader

Exemple de script Writer

• Configurer une tâche de synchronisation pour écrire des données de sommets dans une base de données GDB

  {
      "order":{
          "hops":[
              {
                  "from":"Reader",
                  "to":"Writer"
              }
          ]
      },
      "setting":{
          "errorLimit":{
              "record":"100"  // Maximum number of dirty data records to tolerate.
          },
          "speed":{
               "throttle":true,// If true, throttling is enabled. If false, throttling is disabled and the mbps parameter is ignored.
              "concurrent":3, // The job concurrency.
              "mbps":"12"// The throttling limit. 1 mbps is equal to 1 MB/s.
          }
      },
      "steps":[
          {
              "category":"reader",
              "name":"Reader",
              "parameter":{
                  "column":[
                      "*"
                  ],
                  "datasource":"_ODPS",
                  "emptyAsNull":true,
                  "guid":"",
                  "isCompress":false,
                  "partition":[],
                  "table":""
              },
              "stepType":"odps"
          },
          {
              "category":"writer",
              "name":"Writer",
              "parameter": {
                  "datasource": "testGDB", // The name of the data source.
                  "label": "person", // The label, which is the name of the vertex.
                  "srcLabel": "", // Not applicable to vertices.
                  "dstLabel": "", // Not applicable to vertices.
                  "labelType": "VERTEX", // The type of the label. "VERTEX" specifies a vertex.
                  "writeMode": "INSERT", // The policy for handling duplicate primary keys.
                  "idTransRule": "labelPrefix", // The conversion rule for the vertex primary key.
                  "srcIdTransRule": "none", // Not applicable to vertices.
                  "dstIdTransRule": "none", // Not applicable to vertices.
                  "column": [
                      {
                          "name": "id", // The field name.
                          "value": "#{0}", // Takes the value from the source column at index 0. Concatenation is supported.
                          "type": "string", // The field type.
                          "columnType": "primaryKey" // The field category. `primaryKey` specifies the primary key.
                      }, // The primary key of the vertex. The field name must be `id`, the type must be STRING, and this record is required.
                      {
                          "name": "person_age",
                          "value": "#{1}", // Takes the value from the source column at index 1. Concatenation is supported.
                          "type": "int",
                          "columnType": "vertexProperty" // The field category. `vertexProperty` specifies a vertex property.
                      }, // A property of the vertex. Supported types: INT, LONG, FLOAT, DOUBLE, BOOLEAN, and STRING.
                      {
                          "name": "person_credit",
                          "value": "#{2}", // Takes the value from the source column at index 2. Concatenation is supported.
                          "type": "string",
                          "columnType": "vertexProperty"
                      } // A property of the vertex.
                  ]
              }
              "stepType":"gdb"
          }
      ],
      "type":"job",
      "version":"2.0"
  }

• Configurer une tâche de synchronisation pour écrire des données d'arêtes dans une base de données GDB

  {
      "order":{
          "hops":[
              {
                  "from":"Reader",
                  "to":"Writer"
              }
          ]
      },
      "setting":{
          "errorLimit":{
              "record":"100" // Maximum number of dirty data records to tolerate.
          },
          "jvmOption":"",
          "speed":{
              "throttle":true,// If true, throttling is enabled. If false, throttling is disabled and the mbps parameter is ignored.
              "concurrent":3, // The job concurrency.
              "mbps":"12"// The throttling limit. 1 mbps is equal to 1 MB/s.
          }
      },
      "steps":[
          {
              "category":"reader",
              "name":"Reader",
              "parameter":{
                  "column":[
                      "*"
                  ],
                  "datasource":"_ODPS",
                  "emptyAsNull":true,
                  "guid":"",
                  "isCompress":false,
                  "partition":[],
                  "table":""
              },
              "stepType":"odps"
          },
          {
              "category":"writer",
              "name":"Writer",
              "parameter": {
                  "datasource": "testGDB", // The name of the data source.
                  "label": "use", // The label, which is the name of the edge.
                  "labelType": "EDGE", // The type of the label. `EDGE` specifies an edge.
                  "srcLabel": "person", // The label of the start vertex.
                  "dstLabel": "software", // The label of the end vertex.
                  "writeMode": "INSERT", // The policy for handling duplicate primary keys.
                  "idTransRule": "labelPrefix", // The conversion rule for the edge primary key.
                  "srcIdTransRule": "labelPrefix", // The conversion rule for the primary key of the start vertex.
                  "dstIdTransRule": "labelPrefix", // The conversion rule for the primary key of the end vertex.
                  "column": [
                      {
                          "name": "id", // The field name.
                          "value": "#{0}", // Takes the value from the source column at index 0. Concatenation is supported.
                          "type": "string", // The field type.
                          "columnType": "primaryKey" // The field category. `primaryKey` specifies the primary key.
                      }, // The primary key of the edge. The field name must be `id` and the type must be STRING. This record is optional.
                      {
                          "name": "id",
                          "value": "#{1}", // Concatenation is supported. Ensure the mapping rule is consistent with the one used when importing vertices.
                          "type": "string",
                          "columnType": "srcPrimaryKey" // The field category. `srcPrimaryKey` specifies the primary key of the start vertex.
                      }, // The primary key of the start vertex. The field name must be `id`, the type must be STRING, and this record is required.
                      {
                          "name": "id",
                          "value": "#{2}", // Concatenation is supported. Ensure the mapping rule is consistent with the one used when importing vertices.
                          "type": "string",
                          "columnType": "dstPrimaryKey" // The field category. `dstPrimaryKey` specifies the primary key of the end vertex.
                      }, // The primary key of the end vertex. The field name must be `id`, the type must be STRING, and this record is required.
                      {
                          "name": "person_use_software_time",
                          "value": "#{3}", // Concatenation is supported.
                          "type": "long",
                          "columnType": "edgeProperty" // The field category. `edgeProperty` specifies an edge property.
                      }, // A property of the edge. Supported types: INT, LONG, FLOAT, DOUBLE, BOOLEAN, and STRING.
                      {
                          "name": "person_regist_software_name",
                          "value": "#{4}", // Concatenation is supported.
                          "type": "string",
                          "columnType": "edgeProperty"
                      }, // A property of the edge.
                      {
                          "name": "id",
                          "value": "#{5}", // Concatenation is supported.
                          "type": "long",
                          "columnType": "edgeProperty"
                      } // A property of the edge with the field name `id`. This is a regular property, not a primary key, and is optional.
                  ]
              }
              "stepType":"gdb"
          }
      ],
      "type":"job",
      "version":"2.0"
  }

Paramètres du script Writer

{"properties":[
    {"k":"name","t":"string","v":"tom"},
    {"k":"age","t":"int","v":"20"},
    {"k":"sex","t":"string","v":"male"}
]}