DataWorks:Embedding

Step 1: Create an offline sync node

1. Go to the Workspaces page in the DataWorks console. In the top navigation bar, select the target region. Find the workspace and choose Shortcuts > Data Studio in the Actions column.

2. In the project folder, click  > Create Node > Data Integration > Batch Synchronization. Set the Data Source and Destination (source: MaxCompute, destination: Milvus) and the node Name, then click Confirm.

Step 2: Configure the offline sync task

1. Configure basic information.

   • Data Source: Select the data sources for the source and destination.

   • Resource Group: Select the Serverless resource group attached to the workspace.

   If no data sources or resource groups are available, complete the prerequisites first.

2. Configure the Data Source. The following table describes the key parameters for the MaxCompute source. If you use a different source, parameters may vary. Click Data Preview to verify the configuration.

   Parameter	Description
   Tunnel Resource Group	The default value for Tunnel Quota is Public Transmission Resource, which is the free quota for MaxCompute. For details on selecting a transmission resource, see Purchase and use exclusive Data Transmission Service resource groups. Important If an exclusive Tunnel Quota becomes unavailable due to an overdue payment or expiration, a running job automatically switches to Public Transmission Resource.
   Table	The source table to synchronize. If no source tables are available, prepare the test data first.
   Filtering Method	Supports Partition Filter and Data Filtering: if the source table is partitioned, filter by partition; if it is not partitioned, use a WHERE clause to select rows.

3. Configure Data Processing.

   1. Enable the data processing switch. In the Data Processing List, click Add Node > Data Embedding to add an embedding processing node.

   2. Configure the data vectorization node.

      Note

      • Throughput depends on the embedding model. Alibaba Cloud Model Studio (QWen models) has a queries per second (QPS) limit. For the Alibaba Cloud PAI model marketplace, performance depends on the resource specifications of the deployed EAS service.

      • For a given set of parameters, embedding models produce deterministic vectors. Data Integration uses a Least Frequently Used (LFU) cache to skip redundant embedding calls for identical input data, improving throughput and reducing cost.

      Parameter	Description
      Model Provider	The embedding model provider. Supported options: Alibaba Cloud DataWorks model service, Alibaba Cloud Model Studio, and Alibaba Cloud PAI model marketplace.
      Model Name	The embedding model to use. Select one based on your provider.
      Model API Key	The API key for the selected model. For Alibaba Cloud Model Studio, see Obtain a Model Studio API key. For the Alibaba Cloud PAI model marketplace, open Online Debugging on the deployed EAS task and copy the value of the Authorization header.
      Model Endpoint	Required when Model Provider is Alibaba Cloud PAI Model Marketplace. Enter the endpoint API address.
      Batch Size	The number of records sent to the embedding model per call. Batch processing improves throughput and reduces cost. Default: 10.
      Select Fields To Vectorize	The source columns to vectorize. Supports a single field or a concatenation of multiple fields. Also defines the output field name.
      Vectorization Output Field	The name of the field that stores the output vector.
      Vector Dimension	The dimension of the output vector. The configured embedding model must support the defined vector dimensions. Default: 1024.
      Convert NULL To Empty String	Embedding models reject NULL inputs. Enable this option to convert NULL values to empty strings and avoid errors. Disabled by default.
      Concatenate Field Name	When enabled, prepends the field name to the field value before vectorization. Requires configuring Field Name Delimiter. Disabled by default.
      Skip Empty Fields	When concatenating multiple fields for vectorization, specifies whether to skip empty fields. By default, this option is selected, and empty fields are skipped.

   3. Preview the output. Click Data Output Preview in the upper-right corner of the vectorization node configuration area, then click Preview to inspect the vectorized results before running the full task. > Tip: Click Simulate Run at the top of the offline sync edit page for an alternative preview.

4. Configure the Destination. The following table describes the key parameters for the Milvus destination.

   Parameter	Description
   Collection	The Milvus collection that receives the vector data.
   Partition Key	Optional. If the collection is partitioned, specify the target partition.
   Write Mode	upsert: if autoid is disabled, updates an entity by primary key; if autoid is enabled, replaces the primary key with an auto-generated one and inserts the record. insert: inserts data into collections with autoid enabled; Milvus generates the primary key automatically. Note Using insert on a collection without autoid enabled causes data duplication.

   

5. Configure Destination Field Mapping. After configuring the source, data processing, and destination, the task generates a field mapping automatically. Because the destination has no fixed schema, mappings are created row by row. Click Edit next to a Source Field or Target Field to adjust the mapping order or remove unnecessary fields. For this example, delete the fields that are not needed. The adjusted mapping looks like this:

   

6. Configure Advanced Configuration. Click Advanced Configuration on the right side of the node configuration page to set task concurrency, sync rate, and dirty data policy.

Step 3: Test run

1. On the right side of the offline sync node edit page, click Run Configuration. Set the Resource Group and Script Parameters, then click Run in the top toolbar to test the pipeline.

2. Go to Milvus and verify that the destination collection contains the expected data.

Step 4: Configure scheduling and publish

Click Scheduling on the right side of the offline sync task. Set the scheduling configuration parameters for periodic runs, then click Publish in the top toolbar. Follow the on-screen instructions to publish the task.

Step 1: Create an offline sync node

Follow the same steps as Step 1 in the codeless UI configuration.

Step 2: Configure the sync script

1. Click in the toolbar to switch to the code editor.

2. Enter the JSON configuration for the sync task. All examples use the embedding-transformer step to define the embedding stage between the Reader and Writer. The script for this example is:

   • For Reader and Writer parameter details, see MaxCompute data source and Milvus data source.

   • For other source and destination types, see Data source list.

   {
       "type": "job",
       "version": "2.0",
       "steps": [
           {
               "stepType": "odps",
               "parameter": {
                   "partition": [
                       "split=dev"
                   ],
                   "datasource": "MaxCompute_Source",
                   "successOnNoPartition": true,
                   "tunnelQuota": "default",
                   "column": [
                       "sentence"
                   ],
                   "enableWhere": false,
                   "table": "test_tb"
               },
               "name": "Reader",
               "category": "reader"
           },
           {
               "category": "flatmap",
               "stepType": "embedding-transformer",
               "parameter": {
                   "modelProvider": "bailian",
                   "modelName": "text-embedding-v4",
                   "embeddingColumns": {
                       "sourceColumnNames": [
                           "sentence"
                       ],
                       "embeddingColumnName": "sentence_e"
                   },
                   "apiKey": "sk-****",
                   "dimension": 128,
                   "nullAsEmptyString": true
               },
               "displayName": "sentence2emb",
               "description": ""
           },
           {
               "stepType": "milvus",
               "parameter": {
                   "schemaCreateMode": "ignore",
                   "enableDynamicSchema": true,
                   "datasource": "Milvus_Source",
                   "column": [
                       {
                           "name": "sentence",
                           "type": "VarChar",
                           "elementType": "None",
                           "maxLength": "32"
                       },
                       {
                           "name": "sentence_e",
                           "type": "FloatVector",
                           "dimension": "128",
                           "elementType": "None",
                           "maxLength": "65535"
                       }
                   ],
                   "writeMode": "insert",
                   "collection": "Milvus_Collection",
                   "batchSize": 1024,
                   "columnMapping": [
                       {
                           "sourceColName": "sentence",
                           "dstColName": "sentence"
                       },
                       {
                           "sourceColName": "sentence_e",
                           "dstColName": "sentence_e"
                       }
                   ]
               },
               "name": "Writer",
               "category": "writer"
           }
       ],
       "setting": {
           "errorLimit": {
               "record": "0"
           },
           "speed": {
               "concurrent": 2,
               "throttle": false
           }
       },
       "order": {
           "hops": [
               {
                   "from": "Reader",
                   "to": "Writer"
               }
           ]
       }
   }

3. The embedding-transformer step parameters are:

   Parameter	Description	Required
   modelProvider	The embedding model provider. Options: dataworksModelService (DataWorks Large Language Model Service), bailian (Alibaba Cloud Model Studio, supports QWen models), paiModelGallery (Alibaba Cloud PAI model marketplace, supports BGE-M3 models).	Yes
   modelName	The embedding model name. When modelProvider is bailian: text-embedding-v4 or text-embedding-v3. When modelProvider is paiModelGallery: bge-m3.	Yes
   apiKey	The API key for the model provider.	Yes
   endpoint	The endpoint API address. Required when modelProvider is paiModelGallery.	No
   batchSize	Number of records per embedding call. Batch processing improves throughput and reduces cost. Default: 10.	No
   embeddingColumns	Defines which source columns to vectorize and the output field name. Supports a single field or multiple concatenated fields. Example: {"sourceColumnNames": ["col1", "col2"], "embeddingColumnName": "my_vector"}.	Yes
   appendDelimiter	Delimiter used to join multiple field values before vectorization. Default: \n.	No
   skipEmptyValue	When concatenating multiple fields, skip fields with empty values. Default: false.	No
   dimension	Output vector dimensions. The chosen model must support the configured value. Default: 1024.	No
   nullAsEmptyString	Convert NULL field values to empty strings before embedding. Embedding models reject NULL inputs. Default: false.	No
   appendFieldNameEnable	Prepend the field name to the field value before vectorization. When enabled, also configure appendFieldNameDelimiter. Default: false.	No
   appendFieldNameDelimiter	Delimiter for concatenating field names. Takes effect only when appendFieldNameEnable is true.	No

4. Click Dry Run at the top of the offline sync node edit page. Then, click Start Sampling and Preview to view the vectorized results and confirm that the configuration is correct.

5. Click Advanced Configuration on the right side of the node configuration page to set task concurrency, sync rate, and dirty data policy.

Step 3: Test run

1. Click Run Configuration on the right side of the edit page. Set the Resource Group and Script Parameters, then click Run to test the pipeline.

2. Go to Milvus and verify that the destination collection contains the expected data.

Step 4: Configure scheduling and publish

Click Scheduling on the right side of the task. Set the scheduling configuration parameters for periodic runs, then click Publish in the top toolbar. Follow the on-screen instructions to publish the task.