Platform For AI:Multimodal data management and usage

Enable PAI and configure workspace

1. Use your root account to enable PAI and create a workspace. Go to the PAI console. In the upper-left corner, select your region. Then click one-click authorization to enable the service.

2. Grant permissions to your operation account. Skip this step if using your root account. For RAM users, assign the workspace administrator role. For details, see Manage Workspaces > Member Role Configuration.

Activate Model Studio and create API key

Enable Alibaba Cloud Model Studio and create an API key. For instructions, see Get an API Key.

Create vector database

Create vector database instance

Multimodal dataset management supports the following Alibaba Cloud vector databases:

• Elasticsearch (Vector Enhanced Edition, version 8.17.0 or later)

• OpenSearch (Vector Search Edition)

• Milvus (version 2.4 or later)

• Hologres (version 4.0.9 or later)

• Lindorm (Vector Engine Edition)

For instructions on creating vector database instances, see corresponding product documentation.

Network and whitelist configuration

• Public network access

  If your vector database instance has a public endpoint, add the IP addresses below to the public access whitelist. Multimodal data management can then access the instance over the Internet. For Elasticsearch, see Configure Public or Private Network Access Whitelist.

  Region	IP list
  Hangzhou	47.110.230.142, 47.98.189.92
  Shanghai	47.117.86.159, 106.14.192.90
  Shenzhen	47.106.88.217, 39.108.12.110
  Ulanqab	8.130.24.177, 8.130.82.15
  Beijing	39.107.234.20, 182.92.58.94

• Private network access

  Submit a ticket to request access.

Create vector index table (optional)

The system can automatically create an index table. Skip this step if custom index table is not needed.

In some vector databases, an index table is called a collection or index.

Index table structure definition (Follow this structure):

{
    "id":"text",                    // Primary key ID. Required in OpenSearch. Default in other databases.
    "index_set_id": "keyword",      // Index set ID. Must be indexed.
    "file_meta_id": "text",         // File metadata ID.
    "dataset_id": "text",           // Dataset ID.
    "dataset_version": "text",      // Dataset version.
    "uri": "text",                  // OSS URI of the file.
    "file_vector": {                // Vector field.
        "type": "float",            // Vector type: float.
        "dims": 1536,               // Vector dimensions. Customize as needed.
        "similarity": "DotProduct"  // Similarity algorithm: cosine or dot product.
    }
}

This topic uses Elasticsearch as an example. The following Python code shows how to create a semantic index table. For other vector databases, see their documentation.

from elasticsearch import Elasticsearch

# 1. Connect to your Alibaba Cloud Elasticsearch instance.
# Note:
# (1) Use Python 3.9 or later: python3 -V
# (2) Install Elasticsearch client version 8.x: pip show elasticsearch
# (3) If using a VPC endpoint, ensure your client and Elasticsearch instance are in the same VPC.
#     Otherwise, use the public endpoint and add your client's public IP to Elasticsearch whitelist.
# Default username is elastic.
es_client = Elasticsearch(
    hosts=["http://es-cn-l4p***5z.elasticsearch.aliyuncs.com:9200"],
    basic_auth=("{userName}", "{password}"),
)

# 2. Define the index name and mapping. HNSW is used by default.
index_name = "dataset_embed_test"
index_mapping = {
    "settings": {
        "number_of_shards": 1,          # Number of shards.
        "number_of_replicas": 1         # Number of replicas.
    },
    "mappings": {
        "properties": {
            "index_set_id": {
                "type": "keyword"
            },
            "uri": {
                "type": "text"
            },
            "file_meta_id": {
                "type": "text"
            },
            "dataset_id": {
                "type": "text"
            },
            "dataset_version": {
                "type": "text"  
            },
            "file_vector": {
                "type": "dense_vector",  # Define file_vector as dense vector.
                "dims": 1536,  # Vector dimensions: 1536.
                "similarity": "dot_product"  # Similarity method: dot product.
            }
        }
    }
}

# 3. Create the index.
if not es_client.indices.exists(index=index_name):
    es_client.indices.create(index=index_name, body=index_mapping)
    print(f"Index {index_name} created successfully!")
else:
    print(f"Index {index_name} already exists. No action taken.")

# 4. View the index mapping (optional).
# indexes = es_client.indices.get(index=index_name)
# print(indexes)

Create connection for intelligent labeling

1. In your PAI workspace, click AI Asset Management > Connection > Model Service > Create Connection to open the connection creation page.

   

2. Select Alibaba Cloud Model Studio Service and configure the API key.

   

3. After successful creation, find your new connection in the list.

   

Create connection for semantic indexing

1. Skip this step if using Alibaba Cloud Model Studio’s semantic indexing service. In the left menu, click Model Gallery. Find and deploy the GME multimodal retrieval model. This creates an EAS service. Deployment takes about five minutes. When the status shows Running, deployment is complete.

   Important

   Stop and delete the service when no longer needed to avoid charges.

   

2. In your PAI workspace, click AI Asset Management > Connection > Model Service > Create Connection to open the connection creation page.

3. Configure the model connection based on whether you use Alibaba Cloud Model Studio’s semantic indexing model or your own EAS-deployed model.

   Use Alibaba Cloud Model Studio’s semantic indexing model

   • Connection Type: Select General Multimodal Embedding Model Service.

   • Service Provider: Select Third-party service model.

   • Model Name: tongyi-embedding-vision-plus.

   • base_url: https://dashscope.aliyuncs.com/api/v1/services/embeddings/multimodal-embedding/multimodal-embedding

   • api_key: Get your API key from Get an API Key and enter it here.

   

   Use your own EAS-deployed semantic indexing model

   • Connection Type: Select General Multimodal Embedding Model Service.

   • Service Provider: Select PAI-EAS Model Service.

   • EAS Service: Select the GME multimodal retrieval model you just deployed. If the service is not under your current account, choose a third-party model service.

   

   

4. After successful creation, find your new connection in the list.

   

Create vector database connection

1. In the left menu, click AI Asset Management > Connection > Database > Create Connection to open the connection creation page.

   

2. Multimodal search supports Milvus, Lindorm, OpenSearch, Elasticsearch, and Hologres. This example uses Elasticsearch. Select Elasticsearch and configure uri, username, and password. For details, see Create a Database Connection.

   

   Connection format examples for each vector database:

   Milvus

   uri: http://xxx.milvus.aliyuncs.com:19530 
   database: {your_data_base} 
   token: root:{password}

   OpenSearch

   uri: http://xxxx.ha.aliyuncs.com
   username: {username} 
   password: {password}

   Hologres

   host: xxxx.hologres.aliyuncs.com
   database: {your_data_base} 
   port: {port}
   access_key_id={password}

   Elasticsearch

   uri: http://xxxx.elasticsearch.aliyuncs.com:9200
   username: {username} 
   password: {password}

   Lindorm

   uri: xxxx.lindorm.aliyuncs.com:{port}
   username: {username} 
   password: root:{password}

3. After successful creation, find your new connection in the list.

   

Create intelligent label definition

In the left menu, click AI Asset Management > Datasets > Intelligent Tag Definition > Create Intelligent Tag Definition to open the label configuration page. Example configuration:

• Guide Prompt: As an experienced driver with many years on highways and city roads, you know how to handle common driving scenarios.

• Tag Definition:

  Autonomous driving label example

  {
      "Reflective tape": "Usually yellow or black-and-yellow striped. Attached to permanent obstacles like corners to warn drivers. Strip-shaped—not traffic cones, wheel locks, or water barrels!",
      "Wheel lock": "Also called a parking lock. Prevents unauthorized parking when raised. Always specify if raised or lowered. Raised if there is a frame.",
      "Lit construction vehicle": "Has two arrow-shaped lights, lit. Not present otherwise.",
      "Overturned vehicle": "Vehicle lying on its side.",
      "Fallen water barrel": "A plastic barrier used to divide roads or block traffic. Usually red and wall-shaped. Common on highways, city roads, and overpasses. Larger than cones and flat. Specify if fallen.",
      "Fallen traffic cone": "Also called a traffic cone or snow cone. Cone-shaped temporary road marker. Not rod- or flat-shaped. To check if fallen, see if the base touches the ground.",
      "Charging parking spot": "Near a wall with visible charging equipment or labeled 'new energy vehicle'. Found only in parking lots (indoor or outdoor). Wheel locks are unrelated.",
      "Speed bump": "Usually yellow or black-and-yellow. Narrow ridge across the road to slow vehicles. Never in parking spots.",
      "Deceleration lane markings": "Fishbone-style dashed lines on both sides of the lane, inside solid lines.",
      "Ramp": "Large curved highway segment. Usually on the right side of main highways. Only confirm at toll plazas.",
      "Ground shadow": "Clear shadows on the ground.",
      "Cloudy": "Only if sky is visible and clearly cloudy.",
      "Glare from headlights": "Headlights appear as streaks instead of points—common at night or in rain.",
      "Left-turn, right-turn, U-turn arrows": "White (or sometimes yellow) arrows painted on lanes—not green-white highway signs. Only count clear center-lane arrows. Right-turn: clockwise from base to tip. Left-turn: counterclockwise. U-turn: U-shaped.",
      "Crosswalk": "White parallel stripes on roads or parking lots, for pedestrians. Never on highways, ramps, or tunnels.",
      "Overexposure": "Camera overexposed due to direct sunlight—daytime only.",
      "Motor vehicle": "Any other motorized vehicle in view.",
      "Lane merge/diverge": "Where multiple lanes become one—or one splits into many.",
      "Intersection": "Road intersection without lane markings inside it.",
      "No-parking sign": "Sign hanging or standing on ground with 'no parking' text or circle-with-P-and-slash symbol.",
      "Lane markings": "Road lane lines—especially blurry ones.",
      "Stones or tires on road": "Obstacles blocking traffic.",
      "Tunnel": "Watch for tunnel entrances and exits.",
      "Wet road in rain": "Road surface wet and slippery in rain.",
      "Non-motorized vehicle": "Bicycles, e-bikes, wheelchairs, unicycles, shopping carts—parked or moving."
    }

Create offline intelligent labeling job

1. Click Custom Dataset. Click the dataset name to open its details page. Then click Dataset jobs.

   

2. On the jobs page, click Create job > Smart tag and configure job parameters.

   

   • Dataset Version: Select the version to label, such as v1.

   • Labeling Model Connection: Select your Alibaba Cloud Model Studio model connection.

   • Smart Labeling Model: Supports Qwen-VL-Max and Qwen-VL-Plus.

   • Max Concurrency: Set based on your EAS model service specs. Suggested maximum per GPU: 5.

   • Intelligent Tag Definition: Select the definition you just created.

   • Labeling Mode: Choose Increment or Full.

3. After successful creation, find your labeling job in the list. To monitor or stop it, click the link on the right side of the list.

   Note

   The first run builds metadata. Wait patiently—it may take time.

Export as file

1. Click Export as file. On the config page, set export content and target OSS directory. Click OK.

   

2. To track progress, click AI Asset Management > Job > Dataset jobs.

3. Use the exported result. After export, mount the result file and original dataset to your training environment (such as DLC or DSW instances). Then use code to read the index file and load target files for model training or analysis.

Export to logical dataset version

Export a search result from an advanced dataset to a version of a logical dataset. Later, use the dataset SDK to access that version.

1. Click Export to logical dataset version. Select a target logical dataset and click Confirm.

   

   If a Boolean dataset is not available, see the following:

   Create a logical dataset

   Create a logical dataset. In the left menu, click AI Asset Management > Dataset > Create Dataset. Configure key parameters below. Adjust others as needed:

   • Dataset Type: Select Logical.

   • Metadata OSS path: Select an exported OSS path.

   • Import method: Select Import later.

   Click OK to create the dataset.

2. Use the logical dataset. After the import job finishes, the target logical dataset contains the exported metadata. Load and use it with the SDK. See the dataset details page for SDK usage instructions.

   

   

   Install the SDK with:

   pip install https://pai-sdk.oss-cn-shanghai.aliyuncs.com/dataset/pai_dataset_sdk-1.0.0-py3-none-any.whl