Platform For AI:Manage and use multimodal data

3.1.1 Activate PAI, create a default workspace, and obtain administrator permissions

1. Use a root account to activate PAI and create a workspace. Log on to the PAI console, select a region in the upper-left corner, and then authorize and activate the product.

2. Authorize the operating account. You can skip this step if you are using a root account. RAM users must have the Workspace administrator role. For instructions on how to authorize an account, see the "Configure member roles" section in Create and manage workspaces.

3.1.2 Activate Model Studio and create an API key

To activate Alibaba Cloud Model Studio and create an API key, see Get an API key.

3.1.3 Create a vector database

Create a vector database instance

Multimodal dataset management currently supports the following Alibaba Cloud vector databases:

• Elasticsearch (Vector Search Edition, 8.17.0 or later)

• OpenSearch (Vector Search Edition)

• Milvus (2.4 or later)

• Hologres (4.0.9 or later)

• Lindorm (Vector Engine Edition)

For instructions on how to create an instance for each cloud vector database, refer to the documentation for the respective product.

Configure network and whitelist settings

• Public network access

  If your vector database instance has a public endpoint enabled, add the following IP addresses to the instance's public access whitelist. This allows the multimodal data management service to access the instance over the public network. For instructions on how to set up an Elasticsearch whitelist, see Configure a public or private IP address whitelist for an Elasticsearch cluster.

  Region	IP address list
  Hangzhou	47.110.230.142, 47.98.189.92
  Shanghai	47.117.86.159, 106.14.192.90
  Shenzhen	47.106.88.217, 39.108.12.110
  Ulanqab	8.130.24.177, 8.130.82.15
  Beijing	39.107.234.20, 182.92.58.94

• Private network access

  Please submit a ticket to apply for this option.

Create a vector index table (Optional)

The system can create an index table automatically. You can skip this step unless you need a custom one.

In some vector databases, a vector index table is also known as a Collection or an Index.

The index table structure must be defined as follows:

{
    "id":"text",                    // Primary key ID. This must be defined in OpenSearch. It exists by default in other databases and does not need to be defined.
    "index_set_id": "keyword",      // Index set ID. Must support indexing.
    "file_meta_id": "text",         // File metadata ID.   
    "dataset_id": "text",           // Dataset ID.
    "dataset_version": "text",      // Dataset version.
    "uri": "text",                  // URI of the OSS file.
    "file_vector": {                // Vector field.
        "type": "float",            // Vector type: float.
        "dims": 1536,               // Vector dimensions. Custom.
        "similarity": "DotProduct"  // Vector distance algorithm. Cosine distance or dot product.
    }
}

This section uses Elasticsearch as an example to show how to create a semantic index table with Python. For instructions on how to create index tables for other types of vector databases, refer to the documentation for the respective product. The sample code is as follows:

from elasticsearch import Elasticsearch

# 1. Connect to the Alibaba Cloud Elasticsearch instance.
# Note:
# (1) Python 3.9 or later is required: python3 -V
# (2) The Elasticsearch client must be version 8.x: pip show elasticsearch
# (3) If you use a VPC endpoint, the caller must be able to communicate with the VPC of the ES instance. Otherwise, use a public endpoint and add the public IP address of the caller to the ES whitelist.
# The default userName is elastic.
es_client = Elasticsearch(
    hosts=["http://es-cn-l4p***5z.elasticsearch.aliyuncs.com:9200"],
    basic_auth=("{userName}", "{password}"),
)

# 2. Define the index name and structure. The HNSW index algorithm is used by default.
index_name = "dataset_embed_test"
index_mapping = {
    "settings": {
        "number_of_shards": 1,          # Number of shards.
        "number_of_replicas": 1         # Number of replicas.
    },
    "mappings": {
        "properties": {
            "index_set_id": {
                "type": "keyword"
            },
            "uri": {
                "type": "text"
            },
            "file_meta_id": {
                "type": "text"
            },
            "dataset_id": {
                "type": "text"
            },
            "dataset_version": {
                "type": "text"  
            },
            "file_vector": {
                "type": "dense_vector",  # Define file_vector as a dense vector type.
                "dims": 1536,  # The vector dimension is 1536.
                "similarity": "dot_product"  # The similarity calculation method is dot product.
            }
        }
    }
}

# 3. Create the index.
if not es_client.indices.exists(index=index_name):
    es_client.indices.create(index=index_name, body=index_mapping)
    print(f"Index {index_name} created successfully!")
else:
    print(f"Index {index_name} already exists. It will not be created again.")

# 4. View the schema of the created index table (Optional).
# indexes = es_client.indices.get(index=index_name)
# print(indexes)

3.3.1 Create a Smart Tagging model connection

1. Go to your PAI workspace. In the left-side navigation pane, choose AI Asset Management > Connection > Model Service > Create Connection.

   

2. Select Alibaba Cloud Model Studio Service and configure the Model Studio api_key.

   

3. After the connection is created, you can see the Alibaba Cloud Model Studio Service in the list.

   

3.3.2 Create a Semantic Indexing model connection

1. You can skip this step if you plan to use the Model Studio Semantic Indexing model service. In the left-side navigation pane, click Model Gallery, find and deploy the GME multimodal retrieval model to obtain an EAS service. The deployment takes about 5 minutes. When the status is Running, the deployment is successful.

   Important

   When you no longer need the index model, you can stop and delete the service to avoid further charges.

   

2. Go to your PAI workspace. In the left-side navigation pane, choose AI Asset Management > Connection > Model Service > Create Connection.

3. Configure the model connection information based on whether you chose the Model Studio Semantic Indexing model or a custom-deployed EAS Semantic Indexing model.

   Use the Model Studio Semantic Indexing model

   • For Connection Type, select General Multimodal Embedding Model Service.

   • For Service Provider, select Third-party Model Service.

   • Model Name: tongyi-embedding-vision-plus.

   • base_url: https://dashscope.aliyuncs.com/api/v1/services/embeddings/multimodal-embedding/multimodal-embedding

   • api_key: Get an API key and fill it in.

   

   Use a custom-deployed EAS Semantic Indexing model

   • For Connection Type, select General-purpose Multimodal Embedding Model Service.

   • For Service Provider, select PAI-EAS Model Service.

   • EAS Service: Select the GME multimodal retrieval model that you just deployed. If the service provider is not under the current account, you can select Third-party Service Model.

   

   

4. After the connection is created, you can see the model connection service in the list.

   

3.3.3 Create a vector database connection

1. In the left-side navigation pane, choose AI Asset Management > Connection > Database > Create Connection.

   

2. The multimodal retrieval service supports vector databases such as Milvus, Lindorm, Open Search, Elasticsearch, and Hogress. This topic uses Elasticsearch as an example to describe how to create a database connection. Select Elasticsearch and configure parameters such as url, username, and password. For more information, see Create a database connection.

   

   The connection formats for each vector database are as follows:

   Milvus

   uri: http://xxx.milvus.aliyuncs.com:19530 
   database: {your_database} 
   token: root:{password}

   OpenSearch

   uri: http://xxxx.ha.aliyuncs.com
   username: {username} 
   password: {password}

   Hologres

   host: xxxx.hologres.aliyuncs.com
   database: {your_database} 
   port: {port}
   access_key_id={password}

   Elasticsearch

   uri: http://xxxx.elasticsearch.aliyuncs.com:9200
   username: {username} 
   password: {password}

   Lindorm

   uri: xxxx.lindorm.aliyuncs.com:{port}
   username: {username} 
   password: root:{password}

3. After the connection is created, the vector database connection appears in the list.

   

3.4.1 Create a Smart Tag Definition

In the left menu bar, click AI Asset Management > Datasets > Intelligent Tag Definition > Create Intelligent Tag Definition. The tag configuration page opens. The following is an example configuration:

• Guide Prompt: You are a seasoned driver with extensive experience on both highways and urban roads.

• Tag Definition:

  Sample tag definition for autonomous driving

  {
      "Reflective strips": "Usually yellow, or alternating yellow and black, attached to corners and other permanent protruding obstacles to alert drivers to avoid them. They are strip-shaped, not traffic cones, parking locks, or water-filled barriers.",
      "Parking locks": "Also known as parking space locks, they can be raised to prevent a parking space from being occupied. If a parking lock is present, you must specify whether it is in the raised or lowered state. It is in the raised state if it has a raised frame, otherwise it is in the lowered state.",
      "Lit construction vehicles": "The target is a vehicle with two arrow-shaped lights on the left and right that are lit. Otherwise, it does not exist.",
      "Overturned vehicles": "A vehicle that has overturned on the ground.",
      "Fallen water-filled barriers": "A water-filled barrier is a plastic shell obstacle used to divide road surfaces or form a blockage, typically in the form of a red plastic wall. It is commonly used in road traffic facilities and is often seen on highways, urban roads, and at overpass intersections. It is significantly larger than a traffic cone and has a sheet-like structure. Water-filled barriers are normally upright. If one is lying on the ground, it must be clearly indicated.",
      "Fallen traffic cones": "Also known as conical traffic markers or pylons, commonly called road cones or safety cones, they are cone-shaped temporary road signs. Obstacles that are rod-shaped or sheet-like are not traffic cones because they are not conical. A traffic cone may be knocked over by a car. If a traffic cone is present in the image and you need to determine if it has fallen, observe whether the bottom of the cone (the base of the cone) is in contact with the ground. If it is, it has not fallen. Otherwise, it has.",
      "Charging spaces": "A parking space against a wall with a visible charging gun, charging pile equipment, or marked as a new energy vehicle space is a charging space. It can only appear in a parking lot (indoor or outdoor). Note that parking locks are not related to charging.",
      "Speed bumps": "Usually yellow and black, or just yellow, these are narrow raised strips across the road, perpendicular to the road edge, used to slow down vehicles. They cannot appear within a parking space.",
      "Deceleration lane lines": "Dashed lines in a fishbone pattern on both sides of the lane, inside the solid lines. Both sides must have them to be considered deceleration lane lines.",
      "Ramps": "Can only be identified if there is a clear large curve on a highway. Ramps are usually on the right side of the main highway and are used to enter or exit toll stations.",
      "Ground shadows": "There are clear shadows on the ground.",
      "Cloudy": "Can only be identified if the sky is visible and there are clear clouds in the sky.",
      "Glaring car": "The lights of a car ahead are causing glare (the light changes from a single point to a line of light), usually occurring at night or on rainy days.",
      "Left turn, right turn, U-turn arrows": "Milky white arrow markings painted on the road surface (a few are yellow), not the green and white arrows on highway signs indicating a right curve. When determining the presence of these arrows, only clear arrow markings in the middle of the road lane are the target. Those on the roadside are not. If there are arrows on the ground, the direction is determined as follows: a right-turn arrow rotates clockwise from the base to the tip; a left-turn arrow rotates counter-clockwise from the base to the tip; a U-shaped arrow is a U-turn arrow.",
      "Crosswalks": "Can only exist on the road surface (also possible in parking lots) and at intersections. They must be white lines distributed at repeated intervals parallel to the road edge for pedestrian crossing. They cannot appear on highways, highway ramps, or in tunnels.",
      "Overexposure": "During the day, direct sunlight causes the lens to be overexposed (can only happen during the day).",
      "Motor vehicles": "There are other motor vehicles in the field of view.",
      "Merging in or out": "A place on a highway where multiple lanes merge into one, or one lane divides into multiple lanes.",
      "Intersections": "An intersection where there are no lane lines within the intersection area (refers to the absence of lines within the intersection itself; lines outside the intersection do not matter).",
      "No parking signs": "A sign, either hanging or standing on the ground, with the words 'No Parking' or a symbol of a 'P' in a circle with a diagonal line through it.",
      "Lane lines": "Lane lines on the road, with special attention to blurry lane lines.",
      "Fallen rocks or tires on the road": "Obstacles on the road that affect traffic.",
      "Tunnels": "Pay special attention to distinguishing between entering and exiting a tunnel.",
      "Wet ground on a rainy day": "The ground is slippery due to rain.",
      "Non-motorized vehicles": "Includes non-motorized objects such as bicycles, electric bikes, wheelchairs, unicycles, and shopping carts, which may be parked on the roadside, in parking spaces, or moving on the road."
    }

3.4.2 Create an Offline Smart Tagging Task

1. Click Custom Dataset, click a dataset name to open its details page, and then click the Dataset jobs tab.

   

2. On the task page, click Create job > Smart tag to configure the task parameters.

   

   • Dataset Version: Select the version to label, such as v1.

   • Labeling Model Connection: Select an existing Model Studio model connection.

   • Smart Labeling Model: Supported models include Qwen-VL-MAX and QwenVL-Plus.

   • Max Concurrency: This value depends on the specifications of the EAS model service. For a single card, the recommended maximum concurrency is 5.

   • Intelligent Tag Definition: Select the definition that you just created.

   • Labeling Mode: The available patterns are Increment and Full.

3. After the smart tagging task is created, it appears in the task list. You can click the links in the Actions column to view logs or stop the task.

   Note

   When you start a smart tagging task for the first time, the system builds the metadata. This process may take a long time.

3.9.1 Export to a file

1. Click Export as file. On the configuration page, set the export content and the destination OSS folder, and then click OK.

   

2. You can view the export progress under AI Asset Management > Job > Dataset jobs in the left navigation bar.

3. Use the exported results. After the export is complete, you can mount the exported result file and the original dataset to the training environment, such as a DLC or DSW instance. Then, you can write code to read the exported result file index and load the object files from the original dataset for model training or data analytics.

3.9.2 Export to a logical dataset version

You can import the search results from an advanced dataset into a version of another logical dataset. You can then use the data of that logical dataset version using the dataset software development kit (SDK).

1. Click Export to logical dataset version, select the target logical dataset, and then click Confirm.

   

   If no logical dataset is available, create one as described in the following section:

   Create a logical dataset

   Create a logical dataset. In the navigation pane on the left, click AI Asset Management > Dataset > Create Dataset, and then configure the following key parameters. You can configure the other parameters as needed.

   • Set Dataset Type to Logical.

   • Metadata OSS path: Select the OSS path of the exported metadata.

   • Set Import method to Import later.

   Click OK to create the dataset.

2. Use the logical dataset. After the import task is complete, the destination logical dataset contains the exported metadata. You can use the SDK to load and use the data. For information about how to use the SDK, see the dataset details page.

   

   

   The command to install the SDK is:

   pip install https://pai-sdk.oss-cn-shanghai.aliyuncs.com/dataset/pai_dataset_sdk-1.0.0-py3-none-any.whl