Platform For AI:Manage and use multimodal data

3.1.1. Activate PAI, create a default workspace, and obtain administrator permissions

1. Use your Alibaba Cloud account to activate PAI and create a workspace. Log on to the PAI console, select a region in the upper-left corner, and then authorize and activate the service. For more information, see Activate PAI and create a workspace.

2. Grant permissions to the operating account. You can skip this step if you are using your Alibaba Cloud account. If you are using a Resource Access Management (RAM) user, you must have the administrator role for the workspace. For more information about how to grant permissions, see the "Configure member roles" section of Manage workspaces.

3.1.2. Activate Model Studio and create an API key

For more information about how to activate Alibaba Cloud Model Studio and create an API key, see Obtain an API key.

3.1.3. Create a vector database

Create a vector database instance

Multimodal dataset management currently supports the following Alibaba Cloud vector databases:

• Elasticsearch (Vector-enhanced Edition, version 8.17.0 or later)

• OpenSearch (Vector Search Edition)

• Milvus (version 2.4 or later)

• Hologres (version 4.0.9 or later)

For information about how to create an instance for each cloud vector database, see the documentation for the corresponding product.

Configure the network and whitelist

• Public network access

  If your vector database instance has a public endpoint, add the following IP addresses to the instance's public access whitelist. This allows the multimodal data management service to access the instance over the public network. For information about how to configure the whitelist for Elasticsearch, see Configure a public or private IP address whitelist for an instance.

  Region	IP address list
  Hangzhou	47.110.230.142, 47.98.189.92
  Shanghai	47.117.86.159, 106.14.192.90
  Shenzhen	47.106.88.217, 39.108.12.110
  Ulanqab	8.130.24.177, 8.130.82.15

• Private network access

  To apply, submit a ticket.

Create a vector index table (Optional)

In some vector databases, a vector index table is also called a collection or an index.

The index table schema must be defined as follows:

{
    "id":"text",                    // Primary key ID. This must be defined in OpenSearch. In other databases, it exists by default and does not need to be defined.
    "index_set_id": "keyword",      // Index set ID. Must support indexing.
    "file_meta_id": "text",         // File metadata ID.
    "dataset_id": "text",           // Dataset ID.
    "dataset_version": "text",      // Dataset version.
    "uri": "text",                  // URI of the OSS file.
    "file_vector": {                // Vector field.
        "type": "float",            // Vector type: float.
        "dims": 1536,               // Vector dimensions. Customizable.
        "similarity": "DotProduct"  // Vector distance algorithm. Cosine distance or dot product.
    }
}

This topic uses Elasticsearch as an example to demonstrate how to create a semantic index table using Python. For information about how to create index tables for other types of vector databases, see the documentation for the corresponding cloud product. The following code provides an example:

from elasticsearch import Elasticsearch

# 1. Connect to the Alibaba Cloud Elasticsearch instance.
# Note:
# (1) Python 3.9 or later is required: python3 -V
# (2) The Elasticsearch client must be version 8.x: pip show elasticsearch
# (3) If you use a VPC endpoint, the caller must be able to access the VPC of the ES instance. Otherwise, use a public endpoint and add the public IP address of the caller to the ES whitelist.
# The default userName is elastic.
es_client = Elasticsearch(
    hosts=["http://es-cn-l4p***5z.elasticsearch.aliyuncs.com:9200"],
    basic_auth=("{userName}", "{password}"),
)

# 2. Define the index name and structure. The HNSW index algorithm is used by default.
index_name = "dataset_embed_test"
index_mapping = {
    "settings": {
        "number_of_shards": 1,          # Number of shards.
        "number_of_replicas": 1         # Number of replicas.
    },
    "mappings": {
        "properties": {
            "index_set_id": {
                "type": "keyword"
            },
            "uri": {
                "type": "text"
            },
            "file_meta_id": {
                "type": "text"
            },
            "dataset_id": {
                "type": "text"
            },
            "dataset_version": {
                "type": "text"  
            },
            "file_vector": {
                "type": "dense_vector",  # Define file_vector as a dense vector type.
                "dims": 1536,  # The vector dimension is 1536.
                "similarity": "dot_product"  # The similarity calculation method is dot product.
            }
        }
    }
}

# 3. Create the index.
if not es_client.indices.exists(index=index_name):
    es_client.indices.create(index=index_name, body=index_mapping)
    print(f"Index {index_name} created successfully!")
else:
    print(f"Index {index_name} already exists. It will not be created again.")

# 4. View the structure of the created index table (Optional).
# indexes = es_client.indices.get(index=index_name)
# print(indexes)

3.3.1. Create a smart tagging model connection

1. Go to the PAI workspace. In the navigation pane on the left, click AI Asset Management > Connections > Model Services > Create Connection to open the Create Connection page.

   

2. Select Model Studio Large Model Service and configure the Model Studio api_key.

   

3. After the connection is created, the Model Studio large model service appears in the list.

   

3.3.2. Create a custom semantic indexing model connection

1. In the navigation pane on the left, click Model Gallery. Find and deploy the GME multimodal retrieval model to obtain an EAS service. The deployment takes about 5 minutes. The deployment is successful when the status changes to Running.

   Important

   When you no longer need the index model, stop and delete the service to avoid further charges.

   

2. Go to the PAI workspace. In the navigation pane on the left, click AI Asset Management > Connections > Model Services > Create Connection to open the Create Connection page.

3. Select General Multimodal Embedding Model Service. Click the EAS Service input box and select the GME multimodal retrieval model that you deployed. If the service provider is not under your current account, you can select a third-party model service.

   

   

4. After the connection is created, the model connection service appears in the list.

   

3.3.3. Create a vector database connection

1. In the navigation pane on the left, click AI Asset Management > Connections > Databases > Create Connection to open the Create Connection page.

   

2. The multimodal retrieval service supports Milvus, Lindorm, OpenSearch, and Elasticsearch vector databases. This section uses Elasticsearch as an example to show how to create a connection. Select Search And Analytics Service - Elasticsearch and configure parameters such as uri, username, and password. For more information about the configuration, see Create a database connection.

   

3. After the connection is created, the vector database connection appears in the list.

   

3.4.1. Create smart tag definitions

1. In the navigation pane on the left, click AI Asset Management > Datasets > Smart Tag Definitions > Create Smart Tag to open the tag configuration page. The following example shows how to configure the tags:

   • Set Guiding Prompt to: As an expert driver with many years of experience, you are highly experienced in driving on both highways and urban roads.

   • Set Tag Definition to:

     Sample tag definitions for autonomous driving

     {
         "Reflective strips": "Usually yellow, or yellow and black, attached to permanent protruding obstacles such as wall corners to alert drivers to avoid them. They are strip-shaped, not traffic cones, parking locks, or water-filled barriers!",
         "Parking lock": "Also known as a parking space lock. When raised, it can prevent the parking space from being occupied. If a parking lock is present, you must specify whether it is in the raised or lowered state. It is in the raised state if the frame is up, otherwise it is in the lowered state.",
         "Construction vehicle with lights on": "The target is a vehicle with two arrow-shaped lights on the left and right that are lit. Otherwise, it does not exist.",
         "Overturned vehicle": "A vehicle that has overturned on the ground.",
         "Fallen water-filled barrier": "A water-filled barrier is a plastic shell obstacle used to divide the road or form a barrier, usually in the form of a red plastic wall. It is commonly used in road traffic facilities and is often seen on highways, urban roads, and at overpass intersections. It is significantly larger than a traffic cone and has a sheet-like structure. Water-filled barriers are normally upright. If one is lying on the ground, it needs to be clearly indicated.",
         "Fallen traffic cone": "Also known as a cone-shaped traffic marker or pylon. It is a cone-shaped temporary road sign. Rod-shaped or sheet-like obstacles are not traffic cones because they are not cone-shaped. A traffic cone may be knocked over by a car. If a traffic cone is present in the image and you need to determine if it has fallen, observe whether the bottom of the cone (the base of the cone) is in contact with the ground. If it is, it has not fallen. Otherwise, it has fallen.",
         "Charging space": "A parking space against a wall with a visible charging gun and charging pile equipment, or marked as a new energy vehicle space, is a charging space. It can only appear in a parking lot (both indoor and outdoor are possible). Note that parking locks are not related to charging.",
         "Speed bump": "Usually yellow and black, or just yellow. It is a narrow raised strip across the road, perpendicular to the road edge, used to slow down vehicles. It cannot appear within a parking space.",
         "Rumble strips": "Fishbone-shaped dashed lines on both sides of the lane, inside the solid line. Both sides must have them to be considered rumble strips.",
         "Ramp": "Can only be determined to exist if a large curve on a highway is clearly visible. Ramps are usually on the right side of the main highway road, for entering and exiting toll stations.",
         "Ground shadow": "There are obvious shadows on the ground.",
         "Cloudy": "Can only be determined to exist if the sky is visible and there are obvious clouds in the sky.",
         "Car with glare": "The front lights are causing glare (the light has changed from a single point to a line of light), which usually occurs at night or on rainy days.",
         "Left turn, right turn, U-turn arrow": "A milky white arrow sign painted on the road surface of the lane (a few are yellow), not the green and white arrow on a highway sign indicating a right curve. When determining if these arrows exist, only the clear arrow signs in the middle of the lane surface are the target. Others, such as those on the roadside, are not. If there is an arrow on the ground, the method to determine its direction is: a right-turn arrow rotates clockwise from the base to the tip; a left-turn arrow rotates counter-clockwise from the base to the tip; a U-shaped arrow is a U-turn arrow.",
         "Crosswalk": "Can only exist on the road surface (also possible in a parking lot) or at an intersection. It must be white lines distributed at repeated intervals parallel to the roadside, for pedestrian crossing. It cannot appear on highways, highway ramps, or in tunnels.",
         "Overexposure": "During the day, direct sunlight causes lens overexposure (can only happen during the day).",
         "Motor vehicle": "There are other motor vehicles in the field of view.",
         "Merging in and out": "Where multiple highway lanes become one, or one lane divides into multiple lanes.",
         "Intersection": "An intersection, and there are no lane lines within the intersection (meaning none within the intersection section, it does not matter if there are any outside the intersection).",
         "No parking sign": "A sign hanging or standing on the ground with the words 'No Parking' or a symbol of a P in a circle with a diagonal line through it.",
         "Lane line": "Lane lines on the road, with special attention to blurry lane lines.",
         "Fallen rocks, tires on the road": "Obstacles on the road that affect traffic.",
         "Tunnel": "Pay special attention to distinguish when entering or exiting a tunnel.",
         "Wet ground on a rainy day": "The ground is wet and slippery on a rainy day.",
         "Non-motor vehicle": "Includes non-motorized objects such as bicycles, electric bikes, wheelchairs, unicycles, and shopping carts. They may be parked on the roadside, in parking spaces, or moving on the road."
       }

3.4.2. Create an offline smart tagging task

1. Click Custom Datasets. Click the name of the dataset to go to its details page. Then, click the Dataset jobs tab.

   

2. On the tasks page, click Create Task > Smart Tagging and configure the task parameters.

   

   • Set Dataset Version to the version that you want to tag, such as v1.

   • Set Smart Tagging Model Connection to the Model Studio model connection that you created.

   • Set Smart Tagging Model. Qwen-VL Max and Qwen-VL Plus are supported.

   • Set Maximum Concurrency based on the specifications of the EAS model service. For a single GPU, the recommended maximum concurrency is 5.

   • Set Smart Tag Definition to the smart tag definition that you created.

   Note

   Currently, the tagging mode supports tagging only all files in a dataset version.

3. After the smart tagging task is created, it appears in the task list. You can monitor the running task and click the link on the right side of the list to view logs or stop the task.

   Note

   When you start a smart tagging task for the first time, metadata is built. This process may take a long time.

3.9.1. Export to a file

1. Click Export To File. On the configuration page, set the export content and the destination OSS directory. Then, click OK.

   

2. To view the export progress, in the navigation pane on the left, click AI Asset Management > Tasks > Dataset Tasks.

3. Use the exported results. After the export is complete, you can mount the exported result file and the original dataset to the corresponding training environment, such as a Deep Learning Containers (DLC) or Data Science Workshop (DSW) instance. Then, you can use code to read the exported result file index and load the object files from the original dataset for model training or analysis.

3.9.2. Export to a logical dataset version

You can import the search results from an advanced dataset into a version of a logical dataset. You can then use the dataset software development kit (SDK) to access the data in that logical dataset version.

1. Click Export To Logical Dataset Version, select the destination logical dataset, and click Confirm.

   

   If no logical datasets are available to select, see the following information:

   Create a logical dataset

   Create a logical dataset. In the navigation pane on the left, click AI Asset Management > Datasets > Create Dataset. Then, configure the following key parameters. Configure other parameters as needed:

   • Set Dataset Type to Logical.

   • Set Metadata OSS path to an OSS path for export.

   • Set Import method to Import later.

   Click OK to create the dataset.

2. Use the logical dataset. After the import task is complete, the destination logical dataset contains the exported metadata. You can use the SDK to load and use the data. For information about how to use the SDK, see the dataset's details page.

   

   

   The command to install the SDK is:

   pip install https://pai-sdk.oss-cn-shanghai.aliyuncs.com/dataset/pai_dataset_sdk-1.0.0-py3-none-any.whl