Traditional search methods match and filter media assets based on titles or metadata. Intelligent media asset search uses AI algorithms to analyze audio and video content and allows you to search for media assets based on dimensions such as visual semantics and facial similarity.
Feature description
Intelligent media asset search is also known as intelligent search and multimodal search. You can use AI algorithms to analyze the content of media assets. You can directly search for content that appears in audio and video files without the need to label the media assets.
If you have questions about how to use the intelligent search feature in Intelligent Media Services (IMS), contact technical support in the DingTalk group (ID 30415005038).
The intelligent search feature provides the following capabilities:
Cross-modal video search, which supports multimodal search based on visual information, audio, and text
Supported media assets: videos, audio files, and images
Supported search methods: keyword search, natural language search, image search
Supports up to hundreds of thousands of hours of videos and responses within seconds
Asset type | Feature | Description |
Image | Face | Recognition of more than 500,000 celebrities, politicians, and sensitive figures Search for figures based on face images |
Logo | Recognition of more than 500 TV station logos, including CCTV Recognition of more than 200 mainstream Chinese and English car logos Recognition of common traffic signs Recognition of more than 200,000 brands | |
Subtitles | Recognition of Chinese and English Dynamic subtitles | |
Image semantics | Visual semantic extraction and search based on large multimodal representation models | |
Audio | Audio | Automatic speech recognition (ASR) result search |
Text | Title and description | Title and description search |
Intelligent search provides three search capabilities, which are different in usage scenarios, costs, and fees.
Search capability | Query form | Principle | Scenario | Fee |
Full search | Keyword | Use AI algorithms to automatically extract text information such as tags and keywords from the content of media assets, and then use text information to search for media assets. You can search for names and subtitles that appear in media assets. | Video website content search Video material search | Billed based on the number of stored media assets and the number of searches |
LLM-based search | Natural language | Use large AIGC models to automatically perform vectorized semantic analysis on media asset content to find the content that is closest to the semantics of the query statement. | Video website content search Personal cloud drive search Video material search | Billed based on the length of stored media assets and the number of searches |
Similar face search | Face image | Automatically extract all facial features that appear in media assets, compare and find faces that are similar to the given face image, and return the corresponding media assets and segments. | Personal cloud drive search Face image library search Figure search | Billed based on the number of stored facial features and the number of searches |
Terms
To use intelligent search, you need to understand the following terms:
Term | Description | Remarks |
Media asset | Specific media records. | Audio files, videos, and images are supported. |
Media asset library | A database that stores media assets. | By default, each IMS user has only one media asset library. Videos uploaded in the IMS console are stored in this media asset library. |
Search library | A media asset library that is used to search for media assets |
|
Search index | Search capabilities of the search library. | Used to manage and organize media asset data in the search library. You can enable the following indexes:
|
Use intelligent search in the IMS console
Preparations: Enable intelligent media asset analysis
Log on to the IMS console. In the left-side navigation pane, click Intelligent Retrieval.
Click Intelligent Retrieval Configuration in the upper-right corner to go to the configuration page.
If you want to use full search, turn on Automatic Analysis.
If you want to use LLM-based search, turn on Foundation Model Search.
Upload media assets
Log on to the IMS console. In the left-side navigation pane, choose Media Library - Audio/Video.
Click Upload Audio/Video. For more information about how to upload media assets, see Upload media assets by using the IMS console.
After media assets are uploaded, the media assets are automatically analyzed. You can search for media assets after the media assets are analyzed. If you want to check whether automatic analysis is complete, click Manage in the media asset list to go to the details page and view the Content Search and Save Status of the media asset.
Search for media assets
Log on to the IMS console. In the left-side navigation pane, click Intelligent Retrieval.
Select Full Retrieval or Large Model Retrieval from the drop-down list:
To use full search, we recommend that you enter words or phrases, such as names, places, and keywords.
To use LLM-based search, we recommend that you use natural language, such as "sunrise by the sea" and "cars coming and going on the street".
Click the Media Type icon in the header to search for audio files, videos, or images.
Use self-managed search libraries
In the IMS console, you can only search for content in the default media asset library. If your media asset library is not in IMS and you want to use intelligent search, you can call API operations to create a self-managed search library.
Procedure: Create a search library - Create search indexes - Add media assets to the search library - Search for media assets
After you create an index, the index takes effect only for new media assets that are added to the search library. The index does not take effect for existing media assets.
To check whether a media asset is indexed, call the QueryMediaIndexJob operation.
You can search only for indexed media assets.
The following section describes the steps:
Full search
Creates a search library with a name. For more information, see CreateSearchLib.
Create an index in the search library. For more information, see CreateSearchIndex. To perform a full search, set IndexType to
aiLabel.Call the InsertMediaToSearchLib operation to add media asset information to the search library.
Call the SearchMediaByAILabel operation to search for media assets in two steps:
In the first call, the fuzzy search mode is selected. As a result, a list of matched media assets is returned.
In the second call, the exact search mode is selected. As a result, a list of matched clips in a specific media asset is returned.
LLM-based search
Create a search library with a name. For more information, see CreateSearchLib.
Create an index in the search library. For more information, see CreateSearchIndex. To perform an LLM-based search, you must set IndexType to
mm.Call the InsertMediaToSearchLib operation to add media asset information to the search library.
Call the SearchMediaByMultimodal operation to search for media assets. The matched media assets and the matched clips in each media asset are returned.
Similar face search
Create a search library with a name. For more information, see CreateSearchLib.
Create an index in the search library. For more information, see CreateSearchIndex. To perform a similar face search, set IndexType to
face.Call the InsertMediaToSearchLib operation to add media asset information to the search library.
Call the SearchMediaByFace operation to search for media assets. A list of matched media assets is returned.
Call the SearchMediaClipByFace operation to search for the corresponding clips in a matched media asset. A list of clips and information about faces are returned.