How to use AI to understand media asset content

Intelligent Media Services (IMS) analyzes the content of media assets — videos, audio, and images — using AI. Call the OpenAPI operations described in this guide to run content understanding jobs against your media assets.

Prerequisites

Before you begin, ensure that you have:

IMS activated. For more information, see Activate a service.
The IMS server-side software development kit (SDK) installed. For more information, see SDK installation.

Basic usage

Basic usage runs content understanding against a media asset using a built-in face recognition configuration. All three steps use the same client setup — the imports and authentication code are identical across each code block.

Create an algorithm template.

Call the CreateCustomTemplate operation to create a custom AI-based analysis template. The example below creates a face recognition template that targets celebrities, politicians, and sensitive content.

Sample code:

# -*- coding: utf-8 -*-
from alibabacloud_ice20201109.client import Client as ICE20201109Client
from alibabacloud_credentials.client import Client as CredentialClient
from alibabacloud_tea_openapi import models as open_api_models
from alibabacloud_ice20201109 import models as ice20201109_models
from alibabacloud_tea_util import models as util_models

# For production environments, use a more secure authentication method, such as RAM roles.
# For more information about how to configure credentials, see https://www.alibabacloud.com/help/document_detail/378659.html.
credential = CredentialClient()
config = open_api_models.Config(credential=credential)
# For the endpoint, see https://api.aliyun.com/product/ICE
config.endpoint = 'ice.cn-xxx.aliyuncs.com'
client = ICE20201109Client(config)

create_custom_template_request = ice20201109_models.CreateCustomTemplateRequest(
    name='face_template_001',
    # AnalyseTypes: the analysis type (face, landmark, object, logo, or custom tag)
    # FaceCategoryIds: comma-separated face categories to recognize
    template_config='{"AnalyseTypes":"face","FaceCategoryIds":"celebrity,politician,sensitive"}',
    type=11
)
runtime = util_models.RuntimeOptions()
response = client.create_custom_template_with_options(create_custom_template_request, runtime)
print(response)

Note the template_id in the response — you need it in step 2.

Submit a content understanding job.

Call the SubmitSmarttagJob operation to submit a content understanding job against a media asset. Replace <template_id> with the value from step 1.

Sample code:

# -*- coding: utf-8 -*-
from alibabacloud_ice20201109.client import Client as ICE20201109Client
from alibabacloud_credentials.client import Client as CredentialClient
from alibabacloud_tea_openapi import models as open_api_models
from alibabacloud_ice20201109 import models as ice20201109_models
from alibabacloud_tea_util import models as util_models

# For production environments, use a more secure authentication method, such as RAM roles.
# For more information about how to configure credentials, see https://www.alibabacloud.com/help/document_detail/378659.html.
credential = CredentialClient()
config = open_api_models.Config(credential=credential)
# For the endpoint, see https://api.aliyun.com/product/ICE
config.endpoint = 'ice.cn-xxx.aliyuncs.com'
client = ICE20201109Client(config)

submit_smarttag_job_request_input = ice20201109_models.SubmitSmarttagJobRequestInput(
    type='URL',
    media='https://xxx.jpeg'  # Replace with the URL of your media asset
)
submit_smarttag_job_request = ice20201109_models.SubmitSmarttagJobRequest(
    title='face_test-001',
    input=submit_smarttag_job_request_input,
    template_id='<template_id>'  # From step 1
)
runtime = util_models.RuntimeOptions()
response = client.submit_smarttag_job_with_options(submit_smarttag_job_request, runtime)
print(response)

Note the job_id in the response — you need it in step 3.

Query the job result.

Call the QuerySmarttagJob operation to retrieve the job result. Replace <job_id> with the value from step 2.

Sample code:

# -*- coding: utf-8 -*-
from alibabacloud_ice20201109.client import Client as ICE20201109Client
from alibabacloud_credentials.client import Client as CredentialClient
from alibabacloud_tea_openapi import models as open_api_models
from alibabacloud_ice20201109 import models as ice20201109_models
from alibabacloud_tea_util import models as util_models

# For production environments, use a more secure authentication method, such as RAM roles.
# For more information about how to configure credentials, see https://www.alibabacloud.com/help/document_detail/378659.html.
credential = CredentialClient()
config = open_api_models.Config(credential=credential)
# For the endpoint, see https://api.aliyun.com/product/ICE
config.endpoint = 'ice.cn-xxx.aliyuncs.com'
client = ICE20201109Client(config)

query_smarttag_job_request = ice20201109_models.QuerySmarttagJobRequest(
    job_id='<job_id>'  # From step 2
)
runtime = util_models.RuntimeOptions()
response = client.query_smarttag_job_with_options(query_smarttag_job_request, runtime)
print(response)

Advanced usage

Advanced usage extends basic usage with a custom recognition library. Before submitting a job, build a library of named entities — for example, specific people or brand logos — and attach it to the algorithm template. Steps 4–6 (submit job and query result) follow the same process as the basic usage steps.

Create a custom recognition library.

Call the CreateRecognitionLib operation to create a custom recognition library. Note the lib_id in the response — you need it in steps 2 and 3.

Sample code:

# -*- coding: utf-8 -*-
from alibabacloud_ice20201109.client import Client as ICE20201109Client
from alibabacloud_credentials.client import Client as CredentialClient
from alibabacloud_tea_openapi import models as open_api_models
from alibabacloud_ice20201109 import models as ice20201109_models
from alibabacloud_tea_util import models as util_models

# For production environments, use a more secure authentication method, such as RAM roles.
# For more information about how to configure credentials, see https://www.alibabacloud.com/help/document_detail/378659.html.
credential = CredentialClient()
config = open_api_models.Config(credential=credential)
# For the endpoint, see https://api.aliyun.com/product/ICE
config.endpoint = 'ice.cn-xxx.aliyuncs.com'
client = ICE20201109Client(config)

create_recognition_lib_request = ice20201109_models.CreateRecognitionLibRequest(
    algorithm='face',
    lib_name='face_lib_001'
)
runtime = util_models.RuntimeOptions()
response = client.create_recognition_lib_with_options(create_recognition_lib_request, runtime)
print(response)

Create a custom entity.

Call the CreateRecognitionEntity operation to add a named entity to the recognition library. Replace <lib_id> with the value from step 1.

Sample code:

# -*- coding: utf-8 -*-
from alibabacloud_ice20201109.client import Client as ICE20201109Client
from alibabacloud_credentials.client import Client as CredentialClient
from alibabacloud_tea_openapi import models as open_api_models
from alibabacloud_ice20201109 import models as ice20201109_models
from alibabacloud_tea_util import models as util_models

# For production environments, use a more secure authentication method, such as RAM roles.
# For more information about how to configure credentials, see https://www.alibabacloud.com/help/document_detail/378659.html.
credential = CredentialClient()
config = open_api_models.Config(credential=credential)
# For the endpoint, see https://api.aliyun.com/product/ICE
config.endpoint = 'ice.cn-xxx.aliyuncs.com'
client = ICE20201109Client(config)

create_recognition_entity_request = ice20201109_models.CreateRecognitionEntityRequest(
    algorithm='face',
    lib_id='<lib_id>',      # From step 1
    entity_name='Xiao Shuai'
)
runtime = util_models.RuntimeOptions()
response = client.create_recognition_entity_with_options(create_recognition_entity_request, runtime)
print(response)

Note the entity_id in the response — you need it in step 3.

Add a sample image to the entity.

Call the CreateRecognitionSample operation to attach a sample image or text tag to the entity. Replace <lib_id> and <entity_id> with values from steps 1 and 2.

Sample code:

# -*- coding: utf-8 -*-
from alibabacloud_ice20201109.client import Client as ICE20201109Client
from alibabacloud_credentials.client import Client as CredentialClient
from alibabacloud_tea_openapi import models as open_api_models
from alibabacloud_ice20201109 import models as ice20201109_models
from alibabacloud_tea_util import models as util_models

# For production environments, use a more secure authentication method, such as RAM roles.
# For more information about how to configure credentials, see https://www.alibabacloud.com/help/document_detail/378659.html.
credential = CredentialClient()
config = open_api_models.Config(credential=credential)
# For the endpoint, see https://api.aliyun.com/product/ICE
config.endpoint = 'ice.cn-xxx.aliyuncs.com'
client = ICE20201109Client(config)

create_recognition_sample_request = ice20201109_models.CreateRecognitionSampleRequest(
    algorithm='face',
    lib_id='<lib_id>',        # From step 1
    entity_id='<entity_id>',  # From step 2
    image_url='https://xxx.jpg'  # Replace with the URL of the sample image
)
runtime = util_models.RuntimeOptions()
response = client.create_recognition_sample_with_options(create_recognition_sample_request, runtime)
print(response)

Create an algorithm template with your custom library.

Follow the same process as step 1 in basic usage. In the template_config, set the analysis type and the lib_id of your custom library for the recognition targets — faces, landmarks, objects, logos, or custom tags.
Submit a content understanding job.

Follow the same process as step 2 in basic usage.
Query the job result.

Follow the same process as step 3 in basic usage.