All Products
Search
Document Center

Platform For AI:Usage notes on multimedia analysis SDK for Python

Last Updated:Feb 04, 2024

You can use multimedia analysis SDK for Python provided by Platform for AI (PAI) to call different algorithm-powered services. This topic describes how to use multimedia analysis SDK for Python to call different model services. This topic also provides sample requests and responses.

Step 1: Install multimedia analysis SDK for Python

The environment relies on Python 3 or later. Run the following command to install multimedia analysis SDK for Python:

wget https://ai-service-data.oss-cn-beijing.aliyuncs.com/python-sdk/ai_service_python_sdk-1.1.4-py3-none-any.whl
pip install ai_service_python_sdk-1.1.4-py3-none-any.whl

Step 2: Initialize the client

Run the following command to initialize the environment:

from ai_service_python_sdk.client.api_client import ApiClient
client = ApiClient('<HOST>', '<YOUR-APPID>', '<YOUR-TOKEN>')

Modify the following parameters based on your business requirements.

Parameter

Description

<HOST>

The server address is http://ai-service.ce8cc13b6421545749e7b4605f3d02607.cn-hangzhou.alicontainer.com.

<YOUR-APPID>

After you activate multimedia analysis, you can view AppID on the multimedia analysis page. For more information, see Multimedia Analysis.

<YOUR-TOKEN>

After you activate multimedia analysis, you can view Token on the multimedia analysis page. For more information, see Multimedia Analysis.

Examples

Image quality evaluation

You can evaluate the quality of an image. The model service returns a floating-point number from 0 to 100. The following code block provides a sample request:

from ai_service_python_sdk.client.api.ai_service_image_api import AiServiceImageApi

# The URL of the image. 
image_url = 'https://your_image_url.jpg'
ai_service_api = AiServiceImageApi(client)
# The name of the custom model. 
model_name = ''
# The response settings of the model. If the mode does not have response parameters, you can specify an empty dictionary. 
configure = {}
# The model service that you want to call. 
response = ai_service_api.image_qa(image_url, model_name, configure)
# The ID of the request. 
request_id = response.request_id
# The status of the request. 
code = response.code
# The detailed status information about the request. 
message = response.message
# The return value. 
data = response.data
# Display the return value. 
print(response)

The following table describes the parameters in the preceding statement.

Parameter

Data type

Required

Description

image_url

STRING

Yes

The URL of the image. We recommend that the size of the image is no larger than 5 MB and the resolution of the image is equal to or higher than 240 × 240.

model_name

STRING

Yes

By default, an empty string is specified to call a general-purpose model.

If the general-purpose model cannot meet your business requirements, contact your account manager to obtain a custom model. You can specify the name of a custom model to call the custom model.

configure

Dict

Yes

The response settings of the model. Specify an empty dictionary for the model service.

The following command output is returned:

{
    "code":"OK",
    "data":{
        "image_height":214,
        "image_width":398,
        "iqa_result":66.88
    },
    "message":"success",
    "request_id":"1d4572a1-2800-4213-9e9c-33f9fa7e****"
}

The following table describes the response parameters.

Parameter

Description

request_id

The request ID, which is of the STRING type.

code

The request status code, which is of the STRING type.

message

The detailed status information about the request, which is of the STRING type. Sample command output:

  • success

  • Invalid Input - image data error - base64decode error: base64decode error: incorrect base64 format data

  • Invalid Input - json format error - json parse runtime exception

For more information, see Error codes.

data

The content returned by the model service, which is of the DICT type. Parameter descriptions:

  • image_height: the height of the image in pixels, which is of the INT type.

  • image_width: the width of the image in pixels, which is of the INT type.

  • iqa_result: the quality score of the image, which is of the FLOAT type. The value is accurate to two decimal places.

Face attribute analysis

You can obtain the generated face-related attributes, such as the face shape, hair color, hairstyle, and appearance. The following code block shows a sample request:

from ai_service_python_sdk.client.api.ai_service_image_api import AiServiceImageApi

# The URL of the image. 
image_url = 'https://your_image_url.jpg'
ai_service_api = AiServiceImageApi(client)
# The name of the custom model. 
model_name = ''
# The response settings of the model. If the mode does not have response parameters, you can specify an empty dictionary. 
configure = {}
# The model service to be called. 
response = ai_service_api.face_attr_image(image_url, model_name, configure)
# The ID of the request. 
request_id = response.request_id
# The status of the request. 
code = response.code
# The detailed status information about the request. 
message = response.message
# The return value. 
data = response.data
# Display the return value. 
print(response)

The following table describes the parameters in the preceding statement.

Parameter

Data type

Required

Description

image_url

STRING

Yes

The URL of the image. The face section in the image must have a resolution that is higher than 100 × 100.

model_name

STRING

Yes

By default, an empty string is specified to call a general-purpose model.

If the general-purpose model cannot meet your business requirements, contact your account manager to obtain a custom model. You can specify the name of a custom model to call the custom model.

configure

Dict

Yes

The response settings of the model. Specify an empty dictionary for the model service.

The following result is returned:

{'code': 'OK',
 'data': {'attr': [{'FaceShape': {'Triangular': 0.0013,
                                  'Round': 0.0147,
                                  'Heart': 0.0405,
                                  'Square': 0.0001,
                                  'Oval': 0.9266,
                                  'Diamond': 0.0027,
                                  'Oblong': 0.0141},
                    'FemaleHairStyle': {'BangsType': {'Curtain bangs': 0.5932,
                                                      'Braided bangs': 0.0058,
                                                      'Side-swept bangs': 0.0575,
                                                      'No bangs': 0.3078,
                                                      'See-through bangs': 0.0143,
                                                      'Blunt bangs': 0.0213},
                                        'FemaleCurlyHairType': {'Flow perm': 0.1289,
                                                                'Big wavy curl': 0.2526,
                                                                'Small curls': 0.084,
                                                                'Smooth curls': 0.2027,
                                                                'Air wave perm': 0.1434,
                                                                'Jelly perm': 0.1539,
                                                                'Cone curls': 0.0345},
                                        'FemaleHairType': {'Curly hair': 0.1941,
                                                           'Bun': 0.0112,
                                                           'Straight hair': 0.7947},
                                        FemaleLongShortHair hair ': {' Medium-length': 0.2807,
                                                                'Short hair': 0.0346,
                                                                'Long hair': 0.6846},
                                        'Tied hair': 0.3054,
                                        "Braided hair": 0.3453},
                    'HairColor': {'Others': 0.0001,
                                  'Coffee brown': 0,
                                  'Granny gray': 0,
                                  'Chestnut': 0.0025,
                                  'Brown': 0.0622,
                                  'Gradient': 0.0004,
                                  'Claret': 0.0001,
                                  'Blonde': 0,
                                  'Buff': 0,
                                  'Black': 0.9347},
                    'MaleHairStyle': {'Curtains': 0.9939,
                                      'Buzz cut': 0,
                                      'Bullet': 0.0033,
                                      'Crew cut': 0,
                                      'Induction buzz cut': 0.001,
                                      'Disconnected buzz cut': 0.0008,
                                      'Disconnected bob': 0,
                                      'Slicked back': 0.0001,
                                      'Mohawk': 0.0006},
                    'bbox': [91, 0, 430, 452],
                    'face_score': 3.2431,
                    'index': 0}]},
 'message': 'success',
 'request_id': '9b2f2c0e-d2c0-45f8-b922-058d00edc9ad'}

The following table describes the response parameters.

Parameter

Description

request_id

The request ID, which is of the STRING type.

code

The request status code, which is of the STRING type.

message

The detailed status information about the request, which is of the STRING type. Sample command output:

  • success

  • Invalid Input - image data error - base64decode error: base64decode error: incorrect base64 format data

  • Invalid Input - json format error - json parse runtime exception

For more information, see Error codes.

Data

The content returned by the model service, which is of the DICT type. For more information about the response parameters, see Data parameters.

Table 1. Data parameters

Parameter

Description

attr

The attribute content of the image, which is of the LIST type. Each element in the list indicates one face attribute.

Note

If the list is empty, no faces are detected.

FaceShape

The shape of the face, which is of the DICT type.

FemaleHairStyle

The female hairstyle type, which is of the DICT type.

BangsType

The female bangs type, which is of the DICT type.

FemaleCurlyHairType

The female curl type, which is of the DICT type.

If the FemaleHairType parameter is most likely to be set to a curly hairstyle, the value of this parameter is read.

FemaleHairType

The female hair type, which is of the DICT type.

FemaleLongShortHair

The female hair length, which is of the DICT type.

Tied hair

Specifies whether the hair of the female is tied. The value is of the FLOAT type.

If the value of this parameter is greater than or equal to 0.5, the hair is tied.

Braided hair

Specifies whether the hair of the female is braided. The value is of the FLOAT type.

If the value of this parameter is greater than or equal to 0.5, the hair is braided.

HairColor

The hair color, which is of the DICT type.

MaleHairStyle

The male hairstyle, which is of the DICT type.

bbox

The rectangle section in which the face resides, which is of the LIST type. The value is in the [xmin,ymin,xmax,ymax] format.

index

The ID of the face in the image. The value is of the INT type and starts from 0.

Age analysis

You can identify the age interval of the major face in the image.

Note

The model is a custom model. If you want to use the model, contact your account manager and provide the AppId for configuration.

The following code block shows a sample request:

from ai_service_python_sdk.client.api.ai_service_image_api import AiServiceImageApi

# The URL of the image.
image_url = 'https://your_image_url.jpg'
ai_service_api = AiServiceImageApi(client)
# The name of the custom model.
model_name = 'fairface'
# The response settings of the model. If the mode does not have response parameters, you can specify an empty dictionary.
configure = {}
# The model service to be called.
response = ai_service_api.face_attr_image(image_url, model_name, configure)
# The ID of the request.
request_id = response.request_id
# The status of the request.
code = response.code
# The detailed status information about the request.
message = response.message
# The return value.
data = response.data
# Display the return value.
print(response)

The following table describes the parameters in the statement.

Parameter

Data type

Required

Description

image_url

STRING

Yes

The URL of the image. The face section in the image must have a resolution that is higher than 100 × 100.

model_name

STRING

Yes

The name of the custom model.

configure

Dict

Yes

The response settings of the model. Specify an empty dictionary for the model service.

The following result is returned:

{
    "request_id":"10000",
    "code":"OK",
    "message":"success",
    "data":{
        "age_range":"3-9",
        "age_raneg_prob":0.9525
    }
}

Parameter

Description

request_id

The request ID, which is of the STRING type.

code

The request status code, which is of the STRING type.

message

The detailed status information about the request, which is of the STRING type. Sample command output:

  • success

  • Invalid Input-image data error - {error message}

data

The content returned by the model service, which is of the DICT type. The following table describes the response parameters.

  • age_range: the age interval, which is of the STRING type. Possible values are '0-2', '3-9', '10-19', '20-29', '30-39', '40-49', '50-59', '60-69', '70+'.

  • age_raneg_prob: the probability of the age interval, which is of the FLOAT type.

Multiple image labels

You can add multiple labels to an image. The model service can return top K labels with the highest probabilities and the probability of each label. The model service can also return high-dimensional features of the image. The following code block shows a sample request:

from ai_service_python_sdk.client.api.ai_service_image_api import AiServiceImageApi

# The URL of the image. 
image_url = 'https://your_image_url.jpg'
ai_service_api = AiServiceImageApi(client)
# The name of the custom model. 
model_name = ''
# The response settings of the model. 
configure = {'tag_top_k': 5, 'output_embedding': False}
# The model service to be called. 
response = ai_service_api.multi_label_image_v2(image_url, model_name, configure)
# The ID of the request. 
request_id = response.request_id
# The status of the request. 
code = response.code
# The detailed status information about the request. 
message = response.message
# The return value. 
data = response.data
# Display the return value. 
print(response)

The following table describes the parameters in the preceding statement.

Parameter

Data type

Required

Description

image_url

STRING

Yes

The URL of the image. We recommend that the size of the image is no larger than 5 MB and the resolution of the image is equal to or higher than 240 × 240.

model_name

STRING

Yes

By default, an empty string is specified to indicate a general-purpose model.

If the general-purpose model cannot meet your requirements, you can contact the technical support of PAI to obtain a custom model. You can specify the name of a custom model to call the custom model.

configure

Dict

Yes

The response settings of the model. The default value is None. For more information, see Configure parameters.

Note

If configure is set to None, the parameters in the configure list use the default settings.

Configure parameters

Parameter

Type

Required

Description

tag_top_k

INT

No

The top K labels. Valid values of K: 1 to 10. Default value: 10.

output_embedding

BOOL

No

Specify whether to return the image embedding. Valid values:

  • True (default): return the high-dimensional features of the image.

  • False

The following command output is returned:

{'code': 'OK',
 'data': {'image_height': 510,
          'image_width': 516,
          'tag_result': [{'score': 0.505, 'tag': 'Smiling face'},
                         {'score': 0.493, 'tag': 'Lady'},
                         {'score': 0.475, 'tag': 'Beauty'},
                         {'score': 0.438, 'tag': 'Smile'},
                         {'score': 0.432, 'tag': 'Honey tea brown'}]},
 'message': 'success',
 'request_id': 'c9e5aa33-4d8d-49ce-8991-bcbb5f12e43b'}

The following table describes the response parameters.

Parameter

Description

request_id

The request ID, which is of the STRING type.

code

The request status code, which is of the STRING type.

message

The detailed status information about the request, which is of the STRING type. Sample command output:

  • success

  • Invalid Input - image data error - base64decode error: base64decode error: incorrect base64 format data

  • Invalid Input - json format error - json parse runtime exception

For more information, see Error codes.

data

The content returned by the model service, which is of the DICT type. For more information about the response parameters, see Data parameters.

Table 3. Data parameters

Parameter

Description

tag_result

The labeling result, which is of the LIST type. The labels are sorted in descending order by score (probability).

The length of the list varies based on the value of tag_top_k.

score

The probability of the label, which is of the FLOAT type. The value is accurate to three decimal places.

tag

The name of the label, which is of the STRING type.

embedding_result

The 512-dimensional features of the image, which are of the STRING type. The features are separated by commas (,).

This parameter is returned only when output_embedding is set to True.

image_height

The height of the image in pixels, which is of the INT type.

image_width

The width of the image in pixels, which is of the INT type.

Labeling AI paintings

You can add multiple labels to various types of images that are generated by Stable Diffusion models to improve the training performance of image generation. The output is a synthesized caption and a score related to the original set of labels.

Important

This model is a custom model. If you want to use the model, provide the AppId to your account manager for technical support.

The following code block shows a sample request:

from ai_service_python_sdk.client.api.ai_service_image_api import AiServiceImageApi

# The URL of the image.
image_url = 'https://your_image_url.jpg'
ai_service_api = AiServiceImageApi(client)
# The name of the custom model.
model_name = 'tagger'
# The response settings of the model.
configure = {'threshold': 0.5, 'tagger_type': "SmilingWolf/wd-v1-4-convnextv2-tagger-v2"}
# The model service to be called.
response = ai_service_api.multi_label_image_v2(image_url, model_name, configure)
# The ID of the request.
request_id = response.request_id
# The status of the request.
code = response.code
# The detailed status information about the request.
message = response.message
# The return value.
data = response.data
# Display the return value.
print(response)

The following table describes the parameters in the preceding command.

Parameter

Parameter description

Data type

Required

image_url

The URL of the image. We recommend that the size of the image is no larger than 5 MB and the resolution of the image is equal to or higher than 240 × 240.

STRING

Yes

model_name

The name of the model. Set the parameter to tagger.

STRING

Yes

configure

The response settings of the model. The default is None. For more information, see Configure parameters.

Dict

Yes

Configure parameters

configure

Parameter

Parameter description

Data type

Required

Example

threshold

The threshold.

FLOAT

No

0.5

tagger_type

The labeling model that is used. Valid values:

  • SmilingWolf/wd-v1-4-convnext-tagger-v2

  • SmilingWolf/wd-v1-4-convnextv2-tagger-v2

  • SmilingWolf/wd-v1-4-vit-tagger-v2

  • SmilingWolf/wd-v1-4-swinv2-tagger-v2

  • RAM

  • BLIP

  • GIT

STR

No

SmilingWolf/wd-v1-4-convnextv2-tagger-v2

The following result is returned:

{
    "data": {
    	"caption": "sensitive, 1girl, solo, long hair, looking at viewer, smile, black hair, brown eyes, scarf, lips, realistic",
    	"probs": [1.1920928955078125e-07, ............, 3.5762786865234375e-07, 4.76837158203125e-07],
	}
    "request_id": "12***", 
    "code": "OK", 
    "message": "success"
}

Parameter

Parameter description

Data type

request_id

The ID of the request.

STRING

code

The request status code.

STRING

message

The detailed status information about the request. Examples:

  • success

  • Invalid Input - image data error - base64decode error: incorrect base64 format data

  • Invalid Input - json format error - json parse runtime exception

STRING

data

The content returned by the model service. For more information, see Data parameters.

Dict

Data parameters

Parameters

Parameter description

Data type

caption

The labeling results, which are separated by commas (,).

STR

probs

The score that is obtained by using the specified labeling model.

List

Portrait figure modification

You can modify the figure in the image. You can upload a portrait image and modify the figure to be larger or slimmer by adjusting the degree parameter. For example, you can set the degree parameter to a value that is greater than 0 to slim the figure.

Important

This model is a custom model. If you want to use the model, provide the AppId to your account manager for technical support.

The following code block shows a sample request:

from ai_service_python_sdk.client.api.ai_service_image_api import AiServiceImageApi

# The URL of the image.
image_url = 'https://your_image_url.jpg'
ai_service_api = AiServiceImageApi(client)
# The name of the model that you want to use.
model_name = 'body_reshaping'
# If you want to enlarge the figure, we recommend that you set the degree parameter to a value between -2.5 and -1.5.
configure = {'degree': -1.5}
# The model service to be called.
response = ai_service_api.multi_label_image_v2(image_url, model_name, configure)
# The ID of the request.
request_id = response.request_id
# The status of the request.
code = response.code
# The detailed status information about the request.
message = response.message
# The return value.
data = response.data
# Display the return value.
print(response)

The following table describes the request parameters.

Parameter

Parameter description

Data type

Required

image_url

The URL of the image. We recommend that the size of the image is no larger than 5 MB and the resolution of the image is equal to or higher than 240 × 240.

STRING

Yes

model_name

The name of the model. Set the value to body_reshaping.

STRING

Yes

configure

The response settings of the model. The default is None. For more information, see Configure parameters.

Dict

Yes

Configure parameters

Parameter

Parameter description

Data type

Required

degree

The degree of figure modification. A value that is greater than 0 indicates to slim the figure. Default value: -1.5.

FLOAT

No

The following result is returned:

{
 "code":"OK",
 "data":"imByev+I3vrryTGN****",
 "message":"success",
 "request_id":"1d4572a1-2800-4213-9e9c-33********"
}

The following table describes the response parameters.

Parameter

Parameter description

Data type

request_id

The ID of the request.

STRING

code

The request status code.

STRING

message

The detailed status information about the request. Examples:

  • success

  • Invalid Input - image data error - base64decode error: base64decode error: incorrect base64 format data

  • Invalid Input - json format error - json parse runtime exception

STRING

data

The Base64-encoded image that is returned by the model service.

STRING

Watermark removal

You can remove the watermarks of an image.

Important

This model is a custom model. If you want to use the model, provide the AppId to your account manager for technical support.

The following code block shows a sample request:

from ai_service_python_sdk.client.api.ai_service_image_api import AiServiceImageApi

# The URL of the image.
image_url = 'https://your_image_url.jpg'
ai_service_api = AiServiceImageApi(client)
# The name of the custom model.
model_name = 'watermark_remover'
# Specify an empty dictionary.
configure = {}
# The model service to be called.
response = ai_service_api.multi_label_image_v2(image_url, model_name, configure)
# The ID of the request.
request_id = response.request_id
# The status of the request.
code = response.code
# The detailed status information about the request.
message = response.message
# The return value.
data = response.data
# Display the return value.
print(response)

The following table describes the parameters.

Parameter

Parameter description

Data type

Required

image_url

The URL of the image. We recommend that the size of the image is no larger than 5 MB and the resolution of the image is equal to or higher than 240 × 240.

STRING

Yes

model_name

The name of the model. Set the value to watermark_remover.

STRING

Yes

configure

The response settings of the model. The default is None.

Dict

Yes

The following result is returned:

{
 "code":"OK",
 "data":"imByev+I3vr****",
 "message":"success",
 "request_id":"1d4572a****"
}

The following table describes the returned parameters.

Parameter

Parameter description

Data type

request_id

The ID of the request.

STRING

code

The request status code.

STRING

message

The detailed status information about the request. Examples:

  • success

  • Invalid Input - image data error - base64decode error: incorrect base64 format data

  • Invalid Input - json format error - json parse runtime exception

STRING

data

The Base64-encoded image that is returned by the model service.

STRING

Post categorization and labeling

You can label posts that include multi-modal content. For example, you can categorize and label content that consists of texts and images or texts and videos to increase the labeling accuracy. The model service also returns the embedding results of high-dimensional features.

Note

The model is a custom model. If you want to use the model, contact your account manager and provide the AppId for configuration.

The following code block shows a sample request:

from ai_service_python_sdk.client.api.ai_service_video_api import AiServiceVideoApi

ai_service_api = AiServiceVideoApi(client)
# The name of the model that you want to use.
model_name = 'multi_modal_tag'
# The video URL. If no video URL is available, specify an empty string.
video_url = 'https://YOUR_VIDEO.mp4'
# The image URLs. If no image URL is available, specify an empty array.
images = [
    'https://YOUR_IMAGE_0.jpg',
    'https://YOUR_IMAGE_1.jpg',
    'https://YOUR_IMAGE_2.jpg'
]
# Text content.
text ='Text content'
# The response settings of the model.
configure = {
    'output_classification': True, 
    'classification_top_k': 1,
    'output_tag': True,
    'tag_top_k': 5, 
    'output_embedding': True
}
# The model service to be called.
response = ai_service_api.multimodal_classify(video_url, images, text, model_name, configure)
# The ID of the request.
request_id = response.request_id
# The status of the request.
code = response.code
# The detailed status information about the request.
message = response.message
# The content returned by the request.
data = response.data
# job id
job_id = response.data['job_id']
# Display the return value.
print(response)

The following table describes the parameters in the statement.

Parameter

Data type

Required

Description

video_url

STRING

Yes

The video URL. If no video URL is available, specify an empty string.

images

List[STRING]

Yes

The image URLs. If no image URL is available, specify an empty array. Separate multiple image URLs with commas (,).

text

STRING

Yes

The text content, which consists of the title and the introduction.

model_name

STRING

Yes

The name of the model.

configure

Dict

Yes

The additional configuration of the model, which is of the DICT type. For more information, see Configure parameters.

Configure parameters

Parameter

Type

Required

Description

output_classification

BOOL

No

Specifies whether to return the categorization results. Default value: True.

classification_top_k

INT

No

The top K categorization results. The value ranges from 1 to 10. Default value: 1.

output_tag

BOOL

No

Specifies whether to return the labeling results. Default value: True.

tag_top_k

INT

No

The top K labeling results. The value ranges from 1 to 10. Default value: 5.

output_embedding

BOOL

No

Specifies whether to return the embedding results of the 32-dimensional multi-modal content. Default value: True.

The following result is returned:

{'code': 'OK',
 'data': {'job_id': 1134***},
 'message': 'success',
 'request_id': '49b0c9c2-06be-4f30-9b9f-f04baecf4577'}

Obtain the job_id from the preceding returned results and use the following code to query the results.

from ai_service_python_sdk.client.api.ai_service_job_api import AiServiceJobApi

ai_service_job_api = AiServiceJobApi(client)
# Query results.
result = ai_service_job_api.get_async_job(<YOUR-JOB-ID>)
# Display the return value.
print(result)

Replace <YOUR-JOB-ID> with the job_id that you obtained.

The following result is returned:

  • The following result is returned if the job is running:

    {'code': 'OK',
     'data': {'job': {'Result': '',
                      'app_id': 'YOUR_APPID',
                      'create_time': '2023-10-18T14:32:42.619+08:00',
                      'id': 1134206,
                      'message': 'video downloading',
                      'state': 1}},
     'message': 'success',
     'request_id': '49b0c9c2-06be-4f30-9b9f-f04baecf4577'}
  • The following result is returned if the job is completed:

    {
    'code': 'OK',
    'data': {'job': 
                {'Result': 
                            '{"classification_result":[{"class":"tourism","score":0.67}],
                            "duration":15.16,
                            "embedding_result":"0.915,0.882,0.943,0.978,1.027,1.181,1.066,1.029,0.866,0.716,0.628,1.203,0.689,0.533,0.734,1.038,0.98,0.613,0.96,0.88,0.586,0.702,1.515,0.697,0.987,0.699,1.179,4.274,0.757,0.89,0.805,0.901",
                            "tag_result":[{"score":0.56,"tag":"with subtitles"},{"score":0.456,"tag":"aerial filming"},{"score":0.453,"tag":"Tibet"},{"score":0.437,"tag":"transportation"},{"score":0.429,"tag":"machinery"}
                            "video_height":918,
                            "video_width":552}',
                            'app_id': '202211140827340****',
                            'create_time': '2023-10-18T14:32:42.619+08:00',
                            'id': 113****,
                            'message': 'success',
                            'model_id': 0,
                            'request_id': '',
                            'state': 2,
                            'type': 'Video'
                }
            },
    'message': 'success',
    'request_id': '139a40ad-fdf6-45de-b760-161c52ff3f3c'
    }

The following table describes the response parameters.

Parameter

Restriction

Parameter description

request_id

STRING

The ID of the request.

code

STRING

The request status code.

message

STRING

The detailed status information about the request. Possible values:

  • success

  • Invalid Input - video data error - load video failed

  • Invalid Input - json format error - json parse runtime exception

data

Dict

The content returned by the model service. For more information, see Data parameters.

Data parameters

Parameter

Data type

Description

id

INT

The ID of the job (job_id).

app_id

STRING

The AppId.

state

INT

The status code of the job. Possible values:

  • 0: The job is initializing.

  • 1: The job is running.

  • 2: The job is completed.

  • 3: The job failed.

message

STRING

The execution information of the job. Possible values:

  • success

  • video downloading

create_time

STRING

The time when the job was created.

Result

STRING

The result returned by the model service. For more information about the response parameters, see Result parameters.

Result parameters

Parameter

Data type

Description

classification_result

List

The classification result. The classes are sorted in descending order by score. The length of the list varies based on

classification_top_k. Each item is of the DICT type.

This parameter is returned only when you set output_classification to True.

tag_result

List

The labeling result. The labels are sorted in descending order by score. The length of the list varies based on tag_top_k. Each item is of the DICT type.

This parameter is returned only when you set output_tag to True.

score

FLOAT

The probability of the label, which is accurate to three decimal places.

class

STRING

The name of the class.

tag

STRING

The name of the label.

embedding_result

STRING

The 32-dimensional multi-modal features. Separate multiple features with commas (,).

video_height

INT

The height of the video in pixels. An empty string is returned if you specify an empty string for the video_url parameter.

video_width

INT

The width of the video in pixels. An empty string is returned if you specify an empty string for the video_url parameter.

duration

FLOAT

The duration of the video. An empty string is returned if you specify an empty string for the video_url parameter.

Video quality evaluation

You can evaluate the quality of a video. The model service returns a floating-point number from 0 to 100. You can use one of the following methods to call the model service:

  • Classify a single video. The following code block shows a sample request:

    from ai_service_python_sdk.client.api.ai_service_video_api import AiServiceVideoApi
    
    video_url = 'https://your_video_url.mp4'
    ai_service_api = AiServiceVideoApi(client)
    # The name of the custom model. 
    model_name = ''
    # The title of the video. 
    video_title = ''
    # The response settings of the model. 
    configure = {}
    # The model service to be called. 
    response = ai_service_api.video_qa(video_url, model_name, video_title, configure)
    # The ID of the request. 
    request_id = response.request_id
    # The status of the request. 
    code = response.code
    # The detailed status information about the request. 
    message = response.message
    # The content returned by the model service. 
    data = response.data
    # The ID of the job. 
    job_id = response.data['job_id']
    # Display the return value. 
    print(response)

    The following table describes the parameters in the preceding statement.

    Parameter

    Data type

    Required

    Description

    video_url

    STRING

    Yes

    The URL of the video. We recommend that the size of the video is no larger than 100 MB and the resolution of the video is equal to or higher than 320 × 240.

    video_title

    STRING

    Yes

    The title of the video. Specify an empty string because the model service does not require you to specify a video title.

    model_name

    STRING

    Yes

    By default, an empty string is specified to call a general-purpose model.

    If the general-purpose model cannot meet your business requirements, contact your account manager to obtain a custom model. You can specify the name of a custom model to call the custom model.

    configure

    Dict

    Yes

    The response settings of the model. Specify an empty dictionary for the model service.

    The following command output is returned:

    {'code': 'OK',
     'data': {'job_id': 10**},
     'message': 'success',
     'request_id': '228235a6-db4f-466a-b20b-a4453f81****'}

    Obtain job_id from the preceding returned response and specify the job ID in the following code to query the result.

    from ai_service_python_sdk.client.api.ai_service_job_api import AiServiceJobApi
    
    ai_service_job_api = AiServiceJobApi(client)
    # Query the result. 
    result = ai_service_job_api.get_async_job(<YOUR-JOB-ID>)
    # Display the return value. 
    print(result)

    Replace <YOUR-JOB-ID> with the value of job_id returned by the model service, which is of the INT type.

    • The following result is returned if the job is running:

      {'code': 'OK',
       'data': {'job': {'Result': '',
                        'app_id': '202209070402410****',
                        'create_time': '2022-09-07T14:03:33.8+08:00',
                        'id': 1012,
                        'message': 'video downloading',
                        'state': 1}},
       'message': 'success',
       'request_id': '5259ec57-e55c-4772-b01e-3deafd74****'}
    • The following result is returned if the job is completed:

      {'code': 'OK',
       'data': {'job': {'Result': '{"avg_fps":30,"duration":14,"height":1920,"single_img":0.74072265625,"video_score":20.57750291278764,"width":1080}',
                        'app_id': '202209290847310****',
                        'create_time': '2022-10-12T10:51:30.015+08:00',
                        'id': 1012,
                        'message': 'success',
                        'state': 2}},
       'message': 'success',
       'request_id': '0b4f658f-cd96-4606-8d81-d2fe0a6d****'}

    The following table describes the response parameters.

    Parameter

    Data type

    Description

    request_id

    STRING

    The request ID, which is of the STRING type.

    code

    STRING

    The request status code, which is of the STRING type.

    message

    STRING

    The detailed status information about the request, which is of the STRING type. Sample command output:

    • success

    • Invalid Input - video data error - load video failed

    • Invalid Input - json format error - json parse runtime exception

    For more information, see Error codes.

    data

    Dict

    The content returned by the model service, which is of the DICT type. For more information about the response parameters, see Data parameters.

    Table 4. Data parameters

    Parameter

    Description

    id

    The ID of the job (job_id), which is of the INT type.

    app_id

    The AppId, which is of the INT type.

    state

    The status code of the job, which is of the INT type. Valid values:

    • 0: The job is initializing.

    • 1: The job is running.

    • 2: The job is completed.

    • 3: The job failed.

    message

    The execution information of the job, which is of the STRING type. Valid values:

    • success

    • video downloading

    create_time

    The time when the job was created, which is of the STRING type.

    Result

    The result returned by the model service, which is of the STRING type. For more information, see Result parameters.

    Table 5. Result parameters

    Parameter

    Description

    video_score

    The quality score of the video, which is of the FLOAT type. Valid values: 0 to 100.

    avg_fps

    The average frame rate of the video, which is of the INT type.

    duration

    The duration of the video, which is of the INT type. Unit: seconds.

    single_img

    The difference between the images in the video, which is of the FLOAT type. Valid values: 0 to 1. A smaller value indicates a smaller difference. This means that the video is generated based on static images.

    height

    The height of the video in pixels, which is of the INT type.

    width

    The width of the video in pixels, which is of the INT type.

  • Evaluate the quality of multiple videos in a batch. The following code block shows a sample request:

    from ai_service_python_sdk.client.api.ai_service_video_api import AiServiceVideoApi
    from ai_service_python_sdk.client.models.batch_video_request_videos import BatchVideoRequestVideos
    
    ai_service_api = AiServiceVideoApi(client)
    
    # Create a batch processing request (video URL + video title). 
    videos = [
        BatchVideoRequestVideos('https://your_video_1_url.mp4', ''),
        BatchVideoRequestVideos('https://your_video_2_url.mp4', '')
        ]
    # The name of the custom model. 
    model_name = ''
    # The response settings of the model. 
    configure = {}
    # The model service to be called. 
    response = ai_service_api.batch_video_qa(videos, model_name, configure)
    # The ID of the request. 
    request_id = response.request_id
    # The status of the request. 
    code = response.code
    # The detailed status information about the request. 
    message = response.message
    # The content returned by the model service. 
    data = response.data
    # The ID of the batch job. 
    batch_id = response.data['batch_id']
    # Display the return value. 
    print(response)

    The following command output is returned:

    {'code': 'OK',
     'data': {'batch_id': 14},
     'message': 'success',
     'request_id': '6d5ef1b5-c01f-4e63-9eec-150ddde9****'}

    Obtain batch_id from the preceding returned response and use the batch ID in the following code to query the result:

    from ai_service_python_sdk.client.api.ai_service_job_api import AiServiceJobApi
    
    ai_service_job_api = AiServiceJobApi(client)
    # Query the result. 
    result = ai_service_job_api.get_batch_job_with_id(<YOUR-BATCH-ID>)
    # Display the return value. 
    print(result)

    Replace <YOUR-BATCH-ID> with the value of batch_id returned by the model service, which is of the INT type.

    The following result is returned:

    {'code': 'OK',
     'data': {'jobs': [{'Result': '{"avg_fps":30,"duration":14.633333333333333,"height":1280,"single_img":0.486328125,"video_score":55.3773279862801,"width":720}',
                        'app_id': '202209290847310****',
                        'batch_id': 14,
                        'create_time': '2022-10-26T19:44:47.501+08:00',
                        'id': 44**,
                        'message': 'success',
                        'object': 'https://your_video_1_url.mp4',
                        'state': 2},
                       {'Result': '{"avg_fps":30,"duration":18.2,"height":568,"single_img":0.5791015625,"video_score":67.21071975649835,"width":320}',
                        'app_id': '202209290847310****',
                        'batch_id': 14,
                        'create_time': '2022-10-26T19:44:47.501+08:00',
                        'id': 4484,
                        'message': 'success',
                        'object': 'https://your_video_2_url.mp4',
                        'state': 2}]},
     'message': 'success',
     'request_id': 'e99697ff-7d42-4c0a-a0a0-c6a0c0ba****'}

    The data['jobs'] parameter returned by the batch job is an array. The following table describes the parameters in the array.

    Parameter

    Description

    id

    The ID of the data entry in the batch job, which is of the INT type.

    batch_id

    The ID of the batch job, which is of the INT type.

    app_id

    The AppId, which is of the INT type.

    state

    The status code of the job, which is of the INT type. Valid values:

    • 0: The job is initializing.

    • 1: The job is running.

    • 2: The job is completed.

    • 3: The job failed.

    message

    The execution information of the job, which is of the STRING type. Valid values:

    • success

    • video downloading

    create_time

    The time when the job was created, which is of the STRING type.

    object

    The URL of the video, which is of the STRING type.

    Result

    The result returned by the model service, which is of the STRING type.

    Note

    The Result parameters are the same as the Result parameters returned by the model service that is used to evaluate the quality of a single video.

Video classification by label

You can classify short videos by adding labels to the videos. The mode service can return the class of the videos, the top K labels with the highest probabilities, and the probability of each label. The model service can also return the high-dimensional features of the videos. You can use one of the following methods to classify videos:

  • Classify a single video. The following code block shows a sample request:

    from ai_service_python_sdk.client.api.ai_service_video_api import AiServiceVideoApi
    
    video_url = 'https://your_video_url.mp4'
    ai_service_api = AiServiceVideoApi(client)
    # The name of the custom model. 
    model_name = ''
    # The title of the video. 
    video_title = 'your_video_title'
    # The response settings of the model. 
    configure = {
        'output_classification': True,
        'classification_top_k': 1,
        'output_tag': True,
        'tag_top_k': 5,
        'output_embedding': True
        }
    # The model service to be called. 
    response = ai_service_api.classify_label_video(video_url, video_title, model_name, configure)
    # The ID of the request. 
    request_id = response.request_id
    # The status of the request. 
    code = response.code
    # The detailed status information about the request. 
    message = response.message
    # The content returned by the model service. 
    data = response.data
    # The ID of the job. 
    job_id = response.data['job_id']
    # Display the return value. 
    print(response)

    The following table describes the parameters in the preceding statement.

    Parameter

    Data type

    Required

    Description

    video_url

    STRING

    Yes

    The URL of the video. We recommend that the size of the video is no larger than 100 MB and the resolution of the video is equal to or higher than 320 × 240.

    video_title

    STRING

    Yes

    The title of the video. If the video does not have a title, specify an empty string.

    model_name

    STRING

    Yes

    By default, an empty string is specified to indicate a general-purpose model.

    If the general-purpose model cannot meet your requirements, you can contact the technical support of PAI to obtain a custom model. You can specify the name of a custom model to call the custom model.

    configure

    Dict

    Yes

    The response settings of the model. The default value is None. For more information, see Configure parameters.

    Note

    If configure is set to None, the parameters in the configure list use the default settings.

    Table 6. Configure parameters

    Parameter

    Data type

    Required

    Description

    output_classification

    BOOL

    No

    Specify whether to return the video classification result. Valid values:

    • True (default)

    • False

    classification_top_k

    INT

    No

    The top K classes of the video. Valid values of K: 1 to 10. Default value: 1.

    output_tag

    BOOL

    No

    Specify whether to return the labels of the video. Valid values:

    • True (default)

    • False

    tag_top_k

    INT

    No

    The top K labels of the video. Valid values of K: 1 to 10. Default value: 5.

    output_embedding

    BOOL

    No

    Specify whether to return the 64-dimensional video embedding. Valid values:

    • True (default)

    • False

    The following command output is returned:

    {'code': 'OK',
     'data': {'job_id': 5},
     'message': 'success',
     'request_id': '4f6caecb-e3d6-4997-89fe-26ac7536****'}

    Obtain job_id from the preceding returned response and specify the job ID in the following code to query the result.

    from ai_service_python_sdk.client.api.ai_service_job_api import AiServiceJobApi
    
    ai_service_job_api = AiServiceJobApi(client)
    # Query the result. 
    result = ai_service_job_api.get_async_job(<YOUR-JOB-ID>)
    # Display the return value. 
    print(result)

    Replace <YOUR-JOB-ID> with the value of job_id returned by the model service, which is of the INT type.

    • The following result is returned if the job is running:

      {'code': 'OK',
       'data': {'job': {'Result': '',
                        'app_id': '202209070402410****',
                        'create_time': '2022-09-07T14:03:33.8+08:00',
                        'id': 5,
                        'message': 'video downloading',
                        'state': 1}},
       'message': 'success',
       'request_id': '5259ec57-e55c-4772-b01e-3deafd74****'}
      
    • The following result is returned if the job is completed:

      {'code': 'OK',
       'data': {'job': {'Result':
          '{"classification_result":[
              {"class":"three rural issues","score":0.997}
              ],
          "duration":14.633333333333333,
          "embedding_result":"-0.162,0.483,0.881,0.686,0.319,-1.569,0.266,0.227,-1.352,1.157,-0.784,-1.016,-0.987,-1.214,-0.493,-1.182,0.85,1.473,-1.27,-3.391,3.394,-0.471,-0.434,0.256,2.146,0.787,0.348,1.67,0.229,-0.878,0.299,0.392,-2.161,-0.756,-3.438,-0.158,0.713,-5.812,-1.248,0.253,-1.111,-0.165,2.644,-2.368,-1.636,-0.533,0.468,1.2,-1.431,-1.151,-1.276,-1.954,1.167,-0.938,-0.111,-2.636,-1.248,4.662,1.892,5.485,-1.137,1.589,-1.935,1.018",
          "tag_result":[
              {"score":0.728,"tag":"potted plants"},
              {"score":0.576,"tag":"plants"},
              {"score":0.539,"tag":"planting techniques"},
              {"score":0.466,"tag":"agricultural technology"},
              {"score":0.458,"tag":"planting"}
              ],
          "video_height":1280,
          "video_width":720}\n',
        'app_id': '202209070402410****',
        'create_time': '2022-09-07T14:03:33.8+08:00',
        'id': 5,
        'message': 'success',
        'state': 2}},
       'message': 'success',
       'request_id': '9283066a-f414-40fd-8fd3-0f77c1ef****'}

    The following table describes the response parameters.

    Parameter

    Description

    request_id

    The request ID, which is of the STRING type.

    code

    The request status code, which is of the STRING type.

    message

    The detailed status information about the request, which is of the STRING type. Sample command output:

    • success

    • Invalid Input - video data error - load video failed

    • Invalid Input - json format error - json parse runtime exception

    For more information, see Error codes.

    data

    The content returned by the model service, which is of the DICT type. For more information, see Data ['job'] parameters.

    Table 7. Data ['job'] parameters

    Parameter

    Description

    id

    The ID of the job (job_id), which is of the INT type.

    app_id

    The AppId, which is of the INT type.

    state

    The status code of the job, which is of the INT type. Valid values:

    • 0: The job is initializing.

    • 1: The job is running.

    • 2: The job is completed.

    • 3: The job failed.

    message

    The execution information of the job, which is of the STRING type. Valid values:

    • success

    • video downloading

    create_time

    The time when the job was created, which is of the STRING type.

    Result

    The result returned by the model service, which is of the STRING type. For more information, see Result parameters.

    Table 8. Result parameters

    Parameter

    Description

    classification_result

    The classification result, which is of the LIST type. The classes are sorted in descending order by score (probability).

    The length of the list varies based on classification_top_k.

    This parameter is returned only when output_classification is set to True.

    tag_result

    The labels of the video, which are sorted in descending order by score (probability).

    The length of the list varies based on tag_top_k.

    This parameter is returned only when output_tag is set to True.

    score

    The probability of the label, which is of the FLOAT type. The value is accurate to three decimal places.

    class

    The alias of the class, which is of the STRING type.

    tag

    The name of the label, which is of the STRING type.

    embedding_result

    The 64-dimensional features of the video, which are of the STRING type. The features are separated by commas (,).

    This parameter is returned only when output_embedding is set to True.

    video_height

    The height of the video in pixels, which is of the INT type.

    video_width

    The width of the video in pixels, which is of the INT type.

    duration

    The duration of the video, which is of the FLOAT type.

  • Classify multiple videos in a batch. The following code shows a sample request:

    from ai_service_python_sdk.client.api.ai_service_video_api import AiServiceVideoApi
    from ai_service_python_sdk.client.models.batch_video_request_videos import BatchVideoRequestVideos
    
    ai_service_api = AiServiceVideoApi(client)
    
    # The response settings of the model. 
    configure = {
        'output_classification': True,
        'classification_top_k': 1,
        'output_tag': True,
        'tag_top_k': 5,
        'output_embedding': True
        }
    # Create a batch processing request (video URL + video title). 
    videos = [
        BatchVideoRequestVideos('https://your_video_1_url.mp4', 'your_video_title_1'),
        BatchVideoRequestVideos('https://your_video_2_url.mp4', 'your_video_title_2')
        ]
    # The name of the custom model. 
    model_name = ''
    # The model service to be called. 
    response = ai_service_api.batch_classify_label_video(videos, model_name, configure)
    # The ID of the request. 
    request_id = response.request_id
    # The status of the request. 
    code = response.code
    # The detailed status information about the request. 
    message = response.message
    # The content returned by the model service. 
    data = response.data
    # batch id
    batch_id = response.data['batch_id']
    # Display the return value. 
    print(response)

    The following command output is returned:

    {'code': 'OK',
     'data': {'batch_id': 1},
     'message': 'success',
     'request_id': 'c59eb912-9df4-4afe-8129-23f7b6dc****'}

    Obtain batch_id from the preceding returned response and use the batch ID in the following code to query the result:

    from ai_service_python_sdk.client.api.ai_service_job_api import AiServiceJobApi
    
    ai_service_job_api = AiServiceJobApi(client)
    # Query the result. 
    result = ai_service_job_api.get_batch_job_with_id(<YOUR-BATCH-ID>)
    # Display the return value. 
    print(result)

    Replace <YOUR-BATCH-ID> with the value of batch_id returned by the model service, which is of the INT type.

    The following result is returned:

    {'code': 'OK',
     'data': {'jobs': [{'Result': '{"classification_result":[{"class":"daily life","score":0.998}],"duration":14.633333333333333,"embedding_result":"0.998,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.002","tag_result":[{"score":0.917,"tag":"potted plants"},{"score":0.777,"tag":"plants"},{"score":0.693,"tag":"gardening"},{"score":0.58,"tag":"happy trees"},{"score":0.509,"tag":"titles"}],"video_height":1280,"video_width":720}',
                        'app_id': '202209290847310****',
                        'batch_id': 13,
                        'create_time': '2022-10-26T18:13:20.928+08:00',
                        'id': 4090,
                        'message': 'success',
                        'object': 'https://your_video_1_url.mp4',
                        'state': 2},
                       {'Result': '{"classification_result":[{"class":"sports","score":0.998}],"duration":18.2,"embedding_result":"0.0,0.0,0.0,0.998,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.001,0.0,0.0,0.0,0.0,0.001,0.0,0.0,0.0,0.0","tag_result":[{"score":0.981,"tag":"National Games"},{"score":0.901,"tag":"table tennis"},{"score":0.798,"tag":"sports"},{"score":0.583,"tag":"title"},{"score":0.498,"tag":"table tennis games"}],"video_height":568,"video_width":320}',
                        'app_id': '202209290847310****',
                        'batch_id': 13,
                        'create_time': '2022-10-26T18:13:20.928+08:00',
                        'id': 4091,
                        'message': 'success',
                        'object': 'https://your_video_2_url.mp4',
                        'state': 2}]},
     'message': 'success',
     'request_id': '73e4b48e-42b8-4347-9cb4-7a49c1f8****'}

    The data['jobs'] parameter returned by the batch job is an array. The following table describes the parameters in the array.

    Parameter

    Description

    id

    The ID of the data entry in the batch job, which is of the INT type.

    batch_id

    The ID of the batch job, which is of the INT type.

    app_id

    The AppId, which is of the INT type.

    state

    The status code of the job, which is of the INT type. Valid values:

    • 0: The job is initializing.

    • 1: The job is running.

    • 2: The job is completed.

    • 3: The job failed.

    message

    The execution information of the job, which is of the STRING type. Valid values:

    • success

    • video downloading

    create_time

    The time when the job was created, which is of the STRING type.

    object

    The URL of the video, which is of the STRING type.

    Result

    The result returned by the model service, which is of the STRING type.

    Note

    The Result parameters are the same as the Result parameters returned by the model service that is used to evaluate the quality of a single video.

Error codes

The following table describes the error codes returned by multimedia analysis SDK for Python.

Image analysis service

Request errors

HTTP status code

code

message

Description

400

PARAMETER_ERROR

image is empty

No image is specified in the request.

note found appid

The AppId is invalid.

Unmarshal type error: expected=string, got=number, field=image, offset=42, internal=json: cannot unmarshal number into Go struct field ImageRequest.image of type string

The data type of the image parameter is invalid.

401

PARAMETER_ERROR

sign error

The token is invalid.

404

PARAMETER_ERROR

model not found

The requested model service is not deployed.

422

SEVER_ERROR

Refer to the returned error message.

The model service encounters an error. Refer to the content of the returned message.

Video analysis service

  • Request errors

    HTTP status code

    code

    message

    Description

    400

    PARAMETER_ERROR

    not found appid

    The AppId is invalid.

    401

    PARAMETER_ERROR

    sign error

    The token is invalid.

    404

    PARAMETER_ERROR

    model not found

    The requested model service is not deployed.

  • Result query errors

    state

    message

    Description

    3

    failed to download video, err=fetch resource failed\tcode = 404

    The system failed to download the video from the specified URL.

    3

    Refer to the returned error message.

    The model service encounters an error. Refer to the content of the returned message.