Build Multi-Turn Semantic Search with ContextualRetrieval - IMM - Alibaba Cloud - Intelligent Media Management

Operation description

Precautions

Make sure that you fully understand the billing methods and pricing of Intelligent Media Management before you call this operation.
Before you call this operation, make sure that you have indexed files into a dataset by using the binding method (CreateBinding) or the active indexing method (IndexFileMeta or BatchIndexFileMeta).
The returned results are for reference only. Based on different workflow template configurations, the categories and content of the obtained file metadata may differ from the examples. If you have any questions, search for the DingTalk group number 21714099 in DingTalk to join the group and provide feedback.

Limits

The maximum length of the conversation history is 100, including user messages and assistant messages.
The length of each message cannot exceed 1,000 Chinese characters.

Try it now

Try this API in OpenAPI Explorer, no manual signing needed. Successful calls auto-generate SDK code matching your parameters. Download it with built-in credential security for local usage.

Test

RAM authorization

The table below describes the authorization required to call this API. You can define it in a Resource Access Management (RAM) policy. The table's columns are detailed below:

Action: The actions can be used in the Action element of RAM permission policy statements to grant permissions to perform the operation.
API: The API that you can call to perform the action.
Access level: The predefined level of access granted for each API. Valid values: create, list, get, update, and delete.
Resource type: The type of the resource that supports authorization to perform the action. It indicates if the action supports resource-level permission. The specified resource must be compatible with the action. Otherwise, the policy will be ineffective.
- For APIs with resource-level permissions, required resource types are marked with an asterisk (*). Specify the corresponding Alibaba Cloud Resource Name (ARN) in the Resource element of the policy.
- For APIs without resource-level permissions, it is shown as All Resources. Use an asterisk (*) in the Resource element of the policy.
Condition key: The condition keys defined by the service. The key allows for granular control, applying to either actions alone or actions associated with specific resources. In addition to service-specific condition keys, Alibaba Cloud provides a set of common condition keys applicable across all RAM-supported services.
Dependent action: The dependent actions required to run the action. To complete the action, the RAM user or the RAM role must have the permissions to perform all dependent actions.

Action

Access level

Resource type

Condition key

Dependent action

imm:ContextualRetrieval

none

*All Resource

*

None

Request parameters

Parameter	Type	Required	Description	Example
ProjectName	string	Yes	The name of the project. For more information about how to obtain the project name, see Create a project.	test-project
DatasetName	string	Yes	The dataset used for retrieval.	test-dataset
Messages	array	Yes	The conversation history and tool calling history. The latest message is at the end (index n-1), and the oldest message is at the beginning (index 0). The messages must be in user-assistant pairs, with a total count of 2*n+1, and the length of the latest question cannot exceed 1,000 characters. The conversation history is limited to 100 messages.
	ContextualMessage	No	The list of conversation messages.
RecallOnly	boolean	No	Specifies whether to enable only the recall process (embedding search). If this parameter is set to true, the returned data is not reranked, which allows you to customize the reranking process. Default value: false.	false
SmartClusterIds	array	No	The list of smart cluster IDs, which are used to retrieve files within specific smart clusters.
	string	No	The ID of the smart cluster.	SmartCluster-01686226-be59-4391-a1ac-b7b073455de3

Response elements

Element	Type	Description	Example
	object	Schema of Response
RequestId	string	The ID of the request.	6E93D6C9-5AC0-49F9-914D-E02678D3****
Results	array	The list of retrieved files. The document-related structural content is included in File.Elements.
	File	The file information in the retrieval results.

Examples

Success response

JSON format

{
  "RequestId": "6E93D6C9-5AC0-49F9-914D-E02678D3****",
  "Results": [
    {
      "OwnerId": "102321002467****",
      "ProjectName": "test-project",
      "DatasetName": "test-dataset",
      "ObjectType": "file",
      "ObjectId": "75d5de2c50754e3dadd5c35dbca5f9949369e37eb342a73821f690c94c36c7f7",
      "UpdateTime": "2021-06-29T14:50:13.011643661+08:00",
      "CreateTime": "2021-06-29T14:50:13.011643661+08:00",
      "URI": "oss://test-bucket/test-object.jpg",
      "OSSURI": "oss://examplebucket/sampleobject.jpg",
      "Filename": "sampleobject.jpg",
      "MediaType": "image",
      "ContentType": "image/jpeg",
      "Size": 1000,
      "FileHash": "1d9c280a7c4f67f7ef873e28449dbe17",
      "FileModifiedTime": "2021-06-29T14:50:13.011643661+08:00",
      "FileCreateTime": "2021-06-29T14:50:13.011643661+08:00",
      "FileAccessTime": "2021-06-29T14:50:13.011643661+08:00",
      "ProduceTime": "2021-06-29T14:50:13.011643661+08:00",
      "LatLong": "30.134390,120.074997",
      "Timezone": "\"\"",
      "Addresses": [
        {
          "Language": "zh-hans",
          "AddressLine": "中国浙江省杭州市余杭区文一西路969号",
          "Country": "中国",
          "Province": "浙江省",
          "City": "杭州市",
          "District": "余杭区",
          "Township": "文一西路"
        }
      ],
      "TravelClusterId": "-",
      "Orientation": 0,
      "Figures": [
        {
          "FigureId": "868a9e74-cde5-4c7a-9013-28bb984****",
          "FigureConfidence": 1,
          "FigureClusterId": "Cluster-dbe72fec-b84c-4ab6-885b-3678e64****",
          "FigureClusterConfidence": 1,
          "FigureType": "face",
          "Age": 29,
          "AgeSD": 5,
          "Gender": "female",
          "GenderConfidence": 1,
          "Emotion": "happiness",
          "EmotionConfidence": 0.70784568786621,
          "FaceQuality": 0.960875928401947,
          "Boundary": {
            "Width": 200,
            "Height": 300,
            "Left": 10,
            "Top": 30,
            "Polygon": [
              {
                "X": 10,
                "Y": 10
              }
            ]
          },
          "Mouth": "close",
          "MouthConfidence": 1,
          "Beard": "none",
          "BeardConfidence": 1,
          "Hat": "none",
          "HatConfidence": 1,
          "Mask": "none",
          "MaskConfidence": 1,
          "Glasses": "none",
          "GlassesConfidence": 0.8,
          "Sharpness": 0.7,
          "Attractive": 0.96,
          "HeadPose": {
            "Pitch": 18.385589599609375,
            "Roll": 4.204030513763428,
            "Yaw": 2.4945924282073975
          }
        }
      ],
      "FigureCount": 10,
      "Labels": [
        {
          "Language": "zh-Hans",
          "LabelName": "椅子",
          "LabelLevel": 1,
          "LabelConfidence": 0.9891784601980591,
          "ParentLabelName": "家具",
          "CentricScore": 0.7319999933242798,
          "LabelAlias": "座椅",
          "Clips": [
            {
              "TimeRange": [
                100
              ]
            }
          ]
        }
      ],
      "Title": "test",
      "ImageWidth": 270,
      "ImageHeight": 500,
      "EXIF": "{\"Compression\":{\"value\":\"6\"},\"DateTime\":{\"value\":\"2020:08:19 17:11:11\"}}",
      "ImageScore": {
        "OverallQualityScore": 0.736
      },
      "CroppingSuggestions": [
        {
          "AspectRatio": "2:3",
          "Confidence": 0.7079545259475708,
          "Boundary": {
            "Width": 200,
            "Height": 300,
            "Left": 10,
            "Top": 30,
            "Polygon": [
              {
                "X": 10,
                "Y": 10
              }
            ]
          }
        }
      ],
      "OCRContents": [
        {
          "Language": "zh-hans",
          "Contents": "欢迎使用智能媒体管理",
          "Confidence": 0.8254936695098877,
          "Boundary": {
            "Width": 200,
            "Height": 300,
            "Left": 10,
            "Top": 30,
            "Polygon": [
              {
                "X": 10,
                "Y": 10
              }
            ]
          }
        }
      ],
      "VideoWidth": 1080,
      "VideoHeight": 1920,
      "VideoStreams": [
        {
          "Index": 0,
          "Language": "en",
          "CodecName": "h264",
          "CodecLongName": "H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10",
          "Profile": "High",
          "CodecTimeBase": "1373627/83160000",
          "CodecTagString": "avc1",
          "CodecTag": "0x31637661",
          "Width": 1280,
          "Height": 720,
          "HasBFrames": 2,
          "SampleAspectRatio": "1:1",
          "DisplayAspectRatio": "16:9",
          "PixelFormat": "yuv420p",
          "Level": 40,
          "FrameRate": "25/1",
          "AverageFrameRate": "25/1",
          "TimeBase": "1/12800",
          "StartTime": 0,
          "Duration": 22.88,
          "Bitrate": 5407765,
          "FrameCount": 572,
          "Rotate": "90",
          "BitDepth": 8,
          "ColorSpace": "bt709",
          "ColorRange": "tv",
          "ColorTransfer": "bt709",
          "ColorPrimaries": "bt709"
        }
      ],
      "Subtitles": [
        {
          "Index": 2,
          "Language": "en",
          "CodecName": "mov_text",
          "CodecLongName": "MOV text",
          "CodecTagString": "tx3g",
          "CodecTag": "0x67337874",
          "StartTime": 0,
          "Duration": 71.378,
          "Bitrate": 74,
          "Content": "欢迎",
          "Width": 600,
          "Height": 30
        }
      ],
      "AudioStreams": [
        {
          "Index": 1,
          "Language": "en",
          "CodecName": "aac",
          "CodecLongName": "AAC (Advanced Audio Coding)",
          "CodecTimeBase": "1/44100",
          "CodecTagString": "mp4a",
          "CodecTag": "0x6134706d",
          "TimeBase": "1/48000",
          "StartTime": 0.0235,
          "Duration": 3.690667,
          "Bitrate": 320087,
          "FrameCount": 173,
          "Lyric": "test",
          "SampleFormat": "fltp",
          "SampleRate": 48000,
          "Channels": 2,
          "ChannelLayout": "stereo"
        }
      ],
      "Artist": "Jane",
      "AlbumArtist": "Jane",
      "AudioCovers": [
        {
          "ImageWidth": 500,
          "ImageHeight": 820,
          "EXIF": "{\"FileSize\":{\"value\":\"29304\"},\"Format\":{\"value\":\"jpg\"}}",
          "ImageScore": {
            "OverallQualityScore": 0.736
          },
          "CroppingSuggestions": [
            {
              "AspectRatio": "2:3",
              "Confidence": 0.7079545259475708
            }
          ],
          "OCRContents": [
            {
              "Language": "zh-hans",
              "Contents": "欢迎使用智能媒体管理",
              "Confidence": 0.8254936695098877
            }
          ]
        }
      ],
      "Composer": "Jane",
      "Performer": "Jane",
      "Language": "eng",
      "Album": "FirstAlbum",
      "PageCount": 5,
      "ETag": "\"1D9C280A7C4F67F7EF873E28449****\"",
      "CacheControl": "no-cache",
      "ContentDisposition": "attachment; filename =test.jpg",
      "ContentEncoding": "UTF-8",
      "ContentLanguage": "zh-CN",
      "AccessControlAllowOrigin": "https://aliyundoc.com",
      "AccessControlRequestMethod": "PUT",
      "ServerSideEncryptionCustomerAlgorithm": "SM4",
      "ServerSideEncryption": "AES256",
      "ServerSideDataEncryption": "SM4",
      "ServerSideEncryptionKeyId": "9468da86-3509-4f8d-a61e-6eab1eac****",
      "OSSStorageClass": "Standard",
      "OSSCRC64": "559890638950338001",
      "ObjectACL": "default",
      "ContentMd5": "HZwoCnxPZ/fvhz4oRJ2+Fw==",
      "OSSUserMeta": {
        "key": "val"
      },
      "OSSTaggingCount": 2,
      "OSSTagging": {
        "key": "val"
      },
      "OSSExpiration": "2120-01-01T12:00:00.000Z",
      "OSSVersionId": "CAEQNhiBgMDJgZCA0BYiIDc4MGZjZGI2OTBjOTRmNTE5NmU5NmFhZjhjYmY0****",
      "OSSDeleteMarker": "CAEQMhiBgIDXiaaB0BYiIGQzYmRkZGUxMTM1ZDRjOTZhNjk4YjRjMTAyZjhl****",
      "OSSObjectType": "Normal",
      "CustomId": "\t\nmember-image-id-0001",
      "CustomLabels": {
        "MemberName": "Tim",
        "Enabled": "True",
        "ItemCount": "10"
      },
      "StreamCount": 1,
      "ProgramCount": 1,
      "FormatName": "mov",
      "FormatLongName": "QuickTime / MOV",
      "StartTime": 0,
      "Bitrate": 13091201,
      "Duration": 15.263,
      "SemanticTypes": [
        "content"
      ],
      "Elements": [
        {
          "ElementContents": [
            {
              "Type": "text",
              "Content": "文本片段",
              "URL": "http://aliyun.com",
              "TimeRange": [
                500
              ]
            }
          ],
          "ObjectId": "id1",
          "ElementType": "title - 标题； image - 图片 ； table - 表格 ； narrative-text - 正文",
          "SemanticSimilarity": 0.8,
          "ElementRelations": [
            {
              "Type": "child，parent",
              "ObjectId": "id1"
            }
          ]
        }
      ],
      "SceneElements": [
        {
          "TimeRange": [
            100
          ],
          "FrameTimes": [
            30
          ],
          "VideoStreamIndex": 0,
          "Labels": [
            {
              "Language": "zh-Hans",
              "LabelName": "椅子",
              "LabelLevel": 1,
              "LabelConfidence": 0.9891784601980591,
              "ParentLabelName": "家具",
              "CentricScore": 0.7319999933242798,
              "LabelAlias": "座椅",
              "Clips": [
                {
                  "TimeRange": [
                    100
                  ]
                }
              ]
            }
          ]
        }
      ],
      "OCRTexts": "阿里云IMM",
      "Reason": "[InternalError] The request has been failed due to some unknown error. status: 500, requestId: CC5ACFBD-BB7A-496D-A9D6-****",
      "ObjectStatus": "Indexed",
      "Insights": {
        "Video": {
          "Description": "无。",
          "Caption": "视频中展示了两个不同场景：一个是静止的白色盘子、黑色瓶子和透明玻璃杯，另一个是手拿着标有“YEZOLU”的洗发水瓶在浴室中缓慢上移。"
        },
        "Image": {
          "Caption": "无。",
          "Description": "图片中有一人，穿着深色西装外套，内搭白色衬衫。背景为渐变的浅蓝色至灰色。"
        }
      }
    }
  ]
}

Error codes

See Error Codes for a complete list.

Release notes

See Release Notes for a complete list.