Automate Face Clustering and Retrieve Matched Results - Intelligent Media Management

Face clustering groups images that contain similar faces across a dataset. Each cluster represents one person, so you can find all images of a specific person from a single cluster.

Common use cases:

Cloud drive face albums -- Automatically organize photos by person to generate personalized albums.
Home surveillance -- Face clustering records faces of family members. When the face of a stranger appears, face clustering fails and sends an alert. This helps you identify and handle dangerous personnel and events at the earliest opportunity and ensure the safety of your family members.
Retail customer management -- Deduplicate customer images for accurate traffic counts, then analyze purchase behavior for targeted marketing.

How it works

Face clustering follows a three-step API workflow:

1. Index images         CreateBinding, IndexFileMeta, or BatchIndexFileMeta
        |
        v
2. Cluster faces        CreateFigureClusteringTask
        |
        v
3. Query results        QueryFigureClusters  -->  SimpleQuery (images per cluster)

Index images into a dataset. Use CreateBinding for automatic binding, or IndexFileMeta / BatchIndexFileMeta for manual indexing. Create the dataset with CreateDataset.
Run face clustering. Call CreateFigureClusteringTask to group faces in the dataset. This operation is incremental -- it only processes new images since the last run.
Query clusters and images. Use QueryFigureClusters to list all clusters, then use SimpleQuery to retrieve all images in a specific cluster.

Prerequisites

Before you begin, make sure that:

Images are indexed into a dataset via CreateBinding, IndexFileMeta, or BatchIndexFileMeta
The dataset contains at least 3 images of the same person
At least 3 of those images meet the high-definition face requirements listed below

High-definition face requirements

Criterion	Threshold
Face area	Greater than 75 x 75 pixels
HeadPose	Absolute value of each element (Pitch, Roll, Yaw) less than 30
FaceQuality	Greater than 0.8

Note

Call GetFileMeta to check the face angle and quality values for indexed images.

After a cluster is created, faces that do not meet these thresholds may still be added to the same cluster. For details, see FAQ about image management.

Create a face clustering task

Call CreateFigureClusteringTask to cluster faces across the images indexed in a dataset. This operation generates clusters only -- it does not modify the original images.

Important

Task information is retained for 7 days after the task starts. After 7 days, the task information is no longer available. Use one of the following methods to retrieve task results:

Call the GetTask or ListTasks operation.
Subscribe to a Simple Message Queue (formerly MNS) queue in the same region as the IMM project.
Create an ApsaraMQ for RocketMQ 4.0 instance with a topic and group in the same region.
Use EventBridge in the same region to receive task notifications. For message format details, see Asynchronous message examples.

Request example

{
    "ProjectName": "test-project",
    "DatasetName": "test-dataset"
}

Response example

{
    "TaskId": "CreateFigureClusteringTask-ba5784b8-f61e-485d-8ea0-****",
    "RequestId": "42F4F8FD-006D-0EF0-8F2A-****",
    "EventId": "140-1L5dh6eSUErqdxV1ZvJ****"
}

A successful response includes a TaskId confirming the clustering task was created.

Sample code (Python)

# -*- coding: utf-8 -*-

import os
from alibabacloud_imm20200930.client import Client as imm20200930Client
from alibabacloud_tea_openapi import models as open_api_models
from alibabacloud_imm20200930 import models as imm_20200930_models
from alibabacloud_tea_util import models as util_models
from alibabacloud_tea_util.client import Client as UtilClient


class Sample:
    def __init__(self):
        pass

    @staticmethod
    def create_client(
        access_key_id: str,
        access_key_secret: str,
    ) -> imm20200930Client:
        """
        Initialize the IMM client with an AccessKey pair.
        @param access_key_id:
        @param access_key_secret:
        @return: Client
        @throws Exception
        """
        config = open_api_models.Config(
            access_key_id=access_key_id,
            access_key_secret=access_key_secret
        )
        # Set the endpoint. Replace cn-beijing with your region.
        config.endpoint = f'imm.cn-beijing.aliyuncs.com'
        return imm20200930Client(config)

    @staticmethod
    def main() -> None:
        # Read AccessKey pair from environment variables.
        # Use a RAM user instead of the Alibaba Cloud account for security.
        # For details, see https://www.alibabacloud.com/help/document_detail/2361894.html
        imm_access_key_id = os.getenv("AccessKeyId")
        imm_access_key_secret = os.getenv("AccessKeySecret")
        # Initialize the client.
        client = Sample.create_client(imm_access_key_id, imm_access_key_secret)
        # Build the clustering request.
        create_figure_clustering_task_request = imm_20200930_models.CreateFigureClusteringTaskRequest(
            # IMM project name
            project_name='test-project',
            # Dataset name
            dataset_name='test-dataset'
        )
        runtime = util_models.RuntimeOptions()
        try:
            response = client.create_figure_clustering_task_with_options(
                create_figure_clustering_task_request, runtime)
            print(response.body.to_map())
        except Exception as error:
            UtilClient.assert_as_string(error.message)
            print(error)


if __name__ == '__main__':
    Sample.main()

Query face clusters

After clustering completes, call QueryFigureClusters to list all clusters in a dataset. The response includes the following metadata for each cluster:

Field	Description
`ObjectId`	Cluster ID
`FaceCount`	Number of faces in the cluster
`ImageCount`	Number of images in the cluster
`Gender`	Estimated gender
`AverageAge`, `MinAge`, `MaxAge`	Age statistics
`Cover`	Cover image with detailed face attributes

Request example

{
    "ProjectName": "test-project",
    "DatasetName": "test-dataset"
}

Response example

{
    "FigureClusters": [
        {
            "AverageAge": 27.125,
            "Cover": {
                "Addresses": [],
                "AudioCovers": [],
                "AudioStreams": [],
                "CroppingSuggestions": [],
                "Figures": [
                    {
                        "Attractive": 0.9980000257492065,
                        "Beard": "none",
                        "BeardConfidence": 0.9959999918937683,
                        "Boundary": {
                            "Height": 270,
                            "Left": 573,
                            "Top": 104,
                            "Width": 202
                        },
                        "FaceQuality": 1.0,
                        "FigureId": "d7365ab8-1378-4bec-83cb-eccad8d11e0b",
                        "FigureType": "face",
                        "Glasses": "none",
                        "GlassesConfidence": 0.9990000128746033,
                        "Hat": "none",
                        "HatConfidence": 1.0,
                        "HeadPose": {
                            "Pitch": -0.7369999885559082,
                            "Roll": 2.5399999618530273,
                            "Yaw": 9.138999938964844
                        },
                        "Mask": "none",
                        "MaskConfidence": 0.7269999980926514,
                        "Mouth": "open",
                        "MouthConfidence": 0.9959999918937683,
                        "Sharpness": 1.0
                    }
                ],
                "ImageHeight": 683,
                "ImageWidth": 1024,
                "Labels": [],
                "OCRContents": [],
                "ObjectId": "170ffdeb36cec846f4214c78a0f3a0d4b7e37d0305370216ae780f7b8c72f871",
                "Subtitles": [],
                "URI": "oss://bucket1/photos/2.jpg",
                "VideoStreams": []
            },
            "CreateTime": "2022-07-12T16:41:19.336825716+08:00",
            "DatasetName": "dataset1",
            "FaceCount": 16,
            "Gender": "female",
            "ImageCount": 16,
            "MaxAge": 30.0,
            "MinAge": 23.0,
            "ObjectId": "Cluster-7bdbcedb-bd79-42e7-a1e2-b29a48532bd6",
            "ObjectType": "figure-cluster",
            "OwnerId": "*****",
            "ProjectName": "test-project",
            "UpdateTime": "2022-09-19T17:08:59.374781532+08:00",
            "VideoCount": 0
        },
        {
            "AverageAge": 24.200000762939453,
            "Cover": {
                "Addresses": [],
                "AudioCovers": [],
                "AudioStreams": [],
                "CroppingSuggestions": [],
                "Figures": [
                    {
                        "Attractive": 0.9990000128746033,
                        "Beard": "none",
                        "BeardConfidence": 0.9990000128746033,
                        "Boundary": {
                            "Height": 266,
                            "Left": 301,
                            "Top": 218,
                            "Width": 196
                        },
                        "FaceQuality": 0.8859999775886536,
                        "FigureId": "f58bbdce-f3d1-4674-be6b-43d4b47c08e1",
                        "FigureType": "face",
                        "Glasses": "none",
                        "GlassesConfidence": 1.0,
                        "Hat": "none",
                        "HatConfidence": 1.0,
                        "HeadPose": {
                            "Pitch": 13.963000297546387,
                            "Roll": -12.21399974822998,
                            "Yaw": -6.2210001945495605
                        },
                        "Mask": "none",
                        "MaskConfidence": 0.7490000128746033,
                        "Mouth": "open",
                        "MouthConfidence": 0.9940000176429749,
                        "Sharpness": 1.0
                    }
                ],
                "ImageHeight": 1024,
                "ImageWidth": 683,
                "Labels": [],
                "OCRContents": [],
                "ObjectId": "b9c80e51aa95072413e2a0a6e5262644bc3cba14a4754f54f3fa9850c4d244f1",
                "Subtitles": [],
                "URI": "oss://bucket1/photos/11.jpg",
                "VideoStreams": []
            },
            "CreateTime": "2022-09-19T17:08:59.374932448+08:00",
            "DatasetName": "test-dataset",
            "FaceCount": 5,
            "Gender": "female",
            "ImageCount": 5,
            "MaxAge": 26.0,
            "MinAge": 22.0,
            "ObjectId": "Cluster-856be781-bf5a-46d7-8494-8d7c44f5e282",
            "ObjectType": "figure-cluster",
            "OwnerId": "*****",
            "ProjectName": "test-project",
            "UpdateTime": "2022-09-19T17:08:59.374932448+08:00",
            "VideoCount": 0
        }
    ],
    "NextToken": "",
    "TotalCount": 2,
    "RequestId": "42B3DD92-FE0D-09B7-B582-*****"
}

In this example, the dataset contains 2 clusters. Cluster Cluster-7bdbcedb-bd79-42e7-a1e2-b29a48532bd6 contains 16 images, and cluster Cluster-856be781-bf5a-46d7-8494-8d7c44f5e282 contains 5 images.

Sample code (Python)

# -*- coding: utf-8 -*-

import os
from alibabacloud_imm20200930.client import Client as imm20200930Client
from alibabacloud_tea_openapi import models as open_api_models
from alibabacloud_imm20200930 import models as imm_20200930_models
from alibabacloud_tea_util import models as util_models
from alibabacloud_tea_util.client import Client as UtilClient


class Sample:
    def __init__(self):
        pass

    @staticmethod
    def create_client(
        access_key_id: str,
        access_key_secret: str,
    ) -> imm20200930Client:
        """
        Initialize the IMM client with an AccessKey pair.
        @param access_key_id:
        @param access_key_secret:
        @return: Client
        @throws Exception
        """
        config = open_api_models.Config(
            access_key_id=access_key_id,
            access_key_secret=access_key_secret
        )
        # Set the endpoint. Replace cn-beijing with your region.
        config.endpoint = f'imm.cn-beijing.aliyuncs.com'
        return imm20200930Client(config)

    @staticmethod
    def main() -> None:
        # Read AccessKey pair from environment variables.
        # Use a RAM user instead of the Alibaba Cloud account for security.
        # For details, see https://www.alibabacloud.com/help/document_detail/2361894.html
        imm_access_key_id = os.getenv("AccessKeyId")
        imm_access_key_secret = os.getenv("AccessKeySecret")
        # Initialize the client.
        client = Sample.create_client(imm_access_key_id, imm_access_key_secret)
        # Build the query request.
        query_figure_clusters_request = imm_20200930_models.QueryFigureClustersRequest(
            # IMM project name
            project_name='test-project',
            # Dataset name
            dataset_name='test-dataset'
        )
        runtime = util_models.RuntimeOptions()
        try:
            response = client.query_figure_clusters_with_options(query_figure_clusters_request, runtime)
            print(response.body.to_map())
        except Exception as error:
            UtilClient.assert_as_string(error.message)
            print(error)


if __name__ == '__main__':
    Sample.main()

Query images in a cluster

After retrieving cluster information, call SimpleQuery with the cluster ID to list all images in that cluster. The following example retrieves images from cluster Cluster-7bdbcedb-bd79-42e7-a1e2-b29a48532bd6.

Request example

{
    "ProjectName": "test-project",
    "DatasetName": "test-dataset",
    "Query": "{\"Field\": \"Figures.FigureClusterId\", \"Operation\": \"eq\", \"Value\": \"Cluster-7bdbcedb-bd79-42e7-a1e2-b29a48532bd6\"}",
    "MaxResults": 100
}

Response example

Note

The cluster contains many images. The following example shows the data for a single image.

{
    "Aggregations": [],
    "Files": [
        {
            "Addresses": [],
            "AudioCovers": [],
            "AudioStreams": [],
            "ContentMd5": "ViAbCBHAZgNU4zvs5****==",
            "ContentType": "image/jpeg",
            "CreateTime": "2022-07-12T15:57:47.792615815+08:00",
            "CroppingSuggestions": [],
            "DatasetName": "test-dataset",
            "ETag": "\"56201B0811C0660354E33BECE4C****\"",
            "EXIF": "****",
            "Figures": [
                {
                    "FaceQuality": 1.0,
                    "FigureClusterId": "Cluster-7bdbcedb-bd79-42e7-a1e2-b29a48532bd6",
                    "FigureConfidence": 1.0,
                    "FigureId": "cd9139bf-f339-4ec2-b5fd-****",
                    "FigureType": "face",
                    "Glasses": "none",
                    "GlassesConfidence": 0.9990000128746033,
                    "Hat": "none",
                    "HatConfidence": 1.0,
                    "HeadPose": {
                        "Pitch": -0.8999999761581421,
                        "Roll": 1.1660000085830688,
                        "Yaw": 7.932000160217285
                    },
                    "Mask": "none",
                    "MaskConfidence": 0.6830000281333923,
                    "Mouth": "close",
                    "MouthConfidence": 0.7879999876022339,
                    "Sharpness": 1.0
                }
            ],
            "FileHash": "\"56201B0811C0660354E33BECE****\"",
            "FileModifiedTime": "2022-07-12T15:56:41+08:00",
            "Filename": "3.jpg",
            "ImageHeight": 1024,
            "ImageScore": {
                "OverallQualityScore": 0.7490000128746033
            },
            "ImageWidth": 683,
            "Labels": [
                {
                    "CentricScore": 0.8349999785423279,
                    "LabelConfidence": 1.0,
                    "LabelLevel": 2,
                    "LabelName": "\u7167\u7247\u62cd\u6444",
                    "Language": "zh-Hans",
                    "ParentLabelName": "\u827a\u672f\u54c1"
                }
            ],
            "MediaType": "image",
            "OCRContents": [],
            "OSSCRC64": "3400224321778591044",
            "OSSObjectType": "Normal",
            "OSSStorageClass": "Standard",
            "OSSTaggingCount": 0,
            "ObjectACL": "default",
            "ObjectId": "d132a61122c659f6fc1b42ecee1662aff358c7f1720027bead225****",
            "ObjectType": "file",
            "Orientation": 1,
            "OwnerId": "****",
            "ProduceTime": "2014-02-21T00:03:36+08:00",
            "ProjectName": "test-project",
            "Size": 187674,
            "Subtitles": [],
            "URI": "oss://bucket1/1.jpg",
            "UpdateTime": "2022-07-12T16:41:19.336736388+08:00",
            "VideoStreams": []
        }
    ],
    "NextToken": "",
    "RequestId": "84E4D242-8D15-0312-B976-****"
}

This result shows that the cluster includes an image at oss://bucket1/1.jpg.

Sample code (Python)

# -*- coding: utf-8 -*-

import os
from alibabacloud_imm20200930.client import Client as imm20200930Client
from alibabacloud_tea_openapi import models as open_api_models
from alibabacloud_imm20200930 import models as imm_20200930_models
from alibabacloud_tea_util import models as util_models
from alibabacloud_tea_util.client import Client as UtilClient


class Sample:
    def __init__(self):
        pass

    @staticmethod
    def create_client(
        access_key_id: str,
        access_key_secret: str,
    ) -> imm20200930Client:
        """
        Initialize the IMM client with an AccessKey pair.
        @param access_key_id:
        @param access_key_secret:
        @return: Client
        @throws Exception
        """
        config = open_api_models.Config(
            access_key_id=access_key_id,
            access_key_secret=access_key_secret
        )
        # Set the endpoint. Replace cn-beijing with your region.
        config.endpoint = f'imm.cn-beijing.aliyuncs.com'
        return imm20200930Client(config)

    @staticmethod
    def main() -> None:
        # Read AccessKey pair from environment variables.
        # Use a RAM user instead of the Alibaba Cloud account for security.
        # For details, see https://www.alibabacloud.com/help/document_detail/2361894.html
        imm_access_key_id = os.getenv("AccessKeyId")
        imm_access_key_secret = os.getenv("AccessKeySecret")
        # Initialize the client.
        client = Sample.create_client(imm_access_key_id, imm_access_key_secret)
        # Build the query request.
        request = imm_20200930_models.SimpleQueryRequest()
        params = {
            # Filter by cluster ID
            "Query": {"Field": "Figures.FigureClusterId", "Operation": "eq", "Value": "Cluster-7bdbcedb-bd79-42e7-a1e2-b29a48532bd6"},
            # IMM project name
            "ProjectName": "test-project",
            # Dataset name
            "DatasetName": "test-dataset",
            # Return up to 100 results
            "MaxResults": 100
        }
        request.from_map(params)
        runtime = util_models.RuntimeOptions()
        try:
            response = client.simple_query_with_options(request, runtime)
            print(response.body.to_map())
        except Exception as error:
            UtilClient.assert_as_string(error.message)
            print(error)


if __name__ == '__main__':
    Sample.main()

Best practices

Choose a clustering frequency

CreateFigureClusteringTask is incremental -- there is no need to call it after every image is indexed. Two recommended approaches:

Approach	How it works	When to use
Scheduled polling	Call CreateFigureClusteringTask on each dataset at regular intervals (for example, every 5 minutes).	Simple setups with low indexing volume
Event-driven with delay queue (recommended)	Each time IndexFileMeta is called, push the `DatasetName` into a delay queue. Retrieve dataset names periodically and call CreateFigureClusteringTask at least 10 seconds after the last IndexFileMeta call for that dataset.	Production environments with frequent indexing

Account for asynchronous latency

Metadata indexing is asynchronous. Design your application logic around these typical delays:

Operation	Typical latency
IndexFileMeta	~10 seconds
CreateFigureClusteringTask	Up to 180 seconds (usually a few seconds)
SimpleQuery (after async operations)	Wait at least 10 seconds after completion

Important

CreateFigureClusteringTask depends on IndexFileMeta to detect faces. If subscribing to IndexFileMeta results via Simple Message Queue (formerly MNS), wait at least 10 seconds after IndexFileMeta completes before calling CreateFigureClusteringTask. This makes sure the latest face information is available.

FAQ

Why does face clustering not generate any clusters?

The dataset does not meet the minimum requirements. There must be at least 3 images of the same person, and at least 3 of those images must have:

Face area greater than 75 x 75 pixels
HeadPose values (Pitch, Roll, Yaw) each with an absolute value less than 30
FaceQuality greater than 0.8

Once a cluster is created, additional faces that do not meet these thresholds may still be added to that cluster.

How do I find the cluster ID?

Call QueryFigureClusters and look at the ObjectId field within each entry in the FigureClusters array.

How do I query images in a specific cluster?

Call SimpleQuery with the following Query parameter, replacing the cluster ID with the value from QueryFigureClusters:

{
    "Field": "Figures.FigureClusterId",
    "Operation": "eq",
    "Value": "Cluster-7bdbcedb-bd79-42e7-a1e2-b29a48532bd6"
}

Why are generated clusters not immediately searchable?

Indexing and clustering are asynchronous. After IndexFileMeta and CreateFigureClusteringTask complete, wait at least 10 seconds before querying results with SimpleQuery.

How are images with multiple faces handled?

Each face in an image is evaluated independently. If an image contains multiple faces, each face may be assigned to a different cluster based on its features.

Why does SimpleQuery return faces from other clusters when filtering by FigureClusterId?

SimpleQuery returns results at the image level, so all faces and labels in a matching image are included. To find the specific face that belongs to the cluster, iterate through the Figures array in each result and match on the FigureClusterId value. This gives the position, expression, age, and other attributes of the corresponding face.

How do I get results from a CreateFacesSearchingTask without configuring notifications?

The CreateFacesSearchingTask operation returns results only through notification channels (Simple Message Queue, ApsaraMQ for RocketMQ, or EventBridge). GetTask retrieves task status information but does not include search results.

Related operations

Operation	Description
CreateDataset	Create a dataset
CreateBinding	Bind an OSS data source to a dataset for automatic indexing
IndexFileMeta	Index file metadata into a dataset
BatchIndexFileMeta	Batch index file metadata
CreateFigureClusteringTask	Create a face clustering task
QueryFigureClusters	Query all face clusters in a dataset
SimpleQuery	Query files by conditions (e.g., filter by cluster ID)
GetFileMeta	Get file metadata including face quality and angle
GetTask	Get task status and information
ListTasks	List tasks in a project