How to use face clustering feature - Intelligent Media Management

The face clustering feature allows you to group multiple images that contain similar faces in a dataset. This feature is suitable for scenarios such as face albums in cloud drives, stranger detection in home surveillance, and customer management in the New Retail industry. After face clustering, you can query all images that contain the face of a specific person in a cluster.

Scenarios

Cloud drive face albums

Face clustering groups photos by face in a cloud drive and generates personalized face albums.

Home surveillance

Face clustering records faces of family members. When the face of a stranger appears, face clustering fails and sends an alert. This helps you identify and handle dangerous personnel and events at the earliest opportunity and ensure the safety of your family members.

New retail customer management

Face clustering and deduplication on collected images help you obtain accurate traffic data of customers, which can be used to analyze customer purchase preferences for precise marketing.

Prerequisites

You have indexed the images to be grouped into a dataset (CreateDataset - Create a dataset) by using the binding method (CreateBinding - Create a binding task) or the active indexing method (IndexFileMeta - Index file metadata or BatchIndexFileMeta - Batch index file metadata).
The dataset must contain at least three images of the same person, and At Least three of these images must meet the following requirements for high-definition face images:
The face area of an image must be greater than 75 × 75 pixels.
The absolute value of each of the three elements in the HeadPose parameter must be less than 30.
The value of the FaceQuality parameter must be greater than 0.8.

Note

You can call the GetFileMeta - Get file metadata operation to obtain the face angle and quality information.
After a face cluster is created, faces that do not meet the preceding requirements may also be added to the same cluster. For more information about face clustering, see FAQ about image management.

Create a face clustering task

You can call the CreateFigureClusteringTask - Create a figure clustering task operation to create a face clustering task. This task clusters faces of different persons in the images that you have indexed to a dataset. The following example shows how to cluster faces in the images in the test-dataset dataset.

Note

The operation generates only clusters and does not change the images.

Important

The task information is retained for seven days after the task starts. Task information cannot be obtained after the seven-day window ends. You can use one of the following methods to query task information:

Call the GetTask or ListTasks operation.
In the region in which the IMM project is located, configure a Simple Message Queue (SMQ) subscription to receive task information notifications.. For more information, see Asynchronous message examples. For information about the MNS SDK, see Step 4: Receive and delete the message.
In the region in which the IMM project is located, create an ApsaraMQ for RocketMQ 4.0 instance, a topic, and a group to receive task notifications. For more information, see Asynchronous message examples. For information about how to use ApsaraMQ for RocketMQ, see Use HTTP client SDKs to send and subscribe to normal messages.
In the region in which the IMM project is located, use EventBridge to receive task information notifications. For more information, see IMM events.

Request example

{
    "ProjectName": "test-project",
    "DatasetName": "test-dataset"
}

Response example

{
    "TaskId": "CreateFigureClusteringTask-ba5784b8-f61e-485d-8ea0-****",
    "RequestId": "42F4F8FD-006D-0EF0-8F2A-****",
    "EventId": "140-1L5dh6eSUErqdxV1ZvJ****"
}

Note

If the response is similar to the preceding content, the face clustering task is created.

Sample code

# -*- coding: utf-8 -*-

import os
from alibabacloud_imm20200930.client import Client as imm20200930Client
from alibabacloud_tea_openapi import models as open_api_models
from alibabacloud_imm20200930 import models as imm_20200930_models
from alibabacloud_tea_util import models as util_models
from alibabacloud_tea_util.client import Client as UtilClient


class Sample:
    def __init__(self):
        pass

    @staticmethod
    def create_client(
        access_key_id: str,
        access_key_secret: str,
    ) -> imm20200930Client:
        """
        Use your AccessKey ID and AccessKey secret to initialize the client.
        @param access_key_id:
        @param access_key_secret:
        @return: Client
        @throws Exception
        """
        config = open_api_models.Config(
            access_key_id=access_key_id,
            access_key_secret=access_key_secret
        )
        # Specify the endpoint.
        config.endpoint = f'imm.cn-beijing.aliyuncs.com'
        return imm20200930Client(config)

    @staticmethod
    def main() -> None:
        # The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. Using these credentials to perform operations in Function Compute is a high-risk operation. We recommend that you use a RAM user to call API operations or perform routine O&M.
        # We recommend that you do not include your AccessKey pair (AccessKey ID and AccessKey secret) in your project code for data security reasons.
        # In this example, the AccessKey pair is read from the environment variables to implement identity verification for API access. For information about how to configure environment variables, visit https://www.alibabacloud.com/help/document_detail/2361894.html.
        imm_access_key_id = os.getenv("AccessKeyId")
        imm_access_key_secret = os.getenv("AccessKeySecret")
        # Initialize the client.
        client = Sample.create_client(imm_access_key_id, imm_access_key_secret)
        # Construct a request.
        create_figure_clustering_task_request = imm_20200930_models.CreateFigureClusteringTaskRequest(
            # Specify the name of the IMM project.
            project_name='test-project',
            # Specify the name of the dataset.
            dataset_name='test-dataset'
        )
        runtime = util_models.RuntimeOptions()
        try:
            # Print the response.
            response = client.create_figure_clustering_task_with_options(
                create_figure_clustering_task_request, runtime)
            print(response.body.to_map())
        except Exception as error:
            # Print the error message if necessary.
            UtilClient.assert_as_string(error.message)
            print(error)


if __name__ == '__main__':
    Sample.main()

Query face cluster information

After a face clustering task is created, you can call the QueryFigureClusters - Query figure clusters operation to query cluster information, including the number of clusters and the number of images in each cluster. The following example shows how to query face cluster information in the test-dataset dataset.

Request example

{
    "ProjectName": "test-project",
    "DatasetName": "test-dataset"
}

Response example

{
    "FigureClusters": [
        {
            "AverageAge": 27.125,
            "Cover": {
                "Addresses": [],
                "AudioCovers": [],
                "AudioStreams": [],
                "CroppingSuggestions": [],
                "Figures": [
                    {
                        "Attractive": 0.9980000257492065,
                        "Beard": "none",
                        "BeardConfidence": 0.9959999918937683,
                        "Boundary": {
                            "Height": 270,
                            "Left": 573,
                            "Top": 104,
                            "Width": 202
                        },
                        "FaceQuality": 1.0,
                        "FigureId": "d7365ab8-1378-4bec-83cb-eccad8d11e0b",
                        "FigureType": "face",
                        "Glasses": "none",
                        "GlassesConfidence": 0.9990000128746033,
                        "Hat": "none",
                        "HatConfidence": 1.0,
                        "HeadPose": {
                            "Pitch": -0.7369999885559082,
                            "Roll": 2.5399999618530273,
                            "Yaw": 9.138999938964844
                        },
                        "Mask": "none",
                        "MaskConfidence": 0.7269999980926514,
                        "Mouth": "open",
                        "MouthConfidence": 0.9959999918937683,
                        "Sharpness": 1.0
                    }
                ],
                "ImageHeight": 683,
                "ImageWidth": 1024,
                "Labels": [],
                "OCRContents": [],
                "ObjectId": "170ffdeb36cec846f4214c78a0f3a0d4b7e37d0305370216ae780f7b8c72f871",
                "Subtitles": [],
                "URI": "oss://bucket1/photos/2.jpg",
                "VideoStreams": []
            },
            "CreateTime": "2022-07-12T16:41:19.336825716+08:00",
            "DatasetName": "dataset1",
            "FaceCount": 16,
            "Gender": "female",
            "ImageCount": 16,
            "MaxAge": 30.0,
            "MinAge": 23.0,
            "ObjectId": "Cluster-7bdbcedb-bd79-42e7-a1e2-b29a48532bd6",
            "ObjectType": "figure-cluster",
            "OwnerId": "*****",
            "ProjectName": "test-project",
            "UpdateTime": "2022-09-19T17:08:59.374781532+08:00",
            "VideoCount": 0
        },
        {
            "AverageAge": 24.200000762939453,
            "Cover": {
                "Addresses": [],
                "AudioCovers": [],
                "AudioStreams": [],
                "CroppingSuggestions": [],
                "Figures": [
                    {
                        "Attractive": 0.9990000128746033,
                        "Beard": "none",
                        "BeardConfidence": 0.9990000128746033,
                        "Boundary": {
                            "Height": 266,
                            "Left": 301,
                            "Top": 218,
                            "Width": 196
                        },
                        "FaceQuality": 0.8859999775886536,
                        "FigureId": "f58bbdce-f3d1-4674-be6b-43d4b47c08e1",
                        "FigureType": "face",
                        "Glasses": "none",
                        "GlassesConfidence": 1.0,
                        "Hat": "none",
                        "HatConfidence": 1.0,
                        "HeadPose": {
                            "Pitch": 13.963000297546387,
                            "Roll": -12.21399974822998,
                            "Yaw": -6.2210001945495605
                        },
                        "Mask": "none",
                        "MaskConfidence": 0.7490000128746033,
                        "Mouth": "open",
                        "MouthConfidence": 0.9940000176429749,
                        "Sharpness": 1.0
                    }
                ],
                "ImageHeight": 1024,
                "ImageWidth": 683,
                "Labels": [],
                "OCRContents": [],
                "ObjectId": "b9c80e51aa95072413e2a0a6e5262644bc3cba14a4754f54f3fa9850c4d244f1",
                "Subtitles": [],
                "URI": "oss://bucket1/photos/11.jpg",
                "VideoStreams": []
            },
            "CreateTime": "2022-09-19T17:08:59.374932448+08:00",
            "DatasetName": "test-dataset",
            "FaceCount": 5,
            "Gender": "female",
            "ImageCount": 5,
            "MaxAge": 26.0,
            "MinAge": 22.0,
            "ObjectId": "Cluster-856be781-bf5a-46d7-8494-8d7c44f5e282",
            "ObjectType": "figure-cluster",
            "OwnerId": "*****",
            "ProjectName": "test-project",
            "UpdateTime": "2022-09-19T17:08:59.374932448+08:00",
            "VideoCount": 0
        }
    ],
    "NextToken": "",
    "TotalCount": 2,
    "RequestId": "42B3DD92-FE0D-09B7-B582-*****"
}

Note

The response example shows that the face images in this dataset are divided into 2 clusters. One cluster has the ID Cluster-7bdbcedb-bd79-42e7-a1e2-b29a48532bd6 and contains 16 images. The other cluster has the ID Cluster-856be781-bf5a-46d7-8494-8d7c44f5e282 and contains 5 images.

Sample code

# -*- coding: utf-8 -*-

import os
from alibabacloud_imm20200930.client import Client as imm20200930Client
from alibabacloud_tea_openapi import models as open_api_models
from alibabacloud_imm20200930 import models as imm_20200930_models
from alibabacloud_tea_util import models as util_models
from alibabacloud_tea_util.client import Client as UtilClient


class Sample:
    def __init__(self):
        pass

    @staticmethod
    def create_client(
        access_key_id: str,
        access_key_secret: str,
    ) -> imm20200930Client:
        """
        Use your AccessKey ID and AccessKey secret to initialize the client.
        @param access_key_id:
        @param access_key_secret:
        @return: Client
        @throws Exception
        """
        config = open_api_models.Config(
            access_key_id=access_key_id,
            access_key_secret=access_key_secret
        )
        # Specify the endpoint.
        config.endpoint = f'imm.cn-beijing.aliyuncs.com'
        return imm20200930Client(config)

    @staticmethod
    def main() -> None:
        # The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. We recommend that you use a Resource Access Management (RAM) user to call API operations or perform routine O&M.
        # We recommend that you do not save the AccessKey ID and the AccessKey secret in your project code. Otherwise, the AccessKey pair may be leaked, and this may compromise the security of all resources under your account. 
        # In this example, the AccessKey pair is read from the environment variables to implement identity verification for API access. For information about how to configure environment variables, visit https://www.alibabacloud.com/help/document_detail/2361894.html.
        imm_access_key_id = os.getenv("AccessKeyId")
        imm_access_key_secret = os.getenv("AccessKeySecret")
        # Initialize the client.
        client = Sample.create_client(imm_access_key_id, imm_access_key_secret)
        # Construct a request.
        query_figure_clusters_request = imm_20200930_models.QueryFigureClustersRequest(
            # Specify the IMM project.
            project_name='test-project',
            # Specify the name of the dataset.
            dataset_name='test-dataset'
        )
        runtime = util_models.RuntimeOptions()
        try:
            # Print the response.
            response = client.query_figure_clusters_with_options(query_figure_clusters_request, runtime)
            print(response.body.to_map())
        except Exception as error:
            # Print the error message if necessary.
            UtilClient.assert_as_string(error.message)
            print(error)


if __name__ == '__main__':
    Sample.main()

Query the list of images in a face cluster

After you query cluster information, you can call the SimpleQuery - Simple query operation to query all images in a specific cluster by cluster ID. The following example shows how to query the images in the face cluster whose ID is Cluster-7bdbcedb-bd79-42e7-a1e2-b29a48532bd6 in the test-dataset dataset.

Request example

{
    "ProjectName": "test-project",
    "DatasetName": "test-dataset",
    "Query": "{\"Field\": \"Figures.FigureClusterId\", \"Operation\": \"eq\", \"Value\": \"Cluster-7bdbcedb-bd79-42e7-a1e2-b29a48532bd6\"}",
    "MaxResults": 100
}

Response example

Note

The cluster includes many images. The following content demonstrates the information about only one image.

{
    "Aggregations": [],
    "Files": [
        {
            "Addresses": [],
            "AudioCovers": [],
            "AudioStreams": [],
            "ContentMd5": "ViAbCBHAZgNU4zvs5****==",
            "ContentType": "image/jpeg",
            "CreateTime": "2022-07-12T15:57:47.792615815+08:00",
            "CroppingSuggestions": [],
            "DatasetName": "test-dataset",
            "ETag": "\"56201B0811C0660354E33BECE4C****\"",
            "EXIF": "****",
            "Figures": [
                {
                    "FaceQuality": 1.0,
                    "FigureClusterId": "Cluster-7bdbcedb-bd79-42e7-a1e2-b29a48532bd6",
                    "FigureConfidence": 1.0,
                    "FigureId": "cd9139bf-f339-4ec2-b5fd-****",
                    "FigureType": "face",
                    "Glasses": "none",
                    "GlassesConfidence": 0.9990000128746033,
                    "Hat": "none",
                    "HatConfidence": 1.0,
                    "HeadPose": {
                        "Pitch": -0.8999999761581421,
                        "Roll": 1.1660000085830688,
                        "Yaw": 7.932000160217285
                    },
                    "Mask": "none",
                    "MaskConfidence": 0.6830000281333923,
                    "Mouth": "close",
                    "MouthConfidence": 0.7879999876022339,
                    "Sharpness": 1.0,
                    ...
                }
            ],
            "FileHash": "\"56201B0811C0660354E33BECE****\"",
            "FileModifiedTime": "2022-07-12T15:56:41+08:00",
            "Filename": "3.jpg",
            "ImageHeight": 1024,
            "ImageScore": {
                "OverallQualityScore": 0.7490000128746033
            },
            "ImageWidth": 683,
            "Labels": [
                {
                    "CentricScore": 0.8349999785423279,
                    "LabelConfidence": 1.0,
                    "LabelLevel": 2,
                    "LabelName": "\u7167\u7247\u62cd\u6444",
                    "Language": "zh-Hans",
                    "ParentLabelName": "\u827a\u672f\u54c1"
                },
                ...
            ],
            "MediaType": "image",
            "OCRContents": [],
            "OSSCRC64": "3400224321778591044",
            "OSSObjectType": "Normal",
            "OSSStorageClass": "Standard",
            "OSSTaggingCount": 0,
            "ObjectACL": "default",
            "ObjectId": "d132a61122c659f6fc1b42ecee1662aff358c7f1720027bead225****",
            "ObjectType": "file",
            "Orientation": 1,
            "OwnerId": "****",
            "ProduceTime": "2014-02-21T00:03:36+08:00",
            "ProjectName": "test-project",
            "Size": 187674,
            "Subtitles": [],
            "URI": "oss://bucket1/1.jpg",
            "UpdateTime": "2022-07-12T16:41:19.336736388+08:00",
            "VideoStreams": []
        },
        ...
    ],
    "NextToken": "",
    "RequestId": "84E4D242-8D15-0312-B976-****"
}

Note

The returned example shows that the group contains an image with the OSS address oss://bucket1/1.jpg.

Sample code

# -*- coding: utf-8 -*-

import os
from alibabacloud_imm20200930.client import Client as imm20200930Client
from alibabacloud_tea_openapi import models as open_api_models
from alibabacloud_imm20200930 import models as imm_20200930_models
from alibabacloud_tea_util import models as util_models
from alibabacloud_tea_util.client import Client as UtilClient


class Sample:
    def __init__(self):
        pass

    @staticmethod
    def create_client(
        access_key_id: str,
        access_key_secret: str,
    ) -> imm20200930Client:
        """
        Use your AccessKey ID and AccessKey secret to initialize the client.
        @param access_key_id:
        @param access_key_secret:
        @return: Client
        @throws Exception
        """
        config = open_api_models.Config(
            access_key_id=access_key_id,
            access_key_secret=access_key_secret
        )
        # Specify the endpoint.
        config.endpoint = f'imm.cn-beijing.aliyuncs.com'
        return imm20200930Client(config)

    @staticmethod
    def main() -> None:
        # The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. Using these credentials to perform operations in Function Compute is a high-risk operation. We recommend that you use a RAM user to call API operations or perform routine O&M.
        # We recommend that you do not include your AccessKey pair (AccessKey ID and AccessKey secret) in your project code for data security reasons.
        # In this example, the AccessKey pair is read from the environment variables to implement identity verification for API access. For information about how to configure environment variables, see https://www.alibabacloud.com/help/document_detail/2361894.html.
        imm_access_key_id = os.getenv("AccessKeyId")
        imm_access_key_secret = os.getenv("AccessKeySecret")
        # Initialize the client.
        client = Sample.create_client(imm_access_key_id, imm_access_key_secret)
        # Construct a request.
        request = imm_20200930_models.SimpleQueryRequest()
        params = {
            # Set the query conditions.
            "Query": {"Field": "Figures.FigureClusterId", "Operation": "eq", "Value": "Cluster-7bdbcedb-bd79-42e7-a1e2-b29a48532bd6"},
            # Specify the IMM project.
            "ProjectName": "test-project",
            # Specify the name of the dataset.
            "DatasetName": "test-dataset",
            # Specify that up to 100 query results can be returned.
            "MaxResults": 100
        }
        request.from_map(params)
        runtime = util_models.RuntimeOptions()
        try:
            # Print the response.
            response = client.simple_query_with_options(request, runtime)
            print(response.body.to_map())
        except Exception as error:
            # Print the error message if necessary.
            UtilClient.assert_as_string(error.message)
            print(error)


if __name__ == '__main__':
    Sample.main()

FAQ

Why can't face clustering generate face clusters?

The failure occurs because the requirements to create face image groups are not met. For accuracy of face grouping, the following requirements must be met to create face groups:

You have used the IndexFileMeta - Index file metadata operation to add images to a dataset.
The dataset must contain at least three images of the same person, and at least three of these images must meet the following requirements for high-definition face images:
- The face area of the image must be greater than 75 × 75 pixels.
- The absolute values of the three elements in the HeadPose parameter are all less than 30.
- The value of the FaceQuality parameter is greater than 0.8.

After a face cluster is created, faces that do not meet the preceding requirements may also be added to the same cluster.

How do I query the list of images in a face cluster?

You can call the SimpleQuery - Simple query operation and specify the cluster ID in the Query parameter to query the list of images in the cluster. The following content shows the details.

Note

You can call the QueryFigureClusters - Query figure clusters operation to obtain the ObjectId parameter in the FigureClusters parameter, which is the cluster ID.

{
  "Field ": "Figures.FigureClusterId ",
  "Operation ": "eq ",
  "Value ": "Face image group ID"
}

Why can't I search for the generated face clusters?

Because metadata indexing is asynchronous, it takes some time for face clusters to be searchable after images are indexed. Therefore, when designing your business logic, consider the data consistency issues caused by latency.

Image indexing: It typically takes 10 seconds to index an image by calling the IndexFileMeta - Index file metadata operation.
Face clustering: It takes up to 180 seconds to perform face clustering by calling the CreateFigureClusteringTask - Create a figure clustering task operation. The time depends on the number of images indexed at once, but it usually completes within a few seconds.
Important
The CreateFigureClusteringTask operation depends on the IndexFileMeta operation to detect faces in images. If you subscribe to the call results of the IndexFileMeta operation through MNS messages, you should wait at least 10 seconds after the IndexFileMeta operation is completed before calling the CreateFigureClusteringTask operation. This ensures that the CreateFigureClusteringTask operation can obtain the latest face information in the images.
After asynchronous operations such as IndexFileMeta and CreateFigureClusteringTask are completed, you can search for the latest results using the SimpleQuery operation after a 10-second interval.

Do I need to call the CreateFigureClusteringTask operation every time I index an image?

No, you do not need to call the CreateFigureClusteringTask operation every time an image is indexed. The CreateFigureClusteringTask - Create a figure clustering task operation is an incremental clustering operation. You can call the CreateFigureClusteringTask operation once after a batch of images is indexed to perform batch clustering. The following solutions are recommended:

Solution 1: Call the CreateFigureClusteringTask operation on each dataset at regular intervals, such as every 5 minutes.
Solution 2: Each time you call the IndexFileMeta - Index file metadata operation, push the corresponding dataset into a delay queue. Periodically retrieve the DatasetName from the queue to get datasets with new images. Call the CreateFigureClusteringTask operation 10 seconds after your last call of the IndexFileMeta operation. This solution is recommended.

How are images with multiple faces grouped?

Face clustering is performed based on face features. If an image contains multiple faces, each face may be assigned to a specific cluster.

Why does the SimpleQuery operation return other faces when I search by cluster ID (FigureClusterId)?

SimpleQuery returns results at the image level. All faces and labels in an image are returned. To find the face information of the searched cluster (person) in an image, traverse the Figures field in the results and find the face object whose FigureClusterId matches the FigureClusterId you searched for. This allows you to understand the position, expression, age, and other information of the corresponding face.

How do I get search results when using the IMM CreateFacesSearchingTask operation to create a similar face search task without configuring notification?

When you use the CreateFacesSearchingTask operation to create a similar face search task, you can only receive results through Notification. GetTask can only query task information but cannot obtain search results.