Use a DataCache to build a Miaoya Camera in a Kubernetes cluster - Elastic Container Instance

This topic describes how to use a DataCache to store the base model (ID: ly261666/cv_portrait_model) of the portrait AI-Generated Content (AIGC) and build the eci-facechain application that is similar to Miaoya Camera. The base model of the portrait AIGC is provided by the ModelScope Community. Before you deploy eci-facechain, you can store the model data of the application in a DataCache. When you create the pod that corresponds to the application, you can mount the model data to the pod. This way, the system does not need to pull model data and can deploy the application faster.

Background information

Miaoya Camera is an AIGC-based portrait application. You can upload personal photos in the application to obtain your digital identity and then select your favorite style template to obtain a set of personal photos.

Alibaba DAMO Academy recently published an open source application named FaceChain in ModelScope. FaceChain is similar to Miaoya Camera. FaceChain is a fusion of Stable Diffusion (SD) and Low-Rank Adaptation of Large Language Models (LoRA). The following items describe how FaceChain works:

Trains existing personal photos (original photos) to obtain face-specific LoRA weights.
Generates various styles of personal photos (resulting photos) based on the standard SD base model.
Note
FaceChain optimizes photos based on various portrait-related and skin-related models and then generates photos that have better effects.

You can configure and use open source FaceChain after you download the FaceChain application. However, FaceChain has the following issues:

During training and inference, FaceChain downloads models from the model hub that contains multiple models whose sizes are approximately 25 GB. As a result, the models require an extended period of time to download. In addition, you must have a stable network connection and a specific bandwidth when you download the models over the Internet. You may fail to download the models due to network-related and bandwidth-related reasons.
FaceChain does not support facial memory. The application cannot retain the LoRA weights that are obtained from historical training and performs inference based only on the most recent training results.
If you want to change a model or change the version of a model, you must manually modify the code. FaceChain is complex to maintain and has non-flexible configurations.

To address the preceding issues, Elastic Container Instance published the optimized version of FaceChain called eci-facechain. eci-facechain provides the following benefits:

Stores model data into the DataCache of Elastic Container Instance. The application does not need to download models from the model hub after the application is started.
Supports facial memory. You can add tags to faces during training, and select faces that have been trained based on tags during inference.
Adds environment variables. You can specify the model, version, directory, and application port of the SD by using the environment variables of containers without modifying the application code. For more information about environment variables, see the env.py file.
Optimizes web pages. For example, you can specify the number of pixels for the photos that you want to generate.

Prerequisites

A DataCache custom resource definition (CRD) is deployed in the cluster. For more information, see Deploy a DataCache CRD.
The virtual private cloud (VPC) in which the cluster resides is associated with an Internet NAT gateway. An SNAT entry is configured for the Internet NAT gateway to allow resources in the VPC or resources connected to vSwitches in the VPC to access the Internet.
Note
If the VPC is not associated with an Internet NAT gateway, you must associate an elastic IP address (EIP) with the VPC when you create the DataCache and deploy the application. This way, you can pull data from the Internet.

Prepare a runtime environment

Hardware requirements
You must use an elastic container instance that is based on a GPU-accelerated Elastic Compute Service (ECS) instance type to deploy the application. The application does not have specific vCPU and memory requirements. We recommend that you select an ECS instance type that has a high GPU memory specification. For information about the GPU-accelerated ECS instance types that you can use to create elastic container instances, see Supported instance families.
Software requirements
Elastic Container Instance provides a public container image based on eci-facechain. You can directly use the image or use the image as a base image for secondary development. The image address is registry.cn-hangzhou.aliyuncs.com/eci_open/facechain:1.0.2.

Procedure

Create a DataCache

Access ModelScope to obtain the ID of the model.
In this example, the version of the base model (ID: ly261666/cv_portrait_model) of the portrait AIGC is v4.0. Find the model that you want to use in ModelScope and copy the model ID in the upper part of the model details page.

Write a YAML configuration file for the DataCache. Then, use the YAML file to create the DataCache that is used to store the base model of the portrait AIGC.

kubectl create -f datacache-test.yaml

Example of the datacache-test.yaml file

apiVersion: eci.aliyun.com/v1alpha1
kind: DataCache
metadata:
  name: cv
spec:
  bucket: test  
  path: /model/cv/ 
  dataSource:
    type: URL 
    options:
      repoSource: ModelScope/Model              # Specifies a ModelScope model as the data source.
      repoId: ly261666/cv_portrait_model        # Specifies a model ID.
      revision: v4.0                            # Specifies a model version.
  retentionDays: 1
  netConfig: 
    securityGroupId: sg-2ze63v3jtm8e6s******
    vSwitchId: vsw-2ze94pjtfuj9vay******       # Specifies a vSwitch for which SNAT entries are configured.

Query the status of the DataCache.
```
kubectl get edc
```
After the data is downloaded and the status of the DataCache becomes Available, the DataCache is ready for use. Example:

Build the eci-facechain application

Write the YAML configuration file of the eci-facechain application and then use the YAML file to deploy the application.

kubectl create -f cv.yaml

The following sample code provides an example of the content of the cv.yaml file. You can create a Deployment that contains one pod replica. The pod is based on a GPU-accelerated ECS instance type and is mounted with the base model of the portrait AIGC. The container in the pod uses the public image that is provided by Elastic Container Instance. After the container is started, the container runs a web service and can be accessed over port 8888.

Note

In the following YAML file, the system automatically creates an elastic IP address (EIP) and associates the EIP with the pod. If the virtual private cloud (VPC) to which your cluster belongs is associated with an Internet NAT gateway, you can remove the annotation of the EIP. After the pod is created, you can configure DNAT entries to allow external access to the pod. We recommend that you create a Service to provide uniform access to the pod.
By default, port 8888 is enabled for the container. If you want to change the port, specify another port by using the container environment variable APP_PORT.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cv
  labels:
    app: cv
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cv
  template:
    metadata:
      name: cv
      labels:
        app: cv
        alibabacloud.com/eci: "true" 
      annotations:
        k8s.aliyun.com/eci-use-specs: "ecs.gn7i-c8g1.2xlarge"    # Specifies a GPU-accelerated ECS instance type.
        k8s.aliyun.com/eci-data-cache-bucket: "test"             # Specifies a bucket to store the DataCache.
        k8s.aliyun.com/eci-with-eip: "true"                      # Automatically creates an EIP and associates the EIP with the pod to allow external access to the pod.
    spec:
      containers:
      - name: cv
        image: registry.cn-hangzhou.aliyuncs.com/eci_open/facechain:1.0.2  # # Uses the public image that is provided by Elastic Container Instance.
        resources:
            limits:
              nvidia.com/gpu: "1"           # Specifies the number of GPUs that are required by the container.
        volumeMounts:
        - name: "model-cv"
          mountPath: "/mnt/workspace/.cache/modelscope/ly261666/cv_portrait_model/"  # Specifies the mount path of the model data in the container.
      volumes: 
      - name: "model-cv"
        hostPath:             
          path: "/model/cv/"    # Specifies the path in which the model data is mounted.

Check whether the application is deployed.
```
kubectl get deployment cv
kubectl get Pod
```
The following sample output indicates that the application is deployed.
Check whether the model is mounted.
```
kubectl exec -it <POD_NAME> -- bash
ls /mnt/workspace/.cache/modelscope/ly261666/cv_portrait_model/
```
The following sample output indicates that the model is mounted in the /mnt/workspace/.cache/modelscope/ly261666/cv_portrait_model/ directory of the container.
Check the EIP that is associated with the pod.
```
kubectl describe pod <pod_name>
```
You can check the EIP that is associated with the pod in the Annotations section of the returned pod details.

Test the model

Add an inbound rule to the security group to which the pod belongs and allow port 8888.
Use a browser to visit the web page of the eci-facechain application.
In this example, an EIP is associated with the eci-facechain application. You can enter the EIP of the pod and the open port of the container (example: 60.205.XX.XX:8888) to access the application.
Test whether the base model of the portrait AIGC has a similar effect as Miaoya Camera.
1. On the Train tab, upload your face photos.
2. Add tags to the face photos based on your requirements, and then click Start training.
3. After the training is complete, click the Inference tab.
4. Select a style model and a cloth style based on your requirements, specify the quantity, height, and width of the photos that you want to generate, and then click Generate.
5. Wait and view the generation result.
  Note
  If an Error message is displayed, view the container log to identify the cause of the error. If the error type is out of memory (OOM), we recommend that you decrease the width and height of the image or use an ECS instance type that has a larger GPU memory to deploy the application.