ENS:Best practices for edge AI inference

Edge cloud heterogeneous computing power

The edge cloud offers four types of heterogeneous computing power at different prices for various scenarios across widely distributed nodes. The GPU memory of a single card ranges from 12 GB to 48 GB. ENS will continue to expand its computing specifications as GPU technology evolves.

These resources support applications that use large models like Tongyi Qianwen and Llama 2, with parameter counts from 0.5 billion to 72 billion. Use cases include lightweight dialogue, intelligent customer service, text-to-image generation, video understanding, code generation, content creation, and intelligent assistants. You can choose different specifications of heterogeneous computing power based on your business requirements.

Edge cloud capabilities

To better support edge AI inference services, the edge cloud provides value-added capabilities at different layers. You can select the capabilities you need to build a basic environment for edge AI inference.

1. Basic resource layer: In addition to heterogeneous computing power, this layer provides CPU computing power and storage services for data persistence.

2. Resource configuration layer: This layer offers comprehensive network services, such as NAT gateway, Edge Load Balancer (ELB), and edge elastic IP addresses (EIPs). It also provides container services. By using container multi-tenancy technology, you can run multiple container services within a single heterogeneous computing instance to improve resource utilization and optimize costs.

3. Service acceleration layer: To improve inference performance, this layer provides Alibaba Cloud's proprietary AIACC inference acceleration engine and the open-source TensorRT toolkit. This allows you to implement services that use different acceleration solutions.

4. Service scheduling layer: The edge cloud also provides service scheduling. You can delegate service scheduling to the edge cloud, which routes customer traffic to the nearest node based on scheduling policies. If a node fails or has insufficient resources, traffic is redirected to an available node. NAS synchronizes data across different nodes, which ensures a consistent user experience regardless of the access point.

Step 1: Create heterogeneous computing resources

1. Log on to the ENS console.

2. In the left-side navigation pane, choose Resources and Images > Instances.

3. On the Instances page, click Create Instance, select the required heterogeneous computing power specification, and complete the resource request. To learn how to create an instance, see Create an instance.

   On the Basic Configurations page, set Billing Method to Subscription, Instance Type to X86 Compute, Instance Specification to ens.gn7s-c48g1.xmlarge (vCPU: 48 Cores | Memory: 192 GB), and Image to a CentOS public image.

Step 2: Create network resources

1. In the left-side navigation pane of the ENS console, under Network Management, create the required network resources.

2. If your heterogeneous computing instance needs to access the internet or provide services over the internet, create an edge elastic IP address (EIP). For more information, see Create and manage edge EIPs.

3. If you need to distribute incoming traffic across multiple backend instances for synchronous processing and eliminate single points of failure, create an Edge Load Balancer (ELB) instance. For more information, see Create an ELB instance.

4. If you require network address translation capabilities, create an edge NAT gateway instance. For more information, see Create and manage edge NAT gateways.

Step 3: Create storage resources

1. In the left-side navigation pane of the ENS console, under Storage and Snapshots, create the required storage resources.

2. If your heterogeneous computing instance requires persistent storage, create a cloud disk. For more information, see Create and manage cloud disks and Create an instance.

3. If you need to share data among multiple computing instances, create a NAS file system. For more information, see Create and manage a file system.

Step 4: Deploy inference acceleration engine

1. Deploy the proprietary AIACC inference acceleration engine on a virtual machine. The following steps use a T4 GPU instance that runs Ubuntu 20.04 as an example.

   1. Log on to the heterogeneous computing resource that you created. For more information, see Connect to an instance.

   2. Install the CUDA Toolkit.

      1. T4 GPUs require CUDA Toolkit 11.8.

      2. Update the PATH and LD_LIBRARY_PATH environment variables.

   3. Install the GPU driver.

      1. T4 GPUs require driver version 12.2. We recommend installing the latest driver version.

      2. Download the corresponding driver from the official NVIDIA website: NVIDIA-Linux-x86_64-535.154.05.run.

   4. Update the environment variables to add the AIACC authorization.

      echo 'export DEEPGPU_EXT_CURL=MzkuOTguMjIuMTI2OjcwNzA=' >> /etc/profile
      source /etc/profile

   5. Confirm the component versions.

      1. Python version: 3.8

      2. PyTorch version: 2.1.0

      3. Deepytorch Inference version: deepytorch_inference==0.7.18+pt2.1.0cu118-cp38-cp38

      pip3 install torch==2.1.0 torchvision==0.16.0 numpy transformers

2. Deploy the proprietary AIACC inference acceleration engine by using a container. The following steps use a T4 GPU instance that runs Ubuntu 20.04 as an example.

   1. Log on to the heterogeneous computing resource that you created.

   2. Install the GPU driver.

      1. Driver version: 535.154.05, CUDA version: 12.2.

   3. Install the NVIDIA Container Toolkit. For more information, see the NVIDIA Container Toolkit Installation Guide.

   4. Obtain the AIACC container image deepytorch_t4_ubuntu20.04.tar.gz. Using a container simplifies environment configuration compared to deploying on a virtual machine. Contact your Alibaba Cloud account manager to obtain the image.

   5. Load the container image. You can also use Kubernetes to create and manage container instances.

      [root]# docker load -i deepytorch_t4_ubuntu20.04.tar
      [root]# docker images
      REPOSITORY   TAG       IMAGE ID       CREATED      SIZE
      <none>       <none>    7aef27446ff0   3 days ago   22.6GB

   6. Create a container to deploy the proprietary AIACC inference acceleration engine.

      #!/bin/bash
      /usr/bin/docker run --runtime=nvidia -ti -d --gpus all --network=host \
                 7aef27446ff0 sleep 86400000

3. Deploy the open-source TensorRT toolkit by using a container.

   1. Install the NVIDIA GPU driver.

   2. Install the NVIDIA Container Toolkit.

   3. Download the TensorRT toolkit container image. For download links, see the NVIDIA PyTorch Release Notes.

      Note

      The versions of components in the image, such as CUDA and TensorRT, must be compatible with the installed GPU driver version.

   4. Create a container to deploy the TensorRT toolkit environment.

Step 5: Associate resources

Associate resources by using the ENS console or by calling API operations to complete the basic environment setup. This reference architecture supports online AI inference use cases, such as image classification, object detection, speech recognition, and semantic analysis.