Platform For AI:Call services over a VPC direct connection

Use cases

• High-performance requirements: For high-traffic services with a large number of queries per second (QPS), such as image or speech recognition, a direct connection can significantly improve access performance.

• Latency-sensitive applications: For services that require fast response times, such as online recommendation and financial risk control, a direct connection can effectively reduce latency.

How it works

A VPC direct connection relies on two mechanisms:

• Network connectivity: After you configure a VPC for a service, PAI-EAS creates and attaches a free auxiliary elastic network interface for each service instance within the specified VPC and vSwitch. This establishes a direct network path between the VPC and the PAI-EAS service instances. As a result, clients within the VPC can directly access PAI-EAS service instances without passing through a gateway, avoiding Layer 4 SLB and Layer 7 network forwarding.

• Service discovery: PAI-EAS provides a service discovery mechanism that allows clients to retrieve a real-time list of IP:PORT pairs for all service instances. Clients can use this list to implement load balancing and failover.

Cross-region access and billing

Q: What is the difference between cross-region VPC internal access and public network access?

The key difference depends on your gateway type:

• Shared gateway: You only need to establish network connectivity between the two VPCs (for example, using CEN or VPC peering). No additional PAI-EAS gateway configuration is required.

• Dedicated gateway: In addition to establishing VPC connectivity, you must perform additional CEN-related configuration steps in PAI-EAS. For more information, see Cross-VPC access to a dedicated gateway using CEN.

Q: Does cross-region VPC internal access incur traffic costs? Why is it still recommended?

Yes, cross-region internal network traffic incurs fees. Despite this, VPC internal access is recommended for production environments for the following reasons:

• Security: Traffic does not traverse the public internet, eliminating public exposure risk.

• Performance: Lower latency and more stable connections compared to public network access.

• Cost efficiency: Cross-region internal network traffic costs are typically lower than public network egress traffic costs.

Stable endpoint configuration for multi-replica services

When your service scales to multiple replicas, backend instance IPs may change. Use the following approaches to maintain a stable connection address.

Underlying principle: PAI-EAS uses SLB (Server Load Balancer) to expose a unified Service Endpoint for each service. Regardless of how many backend replicas are running, clients only need to connect to this single address — you do not need to track individual instance IP changes.

• Option 1 (Recommended): Use the public domain with a backend IP allowlist. Configure the backend to allow traffic only from your peer VPC's IP range. This provides a stable domain address without additional DNS configuration.

• Option 2: Configure Alibaba Cloud PrivateZone in your peer VPC to resolve a custom domain to the PAI-EAS private IP address. Note that if the IP address changes, you must update the DNS record or integrate a service discovery mechanism to keep resolution current.

Calling other EAS services from an EAS container

If your EAS service needs to call another EAS service from within its container, note the following:

• 401 errors or connection failures: These typically occur when a container application calls another EAS service using a public endpoint. The usual causes are network restrictions in the container environment or incorrect token configuration.

• Recommended approach: Use the VPC internal address for inter-service calls. This ensures connectivity and improves security.

• Alternative: If you must use a public endpoint, configure a dedicated gateway. Note that this involves additional ALB costs.

• Authentication requirement: Regardless of which address type you use, every request must include a valid Authorization token in the request header.

Core logic

When you use a VPC direct connection, the client bypasses the gateway and connects directly to service instances. This means the client is responsible for handling service discovery, load balancing, and failover retry. The official PAI-EAS SDKs encapsulate this logic for you:

• Service discovery: The SDK starts a background thread during initialization to periodically call the PAI-EAS service discovery API. It retrieves and caches a list of backend instance IP:PORT pairs and weights. If the API call fails, the SDK continues to use the last cached list to ensure availability.

• Load balancing: For each request, the SDK selects a target instance from the local cache using a weighted round-robin algorithm. The weights are determined by the instance weights returned from the service discovery API.

• Failover retry: If a request fails due to a network error, connection timeout, or 5xx response, the SDK retries the request on a different instance from the local cache. In a multi-instance deployment, this prevents a single faulty instance from causing request failures.

If you plan to build a custom client without using an SDK, you must implement equivalent capabilities. For more information, see Build a custom client.

Examples

package main

import (
        "fmt"
        "github.com/pai-eas/eas-golang-sdk/eas"
)

func main() {
    // Use the endpoint in the format {Uid}.vpc.{RegionId}.pai-eas.aliyuncs.com. Replace {Uid} with your account ID and specify the service region and name. For example, the region ID for China (Shanghai) is cn-shanghai.
    client := eas.NewPredictClient("123**********.vpc.cn-shanghai.pai-eas.aliyuncs.com", "scorecard_pmml_example")
    // Replace with your service token. You can find the token under Invocation Information on the service details page.
    client.SetToken("YWFlMDYyZDNmNTc3M2I3MzMwYmY0MmYwM2Y2MTYxMTY4NzBkNzdj****")
    client.SetEndpointType(eas.EndpointTypeDirect)
    client.Init()
    req := "[{\"fea1\": 1, \"fea2\": 2}]"
    for i := 0; i < 100; i++ {
        resp, err := client.StringPredict(req)
        if err != nil {
            fmt.Printf("failed to predict: %v\n", err.Error())
        } else {
            fmt.Printf("%v\n", resp)
        }
    }
}

Service discovery API

PAI-EAS provides an HTTP API for service discovery within the configured VPC. The API returns a list of IP addresses, ports, and weights for all backend instances of a service.

• URL: http://<Your Alibaba Cloud Account ID>.vpc.<RegionID>.pai-eas.aliyuncs.com/exported/apis/eas.alibaba-inc.k8s.io/v1/upstreams/<Service Name>

• Authentication: No token is required. The API is accessible only from within the VPC configured for the service.

• Polling interval: This is a side-channel API and should be called periodically by a background task, for example, every 5 to 10 seconds. Do not call this API for every inference request. Doing so severely degrades performance.

Example:

The following example shows how to query a service named mnist_saved_model_example that is deployed in China (Hangzhou) and has two instances.

$curl http://123**********.vpc.cn-hangzhou.pai-eas.aliyuncs.com/exported/apis/eas.alibaba-inc.k8s.io/v1/upstreams/mnist_saved_model_example

Replace 123********** with your account ID.

The following response shows a list of backend service addresses:

{
  "correlative": [
    "mnist_saved_model_example"
  ],
  "endpoints": {
    "items": [
      {
        "app": "mnist-saved-model-example",
        "ip": "172.16.XX.XX",
        "port": 50000,
        "weight": 100
      },
      {
        "app": "mnist-saved-model-example",
        "ip": "172.16.XX.XX",
        "port": 50000,
        "weight": 100
      }
    ]
  }
}

Troubleshooting: VPC domain access failures

Symptom: The VPC domain is unreachable, but the service is accessible via the direct IP address returned by the service discovery API.

Common cause: Incorrect curl command syntax — for example, the URL is not enclosed in quotes, which causes the shell to misinterpret special characters such as & or ? in the URL.

Solution: Enclose the URL in double quotes and run a minimal test request via the IP address or VPC domain to verify connectivity:

# Test using the VPC domain (URL must be enclosed in double quotes)
curl "http://123**********.vpc.cn-hangzhou.pai-eas.aliyuncs.com/api/predict/your-service-name" \
  -H "Authorization: your-token"

# Test using a direct IP address (replace with an IP from the service discovery API response)
curl "http://172.16.XX.XX:50000/api/predict/your-service-name" \
  -H "Authorization: your-token"