Platform For AI:EAS FAQ

A service enters the Failed state in the following two scenarios:

• During service deployment: If a specified resource, such as a model address, does not exist during deployment, the cause of the error is displayed in the service's current status information. You can usually determine the cause of the deployment failure from this error message.

• During service startup: The service fails to start after it is deployed and scheduled to resources. In this case, the status message Instance <network-test-5ff76448fd-h9dsn> not healthy: Instance crashed, please inspect instance log. appears.

  This status message indicates that a service instance failed to start. You must check the failed status in the Service Instances list on the service's Overview page to determine the specific cause. The following are possible causes for instance failure:

  • The service instance was terminated by the system due to an out-of-memory (OOM) error during startup. In this case, you must increase the service memory and redeploy the service. The following figure shows the instance status:

  • A service may crash because of a code error during startup. In this case, the Last Status is Error (error code). Click the Log button in the Actions column for the instance to check the service logs and identify the cause of the failure. The following figure shows the instance status:

  • The service image failed to be pulled. For more information, see What do I do if an image fails to be pulled (ImagePullBackOff)?.

Problem description: An EAS service automatically restarts some time after it is stopped.

Cause:

This occurs because Auto Scaling is configured for the service, and the minimum number of instances is set to 0. After a period with no traffic, the number of instances is automatically scaled in to 0. If a request arrives when no instances are available, a scale-out is automatically triggered, even if the configured scale-out metric threshold is not met.

You can determine whether a scale-out was automatically triggered based on the auto scaling description in the deployment events.

Solution:

• If you no longer need the service, you can delete it.

• If you do not want to delete the service, you can manually stop it by clicking Stop in the console or by calling the StopService API operation. A manually stopped service cannot be scaled out by traffic.

• If you do not want the service to be automatically stopped due to elastic scaling, do not set the minimum number of instances to 0.

• You can also disable Auto Scaling as needed to prevent unexpected traffic from triggering a scale-out.

Problem description:

PAI-EAS reports the following error upon startup:

[2024-10-21 20:59:33] serialize_file(_flatten(tensors), filename, metadata=metadata)

[2024-10-21 20:59:33] safetensors_rust.SafetensorError: Error while serializing: IoError(Os { code: 28, kind: StorageFull, message: "No space left on device" })

[2024-10-21 20:59:35] time="2024-10-21T12:59:35Z" level=info msg="program stopped with status:exit status 1" program=/bin/sh

Cause: The system disk of the EAS instance is full because of too many model files, which prevents the service from starting properly.

Solution:

Solution 1: Scale out the system disk for the EAS instance.

Solution 2: If the model files are too large, you can store them in external storage, such as OSS or NAS, and read them using storage mounts.

The exec format error indicates that the operating system cannot execute the target program file. The most common cause is that the CPU architecture of the executable file or container image is incompatible with the host system architecture.

You can try switching to a different resource specification.

To maximize communication efficiency between multiple GPUs, the number of GPUs specified for a single service must be a power of 2.

You can allocate 0, 1, 2, 4, 8, or 16 GPUs.

If you see ImagePullBackOff as the Last Exit Reason in the service instance list, the image pull has likely failed. If the following icon appears in the Status column, you can click it to view the specific cause.

Common causes for image pull failures include the following:

The following are recommendations based on the regions of the image service and the EAS service:

• Same region: We recommend that you use the internal endpoint of the image to pull it.

• Different regions: For ACR Personal Edition, you can only use the public endpoint of the image. For ACR Enterprise Edition, you can choose one of the following options:

  • If you have high requirements for security and stability, you can use the internal endpoint of the image. You must connect the VPCs using CEN. For more information, see Access an ACR Enterprise instance from another region or an IDC.

  • If the business scenario is simple or you cannot connect to the internal network, you can use the public endpoint of the image as a temporary solution. The download speed over the public internet is slow.

Note the following points about ACR Enterprise instances:

• You must configure access control for VPCs and the public internet as needed.

• If the repository is not configured for public anonymous pulls, you must configure the username and password for the image repository in the EAS service when you pull the image from another region using the public endpoint.

No. PAI official images are internal platform images. You can use them only on the PAI platform. You cannot download them outside the platform's containers.

This is usually because resources are insufficient in the current region.

For subscription machine instances, if creation fails due to insufficient resources, the system automatically creates a refund order. The paid amount is returned to your account.

You can go to the Alibaba Cloud Unsubscribe page to unsubscribe from unused EAS subscription dedicated machines. On the page, perform the following steps:

• For Type, select Partial Unsubscribe.

• For Product Name, select EAS Dedicated Machine Subscription.

Click Search to find the resource you want to unsubscribe from. Then, click Unsubscribe Resource in the Actions column and follow the instructions in the console to complete the process.

No, service instance data is not retained.

To prevent issues, the 1-core, 2 GB resource specification is no longer available. EAS deploys system components on each machine node, and these components consume some machine resources. If the machine specification is too small, the proportion of resources occupied by system components is too high, which reduces the proportion of resources available for your use.

The maximum number of service instances that you can deploy in EAS depends on the remaining resource usage. You can view the remaining usage in the machine list of the resource group in the console. For more information, see Use EAS resource groups.

If you allocate tasks based on the number of CPU cores, the maximum number of instances you can deploy is (Number of CPU cores - 1) / Number of cores used by each instance.

The ecs.gn8ia-2x.8xlarge specification has performance similar to a 4090 graphics card.

The maximum concurrency of a model service is related to multiple factors, such as the model, scenario, and resource configuration. We recommend that you use automatic stress testing to determine the service performance.

No. EAS does not support remote SSH connections. You cannot enter the container to debug. To execute commands, we recommend that you configure them in the run command.

EAS services can have the following statuses. You can also go to the Elastic Algorithm Service (EAS) page and view the Service Status column.

• Creating: The service is being created.

• Waiting: The service is waiting for instances to start.

• Stopped: The service is stopped.

• Failed: The service has failed.

• Updating: The service is being updated. Instances will be updated.

• Stopping: The service is being stopped.

• HotUpdate: The service is being updated. This is a hot update and instances are not updated.

• Starting: The service is starting.

• DeleteFailed: The service failed to be deleted.

• Running: The service is running.

• Scaling: The service is being updated. Instances are being scaled.

• Pending: The service is waiting for a specific action.

• Deleting: The service is being deleted.

• Completed: The task is complete.

• Preparing: The service is being prepared.

For more information about status codes, see Appendix: Service status codes and common errors.

Yes, it does. You can simply replace http with https for more secure, encrypted transmission. Make sure that the calling environment supports HTTPS certificate validation. If you encounter an SSL Certificate validation error in the client, such as with Python requests, this is usually a client environment configuration issue and not an issue with the EAS service.

You cannot block HTTP access on a shared gateway. For a dedicated gateway, you can enable HTTPS Redirection. This redirects all HTTP requests to the HTTPS protocol.

Yes, you can. You need to use a dedicated gateway. For more information, see Invoke a service using a dedicated gateway.

No, it does not. After a service is deployed, the token is valid for a long time. Restarting the service does not change the token. The token changes only if you manually reset it by modifying it with custom authentication or if you delete the service.

No, you cannot. An EAS service instance supports only one fixed authentication token. You cannot create multiple tokens for a single service for access control or separate metering. To implement multi-user authentication management, we recommend that you use a more complex solution, such as Alibaba Cloud RAM authentication.

The EAS service itself cannot be configured for default streaming responses during deployment. You must explicitly specify that you want streaming output in the body of each API call request.

For example, when you call an LLM service, you can add the "stream": true parameter to the JSON request body.

• VPC endpoint invocation: This method uses an internal-facing SLB and a gateway. This is a classic request model. In this model, requests are forwarded through Layer 4 of SLB and Layer 7 of the gateway to reach the service instance. In high-traffic and high-concurrency scenarios, this forwarding causes some performance overhead. The gateway also has a bandwidth limit, which is 1 Gbps by default.

• VPC direct connection: EAS provides a high-speed direct connection access mode that resolves performance and extensibility issues at no extra cost. After you enable VPC direct connection, a network path is established between your VPC and the EAS service VPC. Your requests use the service discovery feature provided by EAS to locate the service and then initiate load-balanced service requests in the client code. This process requires you to use the SDK provided by EAS and set endpoint_type to DIRECT.

  For example, in the scenario described in Python SDK usage instructions, you can add the following line of code to switch from gateway invocation to direct connection:

  client = PredictClient('http://pai-eas-vpc.cn-hangzhou.aliyuncs.com', 'mnist_saved_model_example')
  client.set_token('M2FhNjJlZDBmMzBmMzE4NjFiNzZhMmUxY2IxZjkyMDczNzAzYjFi****')
  client.set_endpoint_type(ENDPOINT_TYPE_DIRECT) # Direct link
  client.init()

After an EAS online service is successfully deployed, you can use the curl command to call the service using its public or VPC endpoint. The procedure is as follows:

1. Obtain the service endpoint and token.

   1. On the Elastic Algorithm Service (EAS) page, click the target service to go to its overview page.

   2. In the Basic Information section, click View Invocation Information.

   3. In the Invocation Information dialog box, on the Public Endpoint Invocation or VPC Endpoint Invocation tab, obtain the service endpoint and token.

2. Use the curl command to call the service.

   Example:

   $ curl <service_url> -H 'Authorization: <service_token>' -d '[{"sex":0,"cp":0,"fbs":0,"restecg":0,"exang":0,"slop":0,"thal":0,"age":0,"trestbps":0,"chol":0,"thalach":0,"oldpeak":0,"ca":0}]'

   Where:

   • <service_url>: Replace this with the service endpoint you obtained.

   • <service_token>: Replace this with the token you obtained.

   • -d: Use this to configure the service request data.