New Features

API Gateway - AI gateway available in Cloud-native API Gateway

Dec 26 2024

API Gateway
Cloud-native API Gateway provides the AI gateway feature. This feature provides the model service forwarding, content moderation, AI observation, AI throttling, and AI caching capabilities and can function as a unified portal for accessing Large Language Model (LLM) services for enterprises.
Content

Target customers: Enterprises that need to call LLM APIs The AI gateway feature helps enterprises solve the following problems: 1. Inability to use the LLMs of different suppliers at the same time 2. Security risks facing internal data of enterprises 3. Regulatory compliance risks when enterprises use LLM APIs, especially models built by enterprises themselves or provided by an overseas supplier 4. Low success rate of LLM service calling 5. Long response time and high cost of LLM service calling 6. Difficulty in fine-grained control for callers and scenarios New features/specifications: As a unified LLM service portal for enterprises, the AI gateway feature provides the following core capabilities: 1. Support for AI proxy. One set of API standards (compatible with openAI) can be used to uniformly proxy multiple LLM services. Commercial LLM services such as Alibaba Cloud Model Studio and OpenAI, and user-built model services such as Ollama are supported. 2. Support for automatic routing to different LLM services based on model name or request ratio 3. Support for automatic fallback to other LLM services when one LLM service is unavailable 4. Support for AI API debugging in the console 5. Support for token-based throttling for AI calls based on request characteristics 6. Integration with Alibaba Cloud observability servuces to provide statistics and log observation for AI requests 7. Integration with Alibaba Cloud Content Moderation to review requests and responses of AI calls to meet the compliance and data security demands of enterprises 8. Support for caching responses to reduce the cost of LLM service calling

7th Gen ECS Is Now Available

Increase instance computing power by up to 40% and Fully equipped with TPM chips.
Powered by Third-generation Intel® Xeon® Scalable processors (Ice Lake).

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.