Content

Target customers: Enterprises that need to call LLM APIs The AI gateway feature helps enterprises solve the following problems: 1. Inability to use the LLMs of different suppliers at the same time 2. Security risks facing internal data of enterprises 3. Regulatory compliance risks when enterprises use LLM APIs, especially models built by enterprises themselves or provided by an overseas supplier 4. Low success rate of LLM service calling 5. Long response time and high cost of LLM service calling 6. Difficulty in fine-grained control for callers and scenarios New features/specifications: As a unified LLM service portal for enterprises, the AI gateway feature provides the following core capabilities: 1. Support for AI proxy. One set of API standards (compatible with openAI) can be used to uniformly proxy multiple LLM services. Commercial LLM services such as Alibaba Cloud Model Studio and OpenAI, and user-built model services such as Ollama are supported. 2. Support for automatic routing to different LLM services based on model name or request ratio 3. Support for automatic fallback to other LLM services when one LLM service is unavailable 4. Support for AI API debugging in the console 5. Support for token-based throttling for AI calls based on request characteristics 6. Integration with Alibaba Cloud observability servuces to provide statistics and log observation for AI requests 7. Integration with Alibaba Cloud Content Moderation to review requests and responses of AI calls to meet the compliance and data security demands of enterprises 8. Support for caching responses to reduce the cost of LLM service calling