Access LLMs - Intelligent Media Services - Alibaba Cloud Documentation Center

Real-time workflows allow you to access large language models (LLMs) based on the specified specifications.

Access self-developed LLMs that comply with OpenAPI specifications

You can integrate LLMs that comply with OpenAPI specifications into your workflow based on the OpenAPI specifications. You can request LLMs that comply with OpenAPI specifications only in streaming mode.

To integrate an LLM into a workflow, set the Select Model parameter to Access Self-developed Model (Based on OpenAPI Specifications) and configure the following parameters in the configuration panel of the LLM node.

Parameter	Type	Required	Description	Example
ModelId	String	Yes	The model name. This parameter corresponds to the model field in the OpenAPI specifications.	abc
API-KEY	String	Yes	The authentication information. This parameter corresponds to the api_key field in the OpenAPI specifications.	AUJH-pfnTNMPBm6iWXcJAcWsrscb5KYaLitQhHBLKrI
HTTPS URL of Destination Model	String	Yes	The service request URL. This parameter corresponds to the base_url field in the OpenAPI specifications.	http://www.abc.com

When the real-time workflow is running, the OpenAPI specifications data is assembled in a POST request and used to access the HTTPS URL of the self-developed model that you configure to obtain the corresponding result. The following table describes the input parameters.

Parameter	Type	Description	Example
messages	Array	The context of historical conversations. A maximum of 20 context records can be retained. A context record at the top of the array indicates an early question or answer.	[{'role': 'user', 'content': 'What is the weather like today? '},{'role': 'assistant', 'content': 'It is sunny today. '},{'role': 'user', 'content': 'What will the weather be like tomorrow? '}]
model	String	The model name.	abc
stream	Boolean	Specifies whether to access the model in streaming mode. Only the streaming mode is supported.	True
extendData	Object	The supplementary information.	{'instanceId':'68e00b6640e*****3e943332fee7','channelId':'123','userData':'{"aaaa":"bbbb"}'}
instanceId	String	The instance ID.	68e00b6640e*****3e943332fee7
channelId	String	The channel ID.	123
userData	String	The value of the UserData field that is passed when the instance is started.	{"aaaa":"bbbb"}

Access self-developed LLMs that comply with Alibaba Cloud specifications

To integrate your self-developed LLM into a workflow, you must encapsulate your LLM by using an HTTP service that can be accessed over the Internet based on the specified input and output specifications.

To integrate an LLM into a workflow, set the Select Model parameter to Access Self-developed Model (Based on Alibaba Cloud Specifications) and configure the following parameters in the configuration panel of the LLM node.

Parameter	Type	Required	Description	Example
HTTPS URL of Destination Model	String	Yes	The HTTPS URL of the self-developed LLM.	https://www.abc.com
Token	String	No	The service verification token.	AUJH-pfnTNMPBm6iWXcJAcWsrscb5KYaLitQhHBLKrI

When the real-time workflow is running, the form data is assembled in a POST request and used to access the HTTPS URL of the self-developed model that you configure to obtain the corresponding result. The following table describes the input parameters.

Parameter	Type	Required	Description	Example
Message	String	Yes	The context of historical conversations. A maximum of 20 context records can be retained. A context record at the top of the array indicates an early question or answer.	[{'role': 'user', 'content': 'What is the weather like today? '},{'role': 'assistant', 'content': 'It is sunny today. '},{'role': 'user', 'content': 'What will the weather be like tomorrow? '}]
Token	String	No	The service verification token.	AUJH-pfnTNMPBm6iWXcJAcWsrscb5KYaLitQhHBLKrI
ExtendData	String	Yes	The additional information about the self-developed LLM, which includes the instance ID and the value of the UserData field that is passed when the instance is started.	{ "InstanceId": "68e00b6640e*****3e943332fee7", "UserData": "{\"aaaa\":\"bbbb\"}" }

Assemble a response in the JSON format and return the response to the orchestration service. The following table describes the input parameters.

Parameter	Type	Required	Description	Example
Text	string	Yes	The response text of the LLM.	"It is sunny today."

Sample code:

{
	"Text": "Hello. I am your AI assistant."
}

Servers of self-developed LLMs that comply with Alibaba Cloud specifications

Python

import json
from aiohttp import web


async def handle(request):
    data = await request.post()
    message = data.get('Message', "")
    token = data.get('Token', None)
    extend_data = data.get('ExtendData', "")
    print(f"message:{message}, token:{token}, extend_data:{extend_data}")
    # TODO:1. Verify whether the token is valid.
    # TODO:2. Process the message and obtain the processing result of the LLM.

    result = "Hello. I am your AI assistant."
    # Assemble the expected response of the LLM node.
    resp = {
        "Text": result
    }
    return web.Response(text=json.dumps(resp))

app = web.Application()
app.add_routes([web.post('/', handle)])

if __name__ == '__main__':
    web.run_app(app)

References

Create and manage a workflow template