DataWorks:Use large language models

Use a large language model node

The large language model node in Data Studio (new version) provides a dedicated interface for configuring and calling an LLM service. In the large language model node, you can configure a large language model service and call the model.

Use a Shell node

This example calls an LLM service from a Shell node to answer a question.

1. Create a Shell node and add the following command. Replace <endpoint> and <api-key> with the values from Set up before calling.

   curl -X POST http://<endpoint>/v1/completions \
     -H "Authorization: <api-key>" \
     -H "Content-Type: application/json" \
     -d '{"prompt":"What are the differences and connections between AI, machine learning, and deep learning?", "stream":"false", "max_tokens": 1024}' -v

2. In the Run Configuration section on the right, select the resource group whose network connectivity to the model service is configured.

3. Click Run.

Schedule or publish this node:

• Scheduling: In the Scheduling section, select the same resource group and configure scheduling properties under Scheduling Policies.

• Publishing: Click the icon to publish the node. A node runs on a schedule only after it is published to the production environment. See Node scheduling configuration and Node publishing.

Use a Python node

This example calls an LLM service from a Python node with streaming output.

Step 1: Create a custom image with the `requests` library.

This example requires the Python requests library. Create a custom image based on an official DataWorks image using these parameters:

Step 2: Create a Python node with the following code.

Create a Python node and add this code. Replace <endpoint> and <api-key> with the values from Set up before calling.

import requests
import json
import time
import sys

# Replace these values
httpUrl = "http://<endpoint>"
apikey = "<api-key>"

url = httpUrl + "/v1/completions"
headers = {
    "Authorization": apikey,
    "Content-Type": "application/json"
}
data = {
    "prompt": "Please write a poem about spring",
    "stream": True,
    "max_tokens": 512
}

try:
    response = requests.post(url, headers=headers, json=data, stream=True)
    response.raise_for_status()

    full_text = ""  # Accumulate the full response to prevent loss.
    buffer = ""     # Used to handle incomplete JSON lines (optional).

    for line in response.iter_lines():
        if not line:
            continue  # Skip empty lines.

        line_str = line.decode('utf-8').strip()

        if line_str.startswith("data:"):
            data_str = line_str[5:].strip()  # Remove "data: ".

            if data_str == "[DONE]":
                print("\n[Stream response ended]")
                break

            try:
                parsed = json.loads(data_str)
                choices = parsed.get("choices", [])
                if choices:
                    delta_text = choices[0].get("text", "")
                    if delta_text:
                        full_text += delta_text
                        for char in delta_text:
                            print(char, end='', flush=True)
                            sys.stdout.flush()
                            time.sleep(0.03)  # Typewriter effect.

            except json.JSONDecodeError:
                continue

    print(f"\n\n[Full response length: {len(full_text)} characters]")
    print(f"[Full content]:\n{full_text}")

except requests.exceptions.RequestException as e:
    print(f"Request failed: {e}")
except Exception as e:
    print(f"Error: {e}")

Step 3: Run the node.

In the Run Configuration section on the right, select the resource group with network connectivity configured and the custom Image from Step 1.

Click Run.

Schedule or publish this node:

• Scheduling: In the Scheduling Configuration section, select the same resource group and image, then configure properties under Scheduling Policy.

• Publishing: Click the icon to publish. A node runs on a schedule only after it is published to the production environment. See Node scheduling configuration and Node publishing.

Use a large language model node

The large language model node in Data Studio (new version) supports vector models. Configure the model service in the node and call the vector model directly.

Use a Shell node

This example converts text to a vector using BGE-M3 from a Shell node.

1. Create a Shell node and add the following command. Replace <endpoint> and <api-key> with the values from Set up before calling.

   curl -X POST "http://<endpoint>/v1/embeddings" \
     -H "Authorization: <api-key>" \
     -H "Content-Type: application/json" \
     -d '{
       "input": "This is a piece of text that needs to be converted into a vector",
       "model": "bge-m3"
     }'

2. In the Run Configuration section on the right, select the resource group whose network connectivity to the model service is configured.

3. Click Run.

Schedule or publish this node:

• Scheduling: In the Scheduling section, select the same resource group and configure scheduling properties under Scheduling Policies.

• Publishing: Click the icon to publish. A node runs on a schedule only after it is published to the production environment. See Node scheduling configuration and Node publishing.

Use a Python node

This example converts text to a vector using BGE-M3 from a Python node.

Step 1: Create a custom image with the `requests` library.

Follow the same steps as in the LLM Python node example to create a custom image with the requests library installed.

Step 2: Create a Python node with the following code.

Create a Python node and add this code. Replace <endpoint> and <api-key> with the values from Set up before calling.

import requests
import json

# Replace these values
api_url = "http://<endpoint>/v1/embeddings"
token = "<api-key>"

headers = {
    "Authorization": token,
    "Content-Type": "application/json"
}

payload = {
    "input": "Test text",
    "model": "bge-m3"
}

try:
    response = requests.post(api_url, headers=headers, data=json.dumps(payload))
    print("Response status code:", response.status_code)
    print("Response content:", response.text)
except Exception as e:
    print("Request exception:", e)

Step 3: Run the node.

In the Run Configuration section on the right, select the resource group with network connectivity configured and the custom Image from Step 1.

Click Run.

Schedule or publish this node:

• Scheduling: In the Scheduling Configurations section, select the same resource group and image, then configure properties under Scheduling Policies.

• Publishing: Click the icon to publish. A node runs on a schedule only after it is published to the production environment. See Node scheduling configuration and Node publishing.