全部產品
Search
文件中心

Alibaba Cloud Model Studio:Qwen-Omni-Realtime

更新時間:Nov 19, 2026

透過 WebSocket、WebRTC 使用 Qwen-Omni-Realtime 理解文本、圖片、音訊和視訊,並生成語音回覆,適合語音聊天、智能客服等場景。

支援的地區:新加坡、北京,需使用各地區的API Key。

模型與接入方式

即時音訊與視訊對話優先使用 Qwen3.8-Omni-Flash-Realtime。模型支援音訊與視訊雙工交互,接收文本、圖片、流式音訊和視訊輸入,返回文本與音訊。

模型的名稱、上下文、價格、快照版本等資訊請參見百鍊控制台;併發限流條件請參考限流。

Qwen-Omni-Realtime 模型支援 WebSocket、WebRTC 和 AOQ 三種協議接入。WebSocket 適合服務端集成和快速接入;WebRTC 適合瀏覽器端、低延遲語音場景,音訊透過 UDP 直接傳輸,內置回聲消除和降噪。如果是用戶端對接,且更看重穩定的延遲、弱網下的交互能力、即時雙工的降噪與回聲消除,可優先考慮 AOQ。協議對比與選型請參見API 參考。

按當前任務選擇入口:

接入準備

獲取所選地域、業務空間對應的 API Key 和業務空間 ID,並將 API Key 配置到環境變量。qwen3.8-omni-flash-realtime 必須使用下方綁定業務空間的呼叫地址。

WebSocket

配置項說明

呼叫地址

新加坡地域:wss://WorkspaceId.ap-southeast-1.maas.aliyuncs.com/api-ws/v1/realtime

呼叫時請將WorkspaceId替換為真實的業務空間ID。

查詢參數

查詢參數為model,需指定為訪問的模型名。示例:?model=qwen3.8-omni-flash-realtime

請求頭

使用 Bearer Token 鑒權:Authorization: Bearer DASHSCOPE_API_KEY

DASHSCOPE_API_KEY 是您在百鍊上申請的API Key。

WebRTC

配置項

說明

請求地址

新加坡地域:POST https://WorkspaceId.ap-southeast-1.maas.aliyuncs.com/api/v1/webrtc/realtime

呼叫時請將WorkspaceId替換為真實的業務空間ID。

查詢參數

查詢參數為model,需指定為訪問的模型名。示例:?model=qwen3.8-omni-flash-realtime;舊型號示例:?model=qwen3.5-omni-plus-realtime

Content-Type

application/sdp

請求頭

Authorization: Bearer DASHSCOPE_API_KEY

請求體

用戶端生成的 Offer SDP 字串

響應

成功:HTTP 200,返回服務端 Answer SDP 字串。失敗:HTTP 4xx,返回 JSON 錯誤資訊。

AOQ

透過業務 AppServer 發起 HTTP 請求獲取 AOQ 連接參數,具體步驟參見鑒權與連接憑證。

快速開始:最小音訊問答

本例使用 WebSocket 和 Manual 模式,從本地音訊文件完成一次問答。需要持續錄音或瀏覽器交互時,參見即時對話示例。

先配置 API Key,安裝 websocket-client。在下方 Python 範例中選擇地域(cn-beijing 或 ap-southeast-1),填寫對應地址中的業務空間 ID。將 PCM_FILE 設為一段短的 16 kHz、16-bit little-endian、單聲道、無文件頭 PCM 語音文件路徑。

示例使用 Manual 模式:等待會話配置生效後發送音訊、提交緩衝區,再請求回覆。文本打印到終端,音訊寫入 reply.pcm(24 kHz PCM,非 WAV 文件)。

在 macOS 或 Linux 終端中執行以下命令。

python3 -m pip install websocket-client
export PCM_FILE="./input.pcm"

如果已有單聲道、16 kHz、16-bit PCM 編碼的 input.wav,可用 Python 提取 PCM 音訊。此步驟檢查格式,不進行重採樣。

import wave
from pathlib import Path

with wave.open("input.wav", "rb") as source:
    if (source.getnchannels(), source.getframerate(), source.getsampwidth(),
            source.getcomptype()) != (1, 16000, 2, "NONE"):
        raise ValueError("Expected mono 16 kHz, 16-bit PCM WAV audio.")
    Path("input.pcm").write_bytes(source.readframes(source.getnframes()))

準備好輸入後,運行以下用戶端。示例使用“龍安靈心”音色(longanlingxin):

import base64
import json
import os
from pathlib import Path
import websocket

url = "wss://{WorkspaceId}.ap-southeast-1.maas.aliyuncs.com/api-ws/v1/realtime?model=qwen3.8-omni-flash-realtime"
audio = Path(os.environ["PCM_FILE"]).read_bytes()
if not audio or len(audio) % 2:
    raise ValueError("Provide non-empty 16-bit mono PCM audio.")

ws = websocket.create_connection(
    url,
    header=["Authorization: Bearer " + os.environ["DASHSCOPE_API_KEY"]],
    timeout=30,
)

def receive():
    event = json.loads(ws.recv())
    if event["type"] == "error":
        raise RuntimeError(event.get("error"))
    return event

try:
    while receive()["type"] != "session.created":
        pass
    ws.send(json.dumps({
        "type": "session.update",
        "session": {
            "modalities": ["text", "audio"],
            "turn_detection": None,
            "audio": {
                "input": {"format": {
                    "type": "pcm", "sample_rate": 16000,
                    "sample_format": "s16le", "channels": 1,
                    "packing": "interleaved", "channel_layout": "mono",
                }},
                "output": {
                    "voice": "longanlingxin",
                    "format": {"type": "pcm", "sample_rate": 24000},
                },
            },
        },
    }))
    while receive()["type"] != "session.updated":
        pass
    for offset in range(0, len(audio), 3200):
        ws.send(json.dumps({
            "type": "input_audio_buffer.append",
            "audio": base64.b64encode(audio[offset:offset + 3200]).decode("ascii"),
        }))
    ws.send(json.dumps({"type": "input_audio_buffer.commit"}))
    while receive()["type"] != "input_audio_buffer.committed":
        pass
    ws.send(json.dumps({"type": "response.create"}))
    with open("reply.pcm", "wb") as output:
        while True:
            event = receive()
            if event["type"] == "response.audio.delta":
                output.write(base64.b64decode(event["delta"]))
            elif event["type"] in ("response.audio_transcript.delta", "response.text.delta"):
                print(event["delta"], end="", flush=True)
            elif event["type"] == "response.done":
                print("\nResponse status:", event["response"]["status"])
                break
finally:
    ws.close()

成功收到回覆後,將 reply.pcm 包裝為 WAV 文件,再使用音訊播放器播放:

import wave
from pathlib import Path

with wave.open("reply.wav", "wb") as target:
    target.setnchannels(1)
    target.setsampwidth(2)
    target.setframerate(24000)
    target.writeframes(Path("reply.pcm").read_bytes())

即時對話示例

WebSocket 示例提供 VAD 和 Manual 兩種模式;WebRTC 使用 VAD。

WebSocket

DashScope Python SDK

  • 準備運行環境

您的 Python 版本需要不低於 3.10。

首先根據您的操作系統安裝 pyaudio。

macOS

brew install portaudio && pip install pyaudio

Debian/Ubuntu

  • 若未使用虛擬環境,可直接透過系統包管理器安裝:
sudo apt-get install python3-pyaudio
  • 若使用虛擬環境,需先安裝編譯依賴:
sudo apt update
sudo apt install -y python3-dev portaudio19-dev

然後在已激活的虛擬環境中使用 pip 安裝:

pip install pyaudio

CentOS

sudo yum install -y portaudio portaudio-devel && pip install pyaudio

Windows

pip install pyaudio

安裝完成後,透過 pip 安裝依賴:

pip install websocket-client dashscope
  • 選擇交互模式
    • VAD 模式(Voice Activity Detection,自動檢測語音起止)

      服務端自動判斷用戶何時開始與停止說話並作出回應。

    • Manual 模式(手動控制語音起止)

      用戶端控制語音起止。用戶說話結束後,用戶端需主動發送消息至服務端。

    VAD 模式

    新建一個 python 文件,命名为vad_dash.py,並將以下代碼複製到文件中:

    vad_dash.py

    # Dependencies: dashscope >= 2.0.1, pyaudio
    import os
    import base64
    import time
    
    import pyaudio
    from concurrent.futures import ThreadPoolExecutor
    
    import dashscope
    
    # 以下為新加坡(ap-southeast-1)地區的設定。請在範例中填寫實際的業務空間 ID。region 的值因地區而異。
    client = dashscope.Client(
        # If you have not set an environment variable, replace the following line with your Model Studio API key: api_key="sk-xxx",
        # API keys are region-specific. To get an API key, see https://www.alibabacloud.com/help/model-studio/get-api-key
        api_key=os.getenv("DASHSCOPE_API_KEY"),
        workspace="{WorkspaceId}",
        region="ap-southeast-1",
    )
    # Specify the voice
    voice = 'Tina'
    # Specify the model
    model = 'qwen3.8-omni-flash-realtime'
    # Specify the model persona
    instructions = "你是个人助理小云,请用幽默风趣的方式回答用户的问题"
    
    # 1. Initialize the audio device and output stream
    pya = pyaudio.PyAudio()
    out = pya.open(format=pyaudio.paInt16, channels=1, rate=24000, output=True)
    
    # 2. Receiver thread: iterate server events, play audio and print text
    def receive_events(session):
        for response in session.events():
            if response['type'] == 'response.audio.delta':
                # Play the audio
                out.write(base64.b64decode(response['delta']))
            elif response['type'] == 'conversation.item.input_audio_transcription.delta':
                # Streaming preview: text is the confirmed prefix, stash is the pending suffix
                preview = response.get('text', '') + response.get('stash', '')
                print(f"\r[User] {preview}", end='', flush=True)
            elif response['type'] == 'conversation.item.input_audio_transcription.completed':
                # Transcription completed: print the final text and a newline
                print(f"\r[User] {response['transcript']}")
            elif response['type'] == 'response.audio_transcript.done':
                # Print the assistant reply text
                print(f"[LLM] {response['transcript']}")
    
    # 3. Create the session (the connection is established automatically on entering the with context) and configure it
    with client.audio.realtime.connect(model=model) as session:
        session.update_session(
            output_modalities=['audio', 'text'],
            voice=voice,
            instructions=instructions,
        )
        # 4. Initialize the audio input stream
        mic = pya.open(format=pyaudio.paInt16, channels=1, rate=16000, input=True)
        # 5. Main loop for audio input
        print("对话已开始,对着麦克风说话 (Ctrl+C 退出)...")
        try:
            with ThreadPoolExecutor(max_workers=1) as pool:
                receiver = pool.submit(receive_events, session)
                while True:
                    audio_data = mic.read(3200, exception_on_overflow=False)
                    session.append_audio(audio_data)
                    time.sleep(0.01)
        except KeyboardInterrupt:
            # Clean up resources
            session.close()
            mic.close()
            out.close()
            pya.terminate()
            print("\n对话结束")
    

    運行vad_dash.py,透過麥克風即可與 Qwen-Omni-Realtime 模型即時對話,系統會檢測您的音訊起始位置並自動發送到服務器,無需您手動發送。

    Manual 模式

    新建一個 python 文件,命名為manual_dash.py,並將以下代碼複製進文件中:

    manual_dash.py

    # This example uses Qwen3.5-Omni-Plus-Realtime, demonstrating input transcription and session control.
    # Dependencies: dashscope >= 2.0.1, pyaudio.
    import os
    import base64
    import sys
    import threading
    
    import pyaudio
    from concurrent.futures import ThreadPoolExecutor
    
    import dashscope
    
    # 以下為新加坡(ap-southeast-1)地區的設定。請在範例中填寫實際的業務空間 ID。region 的值因地區而異。
    client = dashscope.Client(
        # If you have not set an environment variable, replace the following line with your Model Studio API key: api_key="sk-xxx",
        # API keys are region-specific. To get an API key, see https://www.alibabacloud.com/help/model-studio/get-api-key
        api_key=os.getenv("DASHSCOPE_API_KEY"),
        workspace="{WorkspaceId}",
        region="ap-southeast-1",
    )
    voice = 'Tina'
    
    # Receiver thread: iterate server events, play audio and print text
    def receive_events(session, ctx):
        try:
            for response in session.events():
                t = response['type']
                if t == 'session.created':
                    print('start session: {}'.format(response['session']['id']))
                elif t == 'conversation.item.input_audio_transcription.delta':
                    print('\rquestion: {}'.format(response.get('text', '') + response.get('stash', '')), end='', flush=True)
                elif t == 'conversation.item.input_audio_transcription.completed':
                    print()
                    ctx['transcription_done'].set()
                elif t == 'response.audio_transcript.delta':
                    print('llm text: {}'.format(response['delta']))
                elif t == 'response.audio.delta':
                    # Decode base64 directly and write it to the output stream for playback
                    if ctx['out'] is None:
                        return
                    try:
                        ctx['out'].write(base64.b64decode(response['delta']))
                    except Exception as e:
                        print('[Error] audio playback failed: {}'.format(e))
                elif t == 'response.done':
                    # Mark this turn as complete so the main loop can proceed
                    print('[Metric] response: {}, first text delay: {}, first audio delay: {}'.format(
                        session.last_response_id,
                        session.last_first_text_delay,
                        session.last_first_audio_delay,
                    ))
                    ctx['resp_done'].set()
        except Exception as e:
            print('[Error] {}'.format(e))
    
    def shutdown_ctx(ctx):
        """Safely release audio and PyAudio resources."""
        try:
            if ctx['out'] is not None:
                ctx['out'].close()
                ctx['out'] = None
        except Exception:
            pass
        try:
            if ctx['pya'] is not None:
                ctx['pya'].terminate()
                ctx['pya'] = None
        except Exception:
            pass
    
    def stream_record_and_send(pya_inst, session, sample_rate=16000, chunk_size=3200):
        stop_evt = threading.Event()
        stream = pya_inst.open(
            format=pyaudio.paInt16,
            channels=1,
            rate=sample_rate,
            input=True,
            frames_per_buffer=chunk_size
        )
    
        def _reader():
            while not stop_evt.is_set():
                try:
                    data = stream.read(chunk_size, exception_on_overflow=False)
                    session.append_audio(data)
                except Exception:
                    break
    
        t = threading.Thread(target=_reader, daemon=True)
        t.start()
        input()
        stop_evt.set()
        t.join(timeout=1.0)
        stream.close()
    
    if __name__  == '__main__':
        print('Initializing ...')
        # Runtime context: holds audio and session handles
        ctx = {'pya': None, 'out': None, 'resp_done': threading.Event(), 'transcription_done': threading.Event()}
    
        # Create the session: the connection is established automatically on entering the with context
        with client.audio.realtime.connect(
                model='qwen3.5-omni-plus-realtime',
        ) as session:
            # Initialize the speaker (24k/mono/16bit)
            ctx['pya'] = pyaudio.PyAudio()
            ctx['out'] = ctx['pya'].open(
                format=pyaudio.paInt16,
                channels=1,
                rate=24000,
                output=True
            )
            print('audio output initialized')
    
            # Session configuration: enable text + audio output (disable server-side VAD and use manual recording)
            session.update_session(
                output_modalities=['audio', 'text'],
                voice=voice,
                enable_input_audio_transcription=True,
                input_audio_transcription_model='qwen3-asr-flash-realtime',
                enable_turn_detection=False,
                instructions="你是个人助理小云,请你准确且友好地解答用户的问题,始终以乐于助人的态度回应。"
            )
    
            with ThreadPoolExecutor(max_workers=1) as pool:
                receiver = pool.submit(receive_events, session, ctx)
                try:
                    turn = 1
                    while True:
                        print(f"\n--- 第 {turn} 轮对话 ---")
                        print("按 Enter 开始录音(输入 q 回车退出)...")
                        user_input = input()
                        if user_input.strip().lower() in ['q', 'quit']:
                            print("用户请求退出...")
                            break
                        print("录音中... 再次按 Enter 停止。")
                        if ctx['pya'] is None:
                            ctx['pya'] = pyaudio.PyAudio()
                        stream_record_and_send(ctx['pya'], session)
    
                        ctx['transcription_done'].clear()
                        ctx['resp_done'].clear()
                        session.commit()
                        ctx['transcription_done'].wait(timeout=10)
                        print("等待模型回复...")
                        session.create_response()
                        ctx['resp_done'].wait()
                        turn += 1
                except KeyboardInterrupt:
                    print("\n程序被用户中断")
                finally:
                    shutdown_ctx(ctx)
                    print("程序退出")
    

    運行manual_dash.py,按 Enter 鍵開始說話,再按一次獲取模型響應的音訊。

DashScope Java SDK

選擇交互模式
  • VAD 模式(Voice Activity Detection,自動檢測語音起止)

    Realtime API 自動判斷用戶何時開始與停止說話並作出回應。

  • Manual 模式(手動控制語音起止)

    用戶端控制語音起止。用戶說話結束後,用戶端需主動發送消息至服務端。

VAD 模式

OmniServerVad.java

// 此示例使用 Qwen3.5-Omni-Plus-Realtime,演示输入转写及会话控制。
import com.alibaba.dashscope.audio.omni.*;
import com.alibaba.dashscope.exception.NoApiKeyException;
import com.google.gson.JsonObject;
import javax.sound.sampled.*;
import java.nio.ByteBuffer;
import java.util.Arrays;
import java.util.Base64;
import java.util.Map;
import java.util.Queue;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.atomic.AtomicBoolean;

public class OmniServerVad {
    static class SequentialAudioPlayer {
        private final SourceDataLine line;
        private final Queue<byte[]> audioQueue = new ConcurrentLinkedQueue<>();
        private final Thread playerThread;
        private final AtomicBoolean shouldStop = new AtomicBoolean(false);

        public SequentialAudioPlayer() throws LineUnavailableException {
            AudioFormat format = new AudioFormat(24000, 16, 1, true, false);
            line = AudioSystem.getSourceDataLine(format);
            line.open(format);
            line.start();

            playerThread = new Thread(() -> {
                while (!shouldStop.get()) {
                    byte[] audio = audioQueue.poll();
                    if (audio != null) {
                        line.write(audio, 0, audio.length);
                    } else {
                        try { Thread.sleep(10); } catch (InterruptedException ignored) {}
                    }
                }
            }, "AudioPlayer");
            playerThread.start();
        }

        public void play(String base64Audio) {
            try {
                byte[] audio = Base64.getDecoder().decode(base64Audio);
                audioQueue.add(audio);
            } catch (Exception e) {
                System.err.println("音频解码失败: " + e.getMessage());
            }
        }

        public void cancel() {
            audioQueue.clear();
            line.flush();
        }

        public void close() {
            shouldStop.set(true);
            try { playerThread.join(1000); } catch (InterruptedException ignored) {}
            line.drain();
            line.close();
        }
    }

    public static void main(String[] args) {
        try {
            SequentialAudioPlayer player = new SequentialAudioPlayer();
            AtomicBoolean userIsSpeaking = new AtomicBoolean(false);
            AtomicBoolean shouldStop = new AtomicBoolean(false);

            OmniRealtimeParam param = OmniRealtimeParam.builder()
                    .model("qwen3.5-omni-plus-realtime")
                    .apikey(System.getenv("DASHSCOPE_API_KEY"))
                    // 以下為新加坡地區的 URL,請在範例中填寫實際的業務空間 ID,各地區的 URL 不同。
                    .url("wss://{WorkspaceId}.ap-southeast-1.maas.aliyuncs.com/api-ws/v1/realtime")
                    .build();

            OmniRealtimeConversation conversation = new OmniRealtimeConversation(param, new OmniRealtimeCallback() {
                @Override public void onOpen() {
                    System.out.println("连接已建立");
                }
                @Override public void onClose(int code, String reason) {
                    System.out.println("连接已关闭 (" + code + "): " + reason);
                    shouldStop.set(true);
                }
                @Override public void onEvent(JsonObject event) {
                    handleEvent(event, player, userIsSpeaking);
                }
            });

            conversation.connect();
            conversation.updateSession(OmniRealtimeConfig.builder()
                    .modalities(Arrays.asList(OmniRealtimeModality.AUDIO, OmniRealtimeModality.TEXT))
                    .voice("Tina")
                    .enableTurnDetection(true)
                    .enableInputAudioTranscription(true)
                    .parameters(Map.of("instructions",
                            "你是五星级酒店的AI客服专员,请准确且友好地解答客户关于房型、设施、价格、预订政策的咨询。请始终以专业和乐于助人的态度回应,杜绝提供未经证实或超出酒店服务范围的信息。"))
                    .build()
            );

            System.out.println("请开始说话(自动检测语音开始/结束,按Ctrl+C退出)...");
            AudioFormat format = new AudioFormat(16000, 16, 1, true, false);
            TargetDataLine mic = AudioSystem.getTargetDataLine(format);
            mic.open(format);
            mic.start();

            ByteBuffer buffer = ByteBuffer.allocate(3200);
            while (!shouldStop.get()) {
                int bytesRead = mic.read(buffer.array(), 0, buffer.capacity());
                if (bytesRead > 0) {
                    byte[] chunk = new byte[bytesRead];
                    System.arraycopy(buffer.array(), 0, chunk, 0, bytesRead);
                    try {
                        conversation.appendAudio(Base64.getEncoder().encodeToString(chunk));
                    } catch (Exception e) {
                        if (e.getMessage() != null && e.getMessage().contains("closed")) {
                            System.out.println("对话已关闭,停止录音");
                            break;
                        }
                    }
                }
                Thread.sleep(20);
            }

            conversation.close(1000, "正常结束");
            player.close();
            mic.close();
            System.out.println("\n程序已退出");

        } catch (NoApiKeyException e) {
            System.err.println("未找到API KEY: 请设置环境变量 DASHSCOPE_API_KEY");
            System.exit(1);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    private static void handleEvent(JsonObject event, SequentialAudioPlayer player, AtomicBoolean userIsSpeaking) {
        String type = event.get("type").getAsString();
        switch (type) {
            case "input_audio_buffer.speech_started":
                System.out.println("\n[用户开始说话]");
                player.cancel();
                userIsSpeaking.set(true);
                break;
            case "input_audio_buffer.speech_stopped":
                System.out.println("[用户停止说话]");
                userIsSpeaking.set(false);
                break;
            case "response.audio.delta":
                if (!userIsSpeaking.get()) {
                    player.play(event.get("delta").getAsString());
                }
                break;
            case "conversation.item.input_audio_transcription.delta":
                // 流式预览:text为已确认前缀,stash为待确认后缀
                String preview = event.get("text").getAsString() + event.get("stash").getAsString();
                System.out.print("\r用户: " + preview);
                break;
            case "conversation.item.input_audio_transcription.completed":
                System.out.println();
                break;
            case "response.audio_transcript.done":
                System.out.println("助手: " + event.get("transcript").getAsString());
                break;
            case "response.done":
                System.out.println("回复完成");
                break;
        }
    }
}

運行OmniServerVad.main()方法,透過麥克風即可與 Realtime 模型即時對話,系統會檢測您的音訊起始位置並自動發送到服務器,無需您手動發送。

Manual 模式

OmniWithoutServerVad.java

// 此示例使用 Qwen3.5-Omni-Plus-Realtime,演示输入转写及会话控制。
// DashScope Java SDK 版本不低于2.22.15

import com.alibaba.dashscope.audio.omni.*;
import com.alibaba.dashscope.exception.NoApiKeyException;
import com.google.gson.JsonObject;
import javax.sound.sampled.*;
import java.io.IOException;
import java.util.Arrays;
import java.util.Base64;
import java.util.HashMap;
import java.util.Queue;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicReference;

public class OmniWithoutServerVad {
    // RealtimePcmPlayer 类定义开始
    public static class RealtimePcmPlayer {
        private int sampleRate;
        private SourceDataLine line;
        private AudioFormat audioFormat;
        private Thread decoderThread;
        private Thread playerThread;
        private AtomicBoolean stopped = new AtomicBoolean(false);
        private Queue<String> b64AudioBuffer = new ConcurrentLinkedQueue<>();
        private Queue<byte[]> RawAudioBuffer = new ConcurrentLinkedQueue<>();

        // 构造函数初始化音频格式和音频线路
        public RealtimePcmPlayer(int sampleRate) throws LineUnavailableException {
            this.sampleRate = sampleRate;
            this.audioFormat = new AudioFormat(this.sampleRate, 16, 1, true, false);
            DataLine.Info info = new DataLine.Info(SourceDataLine.class, audioFormat);
            line = (SourceDataLine) AudioSystem.getLine(info);
            line.open(audioFormat);
            line.start();
            decoderThread = new Thread(new Runnable() {
                @Override
                public void run() {
                    while (!stopped.get()) {
                        String b64Audio = b64AudioBuffer.poll();
                        if (b64Audio != null) {
                            byte[] rawAudio = Base64.getDecoder().decode(b64Audio);
                            RawAudioBuffer.add(rawAudio);
                        } else {
                            try {
                                Thread.sleep(100);
                            } catch (InterruptedException e) {
                                throw new RuntimeException(e);
                            }
                        }
                    }
                }
            });
            playerThread = new Thread(new Runnable() {
                @Override
                public void run() {
                    while (!stopped.get()) {
                        byte[] rawAudio = RawAudioBuffer.poll();
                        if (rawAudio != null) {
                            try {
                                playChunk(rawAudio);
                            } catch (IOException e) {
                                throw new RuntimeException(e);
                            } catch (InterruptedException e) {
                                throw new RuntimeException(e);
                            }
                        } else {
                            try {
                                Thread.sleep(100);
                            } catch (InterruptedException e) {
                                throw new RuntimeException(e);
                            }
                        }
                    }
                }
            });
            decoderThread.start();
            playerThread.start();
        }

        // 播放一个音频块并阻塞直到播放完成
        private void playChunk(byte[] chunk) throws IOException, InterruptedException {
            if (chunk == null || chunk.length == 0) return;

            int bytesWritten = 0;
            while (bytesWritten < chunk.length) {
                bytesWritten += line.write(chunk, bytesWritten, chunk.length - bytesWritten);
            }
            int audioLength = chunk.length / (this.sampleRate*2/1000);
            // 等待缓冲区中的音频播放完成
            Thread.sleep(audioLength - 10);
        }

        public void write(String b64Audio) {
            b64AudioBuffer.add(b64Audio);
        }

        public void cancel() {
            b64AudioBuffer.clear();
            RawAudioBuffer.clear();
        }

        public void waitForComplete() throws InterruptedException {
            while (!b64AudioBuffer.isEmpty() || !RawAudioBuffer.isEmpty()) {
                Thread.sleep(100);
            }
            line.drain();
        }

        public void shutdown() throws InterruptedException {
            stopped.set(true);
            decoderThread.join();
            playerThread.join();
            if (line != null && line.isRunning()) {
                line.drain();
                line.close();
            }
        }
    } // RealtimePcmPlayer 类定义结束
    // 录音并实时发送方法
    private static void recordAndSend(TargetDataLine line, OmniRealtimeConversation conversation) {
        byte[] buffer = new byte[3200];
        AtomicBoolean stopRecording = new AtomicBoolean(false);

        // 启动监听Enter键的线程
        Thread enterKeyListener = new Thread(() -> {
            try {
                System.in.read();
                stopRecording.set(true);
            } catch (IOException e) {
                e.printStackTrace();
            }
        });
        enterKeyListener.start();

        // 录音并实时发送
        while (!stopRecording.get()) {
            int count = line.read(buffer, 0, buffer.length);
            if (count > 0) {
                byte[] chunk = new byte[count];
                System.arraycopy(buffer, 0, chunk, 0, count);
                conversation.appendAudio(Base64.getEncoder().encodeToString(chunk));
            }
        }
    }

    public static void main(String[] args) throws InterruptedException, LineUnavailableException {
        OmniRealtimeParam param = OmniRealtimeParam.builder()
                .model("qwen3.5-omni-plus-realtime")
                // 新加坡和北京地域的API Key不同。获取API Key:https://www.alibabacloud.com/help/model-studio/get-api-key
                // 若没有配置环境变量,请用百炼API Key将下行替换为:.apikey("sk-xxx")
                .apikey(System.getenv("DASHSCOPE_API_KEY"))
                // 以下為新加坡地區的 URL,請在範例中填寫實際的業務空間 ID,各地區的 URL 不同。
                .url("wss://{WorkspaceId}.ap-southeast-1.maas.aliyuncs.com/api-ws/v1/realtime")
                .build();
        AtomicReference<CountDownLatch> responseDoneLatch = new AtomicReference<>(null);
        responseDoneLatch.set(new CountDownLatch(1));
        AtomicReference<CountDownLatch> transcriptionDoneLatch = new AtomicReference<>(null);
        transcriptionDoneLatch.set(new CountDownLatch(1));

        RealtimePcmPlayer audioPlayer = new RealtimePcmPlayer(24000);
        final AtomicReference<OmniRealtimeConversation> conversationRef = new AtomicReference<>(null);
        OmniRealtimeConversation conversation = new OmniRealtimeConversation(param, new OmniRealtimeCallback() {
            @Override
            public void onOpen() {
                System.out.println("connection opened");
            }
            @Override
            public void onEvent(JsonObject message) {
                String type = message.get("type").getAsString();
                switch(type) {
                    case "session.created":
                        System.out.println("start session: " + message.get("session").getAsJsonObject().get("id").getAsString());
                        break;
                    case "conversation.item.input_audio_transcription.delta":
                        // 流式预览:text为已确认前缀,stash为待确认后缀
                        String transcriptPreview = message.get("text").getAsString() + message.get("stash").getAsString();
                        System.out.print("\rquestion: " + transcriptPreview);
                        break;
                    case "conversation.item.input_audio_transcription.completed":
                        System.out.println();
                        transcriptionDoneLatch.get().countDown();
                        break;
                    case "response.audio_transcript.delta":
                        System.out.println("got llm response delta: " + message.get("delta").getAsString());
                        break;
                    case "response.audio.delta":
                        String recvAudioB64 = message.get("delta").getAsString();
                        audioPlayer.write(recvAudioB64);
                        break;
                    case "response.done":
                        System.out.println("======RESPONSE DONE======");
                        if (conversationRef.get() != null) {
                            System.out.println("[Metric] response: " + conversationRef.get().getResponseId() +
                                    ", first text delay: " + conversationRef.get().getFirstTextDelay() +
                                    " ms, first audio delay: " + conversationRef.get().getFirstAudioDelay() + " ms");
                        }
                        responseDoneLatch.get().countDown();
                        break;
                    default:
                        break;
                }
            }
            @Override
            public void onClose(int code, String reason) {
                System.out.println("connection closed code: " + code + ", reason: " + reason);
            }
        });
        conversationRef.set(conversation);
        try {
            conversation.connect();
        } catch (NoApiKeyException e) {
            throw new RuntimeException(e);
        }
        OmniRealtimeConfig config = OmniRealtimeConfig.builder()
                .modalities(Arrays.asList(OmniRealtimeModality.AUDIO, OmniRealtimeModality.TEXT))
                .voice("Tina")
                .enableTurnDetection(false)
                // 设定模型角色
                .parameters(new HashMap<String, Object>() {{
                    put("instructions","你是个人助理小云,请你准确且友好地解答用户的问题,始终以乐于助人的态度回应。");
                }})
                .build();
        conversation.updateSession(config);

        // 新增麦克风录音功能
        AudioFormat format = new AudioFormat(16000, 16, 1, true, false);
        DataLine.Info info = new DataLine.Info(TargetDataLine.class, format);

        if (!AudioSystem.isLineSupported(info)) {
            System.out.println("Line not supported");
            return;
        }

        TargetDataLine line = null;
        try {
            line = (TargetDataLine) AudioSystem.getLine(info);
            line.open(format);
            line.start();

            while (true) {
                System.out.println("按Enter开始录音...");
                System.in.read();
                System.out.println("开始录音,请说话...再次按Enter停止录音并发送");
                recordAndSend(line, conversation);
                conversation.commit();
                // 等待转录完成后再触发模型回复,避免输出交错
                transcriptionDoneLatch.get().await(10, TimeUnit.SECONDS);
                System.out.println("等待模型回复...");
                conversation.createResponse(null, null);
                responseDoneLatch.get().await();
                // 重置latch以便下次等待
                responseDoneLatch.set(new CountDownLatch(1));
                transcriptionDoneLatch.set(new CountDownLatch(1));
            }
        } catch (LineUnavailableException e) {
            e.printStackTrace();
        } finally {
            if (line != null) {
                line.stop();
                line.close();
            }
        }
    }}

運行OmniWithoutServerVad.main()方法,按 Enter 鍵開始錄音,錄音過程中再次按 Enter 鍵停止錄音併發送,隨後將接收並播放模型響應。

Python 原生 WebSocket

  • 準備運行環境

    您的 Python 版本需要不低於 3.10。

    首先根據您的操作系統來安裝 pyaudio。

    macOS

    brew install portaudio && pip install pyaudio
    

    Debian/Ubuntu

    sudo apt-get install python3-pyaudio
    
    或者
    
    pip install pyaudio
    

    推薦使用pip install pyaudio。如果安裝失敗,請先根據您的操作系統安裝portaudio依賴。

    CentOS

    sudo yum install -y portaudio portaudio-devel && pip install pyaudio
    

    Windows

    pip install pyaudio
    

    安裝完成後,透過 pip 安裝 websocket 相關的依賴:

pip install websockets==15.0.1
  • 建立用戶端

    在本地新建一個 python 文件,命名為omni_realtime_client.py,並將以下代碼複製進文件中:

    omni_realtime_client.py

    import asyncio
    import websockets
    import json
    import base64
    import time
    from typing import Optional, Callable, List, Dict, Any
    from enum import Enum
    
    class TurnDetectionMode(Enum):
        SERVER_VAD = "server_vad"
        SEMANTIC_VAD = "semantic_vad"  # 使用 Qwen3.8-Omni-Flash-Realtime 或 Qwen3.5-Omni-Realtime 系列模型时推荐
        MANUAL = "manual"
    
    class OmniRealtimeClient:
    
        def __init__(
                self,
                base_url,
                api_key: str,
                model: str = "",
                voice: str = "Ethan",
                instructions: str = "You are a helpful assistant.",
                turn_detection_mode: TurnDetectionMode = TurnDetectionMode.SERVER_VAD,
                on_text_delta: Optional[Callable[[str], None]] = None,
                on_audio_delta: Optional[Callable[[bytes], None]] = None,
                on_input_transcript: Optional[Callable[[str], None]] = None,
                on_output_transcript: Optional[Callable[[str], None]] = None,
                extra_event_handlers: Optional[Dict[str, Callable[[Dict[str, Any]], None]]] = None
        ):
            self.base_url = base_url
            self.api_key = api_key
            self.model = model
            self.voice = voice
            self.instructions = instructions
            self.ws = None
            self.on_text_delta = on_text_delta
            self.on_audio_delta = on_audio_delta
            self.on_input_transcript = on_input_transcript
            self.on_output_transcript = on_output_transcript
            self.turn_detection_mode = turn_detection_mode
            self.extra_event_handlers = extra_event_handlers or {}
    
            # 当前回复状态
            self._current_response_id = None
            self._current_item_id = None
            self._is_responding = False
            # 输入/输出转录打印状态
            self._print_input_transcript = True
            self._output_transcript_buffer = ""
    
        async def connect(self) -> None:
            """与 Realtime API 建立 WebSocket 连接。"""
            url = f"{self.base_url}?model={self.model}"
            headers = {
                "Authorization": f"Bearer {self.api_key}"
            }
            self.ws = await websockets.connect(url, additional_headers=headers)
    
            # 会话配置
            session_config = {
                "modalities": ["text", "audio"],
                "voice": self.voice,
                "instructions": self.instructions,
                # 配置音频输入/输出格式和采样率
                "audio": {
                    "input": {"format": {"type": "pcm", "sample_rate": 16000}},
                    "output": {"format": {"type": "pcm", "sample_rate": 24000}}
                },
                "input_audio_transcription": {
                    "model": "qwen3-asr-flash-realtime"
                }
            }
    
            if self.turn_detection_mode == TurnDetectionMode.MANUAL:
                session_config['turn_detection'] = None
                await self.update_session(session_config)
            elif self.turn_detection_mode == TurnDetectionMode.SERVER_VAD:
                session_config['turn_detection'] = {
                    "type": "server_vad",
                    "threshold": 0.1,
                    "prefix_padding_ms": 500,
                    "silence_duration_ms": 900
                }
                await self.update_session(session_config)
            elif self.turn_detection_mode == TurnDetectionMode.SEMANTIC_VAD:
                session_config['turn_detection'] = {
                    "type": "semantic_vad",
                    "threshold": 0.1,
                    "prefix_padding_ms": 500,
                    "silence_duration_ms": 900
                }
                await self.update_session(session_config)
            else:
                raise ValueError(f"Invalid turn detection mode: {self.turn_detection_mode}")
    
        async def send_event(self, event) -> None:
            event['event_id'] = "event_" + str(int(time.time() * 1000))
            await self.ws.send(json.dumps(event))
    
        async def update_session(self, config: Dict[str, Any]) -> None:
            """更新会话配置。"""
            event = {
                "type": "session.update",
                "session": config
            }
            await self.send_event(event)
    
        async def stream_audio(self, audio_chunk: bytes) -> None:
            """向 API 流式发送原始音频数据。"""
            # 仅支持 16bit 16kHz 单声道 PCM
            audio_b64 = base64.b64encode(audio_chunk).decode()
            append_event = {
                "type": "input_audio_buffer.append",
                "audio": audio_b64
            }
            await self.send_event(append_event)
    
        async def commit_audio_buffer(self) -> None:
            """提交音频缓冲区以触发处理。"""
            event = {
                "type": "input_audio_buffer.commit"
            }
            await self.send_event(event)
    
        async def append_image(self, image_chunk: bytes) -> None:
            """向图像缓冲区追加图像数据。
            图像数据可以来自本地文件,也可以来自实时视频流。
            注意:
                - 图像格式必须为 JPG 或 JPEG。推荐分辨率为 480P 或 720P,最高支持 1080P。
                - 单张图片经Base64编码后不得超过256KB,建议编码前原始图片大小不超过190KB。
                - 将图像数据编码为 Base64 后再发送。
                - 建议以 1张/秒 的频率向服务端发送图像。
                - 在发送图像数据之前,需要至少发送过一次音频数据。
            """
            image_b64 = base64.b64encode(image_chunk).decode()
            event = {
                "type": "input_image_buffer.append",
                "image": image_b64
            }
            await self.send_event(event)
    
        async def create_response(self) -> None:
            """向 API 请求生成回复(仅在手动模式下需要调用)。"""
            event = {
                "type": "response.create"
            }
            await self.send_event(event)
    
        async def cancel_response(self) -> None:
            """取消当前回复。"""
            event = {
                "type": "response.cancel"
            }
            await self.send_event(event)
    
        async def handle_interruption(self):
            """处理用户对当前回复的打断。"""
            if not self._is_responding:
                return
            # 1. 取消当前回复
            if self._current_response_id:
                await self.cancel_response()
    
            self._is_responding = False
            self._current_response_id = None
            self._current_item_id = None
    
        async def handle_messages(self) -> None:
            try:
                async for message in self.ws:
                    event = json.loads(message)
                    event_type = event.get("type")
                    if event_type == "error":
                        print(" Error: ", event['error'])
                        continue
                    elif event_type == "response.created":
                        self._current_response_id = event.get("response", {}).get("id")
                        self._is_responding = True
                    elif event_type == "response.output_item.added":
                        self._current_item_id = event.get("item", {}).get("id")
                    elif event_type == "response.done":
                        self._is_responding = False
                        self._current_response_id = None
                        self._current_item_id = None
                        if event_type in self.extra_event_handlers:
                            self.extra_event_handlers[event_type](event)
                    elif event_type == "input_audio_buffer.speech_started":
                        print("检测到语音开始")
                        if self._is_responding:
                            print("处理打断")
                            await self.handle_interruption()
                    elif event_type == "input_audio_buffer.speech_stopped":
                        print("检测到语音结束")
                    elif event_type == "response.text.delta":
                        if self.on_text_delta:
                            self.on_text_delta(event["delta"])
                    elif event_type == "response.audio.delta":
                        if self.on_audio_delta:
                            audio_bytes = base64.b64decode(event["delta"])
                            self.on_audio_delta(audio_bytes)
                    elif event_type == "conversation.item.input_audio_transcription.delta":
                        preview = event.get("text", "") + event.get("stash", "")
                        print(f"\r用户: {preview}", end='', flush=True)
                    elif event_type == "conversation.item.input_audio_transcription.completed":
                        transcript = event.get("transcript", "")
                        print()
                        if self.on_input_transcript:
                            await asyncio.to_thread(self.on_input_transcript, transcript)
                            self._print_input_transcript = True
                    elif event_type == "response.audio_transcript.delta":
                        if self.on_output_transcript:
                            delta = event.get("delta", "")
                            if not self._print_input_transcript:
                                self._output_transcript_buffer += delta
                            else:
                                if self._output_transcript_buffer:
                                    await asyncio.to_thread(self.on_output_transcript, self._output_transcript_buffer)
                                    self._output_transcript_buffer = ""
                                await asyncio.to_thread(self.on_output_transcript, delta)
                    elif event_type == "response.audio_transcript.done":
                        print(f"大模型: {event.get('transcript', '')}")
                        self._print_input_transcript = False
                    elif event_type in self.extra_event_handlers:
                        self.extra_event_handlers[event_type](event)
            except websockets.exceptions.ConnectionClosed:
                print(" Connection closed")
            except Exception as e:
                print(" Error in message handling: ", str(e))
        async def close(self) -> None:
            """关闭 WebSocket 连接。"""
            if self.ws:
                await self.ws.close()
    
  • 選擇交互模式
    • VAD 模式(Voice Activity Detection,自動檢測語音起止)

      Realtime API 自動判斷用戶何時開始與停止說話並作出回應。

    • Manual 模式(手動控制語音起止)

      用戶端控制語音起止。用戶說話結束後,用戶端需主動發送消息至服務端。

    VAD 模式

    在omni_realtime_client.py的同級目錄下新建另一個 python 文件,命名為vad_mode.py,並將以下代碼複製進文件中:

    vad_mode.py

    # -- coding: utf-8 --
    import os, asyncio, pyaudio, queue, threading
    from omni_realtime_client import OmniRealtimeClient, TurnDetectionMode
    
    # 音频播放器类(处理中断)
    class AudioPlayer:
        def __init__(self, pyaudio_instance, rate=24000):
            self.stream = pyaudio_instance.open(format=pyaudio.paInt16, channels=1, rate=rate, output=True)
            self.queue = queue.Queue()
            self.stop_evt = threading.Event()
            self.interrupt_evt = threading.Event()
            threading.Thread(target=self._run, daemon=True).start()
    
        def _run(self):
            while not self.stop_evt.is_set():
                try:
                    data = self.queue.get(timeout=0.5)
                    if data is None: break
                    if not self.interrupt_evt.is_set(): self.stream.write(data)
                    self.queue.task_done()
                except queue.Empty: continue
    
        def add_audio(self, data): self.queue.put(data)
        def handle_interrupt(self): self.interrupt_evt.set(); self.queue.queue.clear()
        def stop(self): self.stop_evt.set(); self.queue.put(None); self.stream.stop_stream(); self.stream.close()
    
    # 麦克风录音并发送
    async def record_and_send(client):
        p = pyaudio.PyAudio()
        stream = p.open(format=pyaudio.paInt16, channels=1, rate=16000, input=True, frames_per_buffer=3200)
        print("开始录音,请讲话...")
        try:
            while True:
                audio_data = stream.read(3200)
                await client.stream_audio(audio_data)
                await asyncio.sleep(0.02)
        finally:
            stream.stop_stream(); stream.close(); p.terminate()
    
    async def main():
        p = pyaudio.PyAudio()
        player = AudioPlayer(pyaudio_instance=p)
    
        client = OmniRealtimeClient(
            # 以下為新加坡地區的 URL,請在範例中填寫實際的業務空間 ID,各地區的 URL 不同。
            base_url="wss://{WorkspaceId}.ap-southeast-1.maas.aliyuncs.com/api-ws/v1/realtime",
            api_key=os.environ.get("DASHSCOPE_API_KEY"),
            model="qwen3.5-omni-plus-realtime",
            voice="Ethan",
            instructions="你是小云,风趣幽默的好助手",
            # 使用qwen3.5-omni-realtime系列模型时推荐设为SEMANTIC_VAD
            turn_detection_mode=TurnDetectionMode.SEMANTIC_VAD,
            on_text_delta=lambda t: print(f"\nAssistant: {t}", end="", flush=True),
            on_audio_delta=player.add_audio,
        )
    
        await client.connect()
        print("连接成功,开始实时对话...")
    
        # 并发运行
        await asyncio.gather(client.handle_messages(), record_and_send(client))
    
    if __name__ == "__main__":
        try:
            asyncio.run(main())
        except KeyboardInterrupt:
            print("\n程序已退出。")
    

    運行vad_mode.py,透過麥克風即可與 Realtime 模型即時對話,系統會檢測您的音訊起始位置並自動發送到服務器,無需您手動發送。

    Manual 模式

    在omni_realtime_client.py的同級目錄下新建另一個 python 文件,命名為manual_mode.py,並將以下代碼複製進文件中:

    manual_mode.py

    # -- coding: utf-8 --
    import os
    import asyncio
    import time
    import threading
    import queue
    import pyaudio
    from omni_realtime_client import OmniRealtimeClient, TurnDetectionMode
    
    class AudioPlayer:
        """实时音频播放器类"""
    
        def __init__(self, sample_rate=24000, channels=1, sample_width=2):
            self.sample_rate = sample_rate
            self.channels = channels
            self.sample_width = sample_width  # 2 bytes for 16-bit
            self.audio_queue = queue.Queue()
            self.is_playing = False
            self.play_thread = None
            self.pyaudio_instance = None
            self.stream = None
            self._lock = threading.Lock()  # 添加锁来同步访问
            self._last_data_time = time.time()  # 记录最后接收数据的时间
            self._response_done = False  # 添加响应完成标志
            self._waiting_for_response = False  # 标记是否正在等待服务器响应
            # 记录最后一次向音频流写入数据的时间及最近一次音频块的时长,用于更精确地判断播放结束
            self._last_play_time = time.time()
            self._last_chunk_duration = 0.0
    
        def start(self):
            """启动音频播放器"""
            with self._lock:
                if self.is_playing:
                    return
    
                self.is_playing = True
    
                try:
                    self.pyaudio_instance = pyaudio.PyAudio()
    
                    # 创建音频输出流
                    self.stream = self.pyaudio_instance.open(
                        format=pyaudio.paInt16,  # 16-bit
                        channels=self.channels,
                        rate=self.sample_rate,
                        output=True,
                        frames_per_buffer=1024
                    )
    
                    # 启动播放线程
                    self.play_thread = threading.Thread(target=self._play_audio)
                    self.play_thread.daemon = True
                    self.play_thread.start()
    
                    print("音频播放器已启动")
                except Exception as e:
                    print(f"启动音频播放器失败: {e}")
                    self._cleanup_resources()
                    raise
    
        def stop(self):
            """停止音频播放器"""
            with self._lock:
                if not self.is_playing:
                    return
    
                self.is_playing = False
    
            # 清空队列
            while not self.audio_queue.empty():
                try:
                    self.audio_queue.get_nowait()
                except queue.Empty:
                    break
    
            # 等待播放线程结束(在锁外面等待,避免死锁)
            if self.play_thread and self.play_thread.is_alive():
                self.play_thread.join(timeout=2.0)
    
            # 再次获取锁来清理资源
            with self._lock:
                self._cleanup_resources()
    
            print("音频播放器已停止")
    
        def _cleanup_resources(self):
            """清理音频资源(必须在锁内调用)"""
            try:
                # 关闭音频流
                if self.stream:
                    if not self.stream.is_stopped():
                        self.stream.stop_stream()
                    self.stream.close()
                    self.stream = None
            except Exception as e:
                print(f"关闭音频流时出错: {e}")
    
            try:
                if self.pyaudio_instance:
                    self.pyaudio_instance.terminate()
                    self.pyaudio_instance = None
            except Exception as e:
                print(f"终止PyAudio时出错: {e}")
    
        def add_audio_data(self, audio_data):
            """添加音频数据到播放队列"""
            if self.is_playing and audio_data:
                self.audio_queue.put(audio_data)
                with self._lock:
                    self._last_data_time = time.time()  # 更新最后接收数据的时间
                    self._waiting_for_response = False  # 收到数据,不再等待
    
        def stop_receiving_data(self):
            """标记不再接收新的音频数据"""
            with self._lock:
                self._response_done = True
                self._waiting_for_response = False  # 响应结束,不再等待
    
        def prepare_for_next_turn(self):
            """为下一轮对话重置播放器状态。"""
            with self._lock:
                self._response_done = False
                self._last_data_time = time.time()
                self._last_play_time = time.time()
                self._last_chunk_duration = 0.0
                self._waiting_for_response = True  # 开始等待下一轮响应
    
            # 清空上一轮可能残留的音频数据
            while not self.audio_queue.empty():
                try:
                    self.audio_queue.get_nowait()
                except queue.Empty:
                    break
    
        def is_finished_playing(self):
            """检查是否已经播放完所有音频数据"""
            with self._lock:
                queue_size = self.audio_queue.qsize()
                time_since_last_data = time.time() - self._last_data_time
                time_since_last_play = time.time() - self._last_play_time
    
                # ---------------------- 智能结束判定 ----------------------
                # 1. 首选:如果服务器已标记完成且播放队列为空
                #    进一步等待最近一块音频播放完毕(音频块时长 + 0.1s 容错)。
                if self._response_done and queue_size == 0:
                    min_wait = max(self._last_chunk_duration + 0.1, 0.5)  # 至少等待 0.5s
                    if time_since_last_play >= min_wait:
                        return True
    
                # 2. 备用:如果长时间没有新数据且播放队列为空
                #    当服务器没有明确发出 `response.done` 时,此逻辑作为保障
                if not self._waiting_for_response and queue_size == 0 and time_since_last_data > 1.0:
                    print("\n(超时未收到新音频,判定播放结束)")
                    return True
    
                return False
    
        def _play_audio(self):
            """播放音频数据的工作线程"""
            while True:
                # 检查是否应该停止
                with self._lock:
                    if not self.is_playing:
                        break
                    stream_ref = self.stream  # 获取流的引用
    
                try:
                    # 从队列中获取音频数据,超时0.1秒
                    audio_data = self.audio_queue.get(timeout=0.1)
    
                    # 再次检查状态和流的有效性
                    with self._lock:
                        if self.is_playing and stream_ref and not stream_ref.is_stopped():
                            try:
                                # 播放音频数据
                                stream_ref.write(audio_data)
                                # 更新最近播放信息
                                self._last_play_time = time.time()
                                self._last_chunk_duration = len(audio_data) / (
                                            self.channels * self.sample_width) / self.sample_rate
                            except Exception as e:
                                print(f"写入音频流时出错: {e}")
                                break
    
                    # 标记该数据块已处理完成
                    self.audio_queue.task_done()
    
                except queue.Empty:
                    # 队列为空时继续等待
                    continue
                except Exception as e:
                    print(f"播放音频时出错: {e}")
                    break
    
    class MicrophoneRecorder:
        """实时麦克风录音器"""
    
        def __init__(self, sample_rate=16000, channels=1, chunk_size=3200):
            self.sample_rate = sample_rate
            self.channels = channels
            self.chunk_size = chunk_size
            self.pyaudio_instance = None
            self.stream = None
            self.frames = []
            self._is_recording = False
            self._record_thread = None
    
        def _recording_thread(self):
            """录音工作线程"""
            # 在 _is_recording 为 True 期间,持续从音频流中读取数据
            while self._is_recording:
                try:
                    # 使用 exception_on_overflow=False 避免因缓冲区溢出而崩溃
                    data = self.stream.read(self.chunk_size, exception_on_overflow=False)
                    self.frames.append(data)
                except (IOError, OSError) as e:
                    # 当流被关闭时,读取操作可能会引发错误
                    print(f"录音流读取错误,可能已关闭: {e}")
                    break
    
        def start(self):
            """开始录音"""
            if self._is_recording:
                print("录音已在进行中。")
                return
    
            self.frames = []
            self._is_recording = True
    
            try:
                self.pyaudio_instance = pyaudio.PyAudio()
                self.stream = self.pyaudio_instance.open(
                    format=pyaudio.paInt16,
                    channels=self.channels,
                    rate=self.sample_rate,
                    input=True,
                    frames_per_buffer=self.chunk_size
                )
    
                self._record_thread = threading.Thread(target=self._recording_thread)
                self._record_thread.daemon = True
                self._record_thread.start()
                print("麦克风录音已开始...")
            except Exception as e:
                print(f"启动麦克风失败: {e}")
                self._is_recording = False
                self._cleanup()
                raise
    
        def stop(self):
            """停止录音并返回音频数据"""
            if not self._is_recording:
                return None
    
            self._is_recording = False
    
            # 等待录音线程安全退出
            if self._record_thread:
                self._record_thread.join(timeout=1.0)
    
            self._cleanup()
    
            print("麦克风录音已停止。")
            return b''.join(self.frames)
    
        def _cleanup(self):
            """安全地清理 PyAudio 资源"""
            if self.stream:
                try:
                    if self.stream.is_active():
                        self.stream.stop_stream()
                    self.stream.close()
                except Exception as e:
                    print(f"关闭音频流时出错: {e}")
    
            if self.pyaudio_instance:
                try:
                    self.pyaudio_instance.terminate()
                except Exception as e:
                    print(f"终止 PyAudio 实例时出错: {e}")
    
            self.stream = None
            self.pyaudio_instance = None
    
    async def interactive_test():
        """
        交互式测试脚本:允许多轮连续对话,每轮可以发送音频和图片。
        """
        # ------------------- 1. 初始化和连接 (一次性) -------------------
        # 新加坡和北京地域的API Key不同。获取API Key:https://www.alibabacloud.com/help/model-studio/get-api-key
        api_key = os.environ.get("DASHSCOPE_API_KEY")
        if not api_key:
            print("请设置DASHSCOPE_API_KEY环境变量")
            return
    
        print("--- 实时多轮音视频对话客户端 ---")
        print("正在初始化音频播放器和客户端...")
    
        audio_player = AudioPlayer()
        audio_player.start()
    
        def on_audio_received(audio_data):
            audio_player.add_audio_data(audio_data)
    
        transcription_done = threading.Event()
    
        def on_transcription_completed(transcript):
            transcription_done.set()
    
        def on_response_done(event):
            print("\n(收到响应结束标记)")
            audio_player.stop_receiving_data()
    
        realtime_client = OmniRealtimeClient(
            # 以下為新加坡地區的 URL,請在範例中填寫實際的業務空間 ID,各地區的 URL 不同。
            base_url="wss://{WorkspaceId}.ap-southeast-1.maas.aliyuncs.com/api-ws/v1/realtime",
            api_key=api_key,
            model="qwen3.8-omni-flash-realtime",
            voice="Tina",
            instructions="你是个人助理小云,请你准确且友好地解答用户的问题,始终以乐于助人的态度回应。", # 设定模型角色
            on_text_delta=lambda text: print(f"助手回复: {text}", end="", flush=True),
            on_audio_delta=on_audio_received,
            on_input_transcript=on_transcription_completed,
            turn_detection_mode=TurnDetectionMode.MANUAL,
            extra_event_handlers={"response.done": on_response_done}
        )
    
        message_handler_task = None
        try:
            await realtime_client.connect()
            print("已连接到服务器。输入 'q' 或 'quit' 可随时退出程序。")
            message_handler_task = asyncio.create_task(realtime_client.handle_messages())
            await asyncio.sleep(0.5)
    
            turn_counter = 1
            # ------------------- 2. 多轮对话循环 -------------------
            while True:
                print(f"\n--- 第 {turn_counter} 轮对话 ---")
                audio_player.prepare_for_next_turn()
    
                recorded_audio = None
                image_paths = []
    
                # --- 获取用户输入:从麦克风录音 ---
                loop = asyncio.get_event_loop()
                recorder = MicrophoneRecorder(sample_rate=16000)  # 推荐使用16k采样率进行语音识别
    
                print("准备录音。按 Enter 键开始录音 (或输入 'q' 退出)...")
                user_input = await loop.run_in_executor(None, input)
                if user_input.strip().lower() in ['q', 'quit']:
                    print("用户请求退出...")
                    return
    
                try:
                    recorder.start()
                except Exception:
                    print("无法启动录音,请检查您的麦克风权限和设备。跳过本轮。")
                    continue
    
                print("录音中... 再次按 Enter 键停止录音。")
                await loop.run_in_executor(None, input)
    
                recorded_audio = recorder.stop()
    
                if not recorded_audio or len(recorded_audio) == 0:
                    print("未录制到有效音频,请重新开始本轮对话。")
                    continue
    
                # --- 获取图片输入 (可选) ---
                # 以下图片输入功能已被注释,暂时禁用。若需启用请取消下方代码注释。
                # print("\n请逐行输入【图片文件】的绝对路径 (可选)。完成后,输入 's' 或按 Enter 发送请求。")
                # while True:
                #     path = input("图片路径: ").strip()
                #     if path.lower() == 's' or path == '':
                #         break
                #     if path.lower() in ['q', 'quit']:
                #         print("用户请求退出...")
                #         return
                #
                #     if not os.path.isabs(path):
                #         print("错误: 请输入绝对路径。")
                #         continue
                #     if not os.path.exists(path):
                #         print(f"错误: 文件不存在 -> {path}")
                #         continue
                #     image_paths.append(path)
                #     print(f"已添加图片: {os.path.basename(path)}")
    
                # --- 3. 发送数据并获取响应 ---
                print("\n--- 输入确认 ---")
                print(f"待处理音频: 1个 (来自麦克风), 图片: {len(image_paths)}个")
                print("------------------")
    
                # 3.1 发送录制的音频
                try:
                    print(f"发送麦克风录音 ({len(recorded_audio)}字节)")
                    await realtime_client.stream_audio(recorded_audio)
                    await asyncio.sleep(0.1)
                except Exception as e:
                    print(f"发送麦克风录音失败: {e}")
                    continue
    
                # 3.2 发送所有图片文件
                # 以下图片发送代码已被注释,暂时禁用。
                # for i, path in enumerate(image_paths):
                #     try:
                #         with open(path, "rb") as f:
                #             data = f.read()
                #         print(f"发送图片 {i+1}: {os.path.basename(path)} ({len(data)}字节)")
                #         await realtime_client.append_image(data)
                #         await asyncio.sleep(0.1)
                #     except Exception as e:
                #         print(f"发送图片 {os.path.basename(path)} 失败: {e}")
    
                # 3.3 提交并等待响应
                print("录音结束,等待模型回复...")
                await realtime_client.commit_audio_buffer()
                # 等待转录完成后再触发模型回复,避免输出交错
                await asyncio.to_thread(transcription_done.wait, 10)
                transcription_done.clear()
                await realtime_client.create_response()
    
                print("等待并播放服务器响应音频...")
                start_time = time.time()
                max_wait_time = 60
                while not audio_player.is_finished_playing():
                    if time.time() - start_time > max_wait_time:
                        print(f"\n等待超时 ({max_wait_time}秒), 进入下一轮。")
                        break
                    await asyncio.sleep(0.2)
    
                print("\n本轮音频播放完成!")
                turn_counter += 1
    
        except (asyncio.CancelledError, KeyboardInterrupt):
            print("\n程序被中断。")
        except Exception as e:
            print(f"发生未处理的错误: {e}")
        finally:
            # ------------------- 4. 清理资源 -------------------
            print("\n正在关闭连接并清理资源...")
            if message_handler_task and not message_handler_task.done():
                message_handler_task.cancel()
    
            if 'realtime_client' in locals() and realtime_client.ws:
                await realtime_client.close()
                print("连接已关闭。")
    
            audio_player.stop()
            print("程序退出。")
    
    if __name__ == "__main__":
        try:
            asyncio.run(interactive_test())
        except KeyboardInterrupt:
            print("\n程序被用户强制退出。")
    

    運行manual_mode.py,按 Enter 鍵開始說話,再按一次獲取模型響應的音訊。

WebRTC

Python

  • 準備運行環境

    您的 Python 版本需要不低於 3.10。安裝以下依賴:

pip install aiortc aiohttp sounddevice numpy certifi av
  • 運行示例

    新建一個 Python 文件,命名為 webrtc_demo.py,並將以下代碼複製到文件中:

    webrtc_demo.py

    # 依赖安装:pip install aiortc aiohttp sounddevice numpy certifi av
    import asyncio
    import json
    import os
    import queue
    import ssl
    import threading
    
    import aiohttp
    import certifi
    import numpy as np
    import sounddevice as sd
    from aiortc import RTCPeerConnection, RTCConfiguration, RTCSessionDescription
    from aiortc.contrib.media import MediaPlayer
    from av import AudioFrame
    
    # 替换为您的 API Key,或通过环境变量 DASHSCOPE_API_KEY 设置
    API_KEY = os.getenv("DASHSCOPE_API_KEY", "your-api-key")
    MODEL = "qwen3.8-omni-flash-realtime"
    # 以下為新加坡地區的 URL。請在範例中填寫實際的業務空間 ID,各地區的 URL 不同。
    SIGNALING_URL = "https://{WorkspaceId}.ap-southeast-1.maas.aliyuncs.com/api/v1/webrtc/realtime?model=" + MODEL
    
    # --------------- 音频帧解析 ---------------
    
    def _nb_channels(frame: AudioFrame) -> int:
        """获取音频帧的声道数,兼容不同版本的 PyAV"""
        if hasattr(frame.layout, "nb_channels"):
            return int(frame.layout.nb_channels)
        ch = getattr(frame.layout, "channels", 1)
        if isinstance(ch, (tuple, list)):
            return len(ch)
        return int(ch)
    
    def audioframe_to_s16_samples(frame: AudioFrame) -> np.ndarray:
        """
        服务端返回的音频帧是双声道交错排列,直接 reshape 会声道错乱,需要按实际声道数重排为 (采样数, 声道数)。
        aiortc 底层解码库不同版本对同一份音频返回的数组形状不同,这里做统一处理。
        """
        arr = np.asarray(frame.to_ndarray())
        ch = _nb_channels(frame)
        samples = int(frame.samples)
    
        if arr.ndim == 2 and arr.shape[0] == ch and arr.shape[1] == samples:
            return arr.T.copy()
        if arr.ndim == 2 and arr.shape[0] == 1 and arr.shape[1] == samples * ch:
            return arr.reshape(-1).reshape(samples, ch).copy()
        if arr.ndim == 1 and arr.shape[0] == samples * ch:
            return arr.reshape(samples, ch).copy()
    
        flat = arr.reshape(-1)
        if ch > 0 and flat.size % ch == 0:
            return flat.reshape(flat.size // ch, ch).copy()
        raise ValueError(f"unexpected shape={arr.shape}, ch={ch}, samples={samples}")
    
    # --------------- 低延迟音频播放器 ---------------
    
    class RemoteAudioPlayer:
        """
        低延迟音频播放器,每次只取 5ms 的音频块播放,减少延迟。
        支持语音打断:用户开始说话时清空缓存,停止播放模型的旧回复。
        播放时将服务端返回的双声道音频合并为单声道(左右声道取均值)。
        """
        def __init__(self, samplerate=48000, out_channels=1, blocksize=240, max_seconds=0.2):
            self.samplerate = samplerate
            self.out_channels = out_channels
            self.blocksize = blocksize
            self._q = queue.Queue(maxsize=max(5, int(max_seconds * samplerate / blocksize) + 5))
            self._lock = threading.Lock()
            self._rb_size = max(1, int(max_seconds * samplerate))
            self._rb = np.zeros((self._rb_size, out_channels), dtype=np.int16)
            self._rb_w = 0
            self._rb_r = 0
            self._rb_len = 0
            self._stream = None
            self._closed = False
    
        def start(self):
            if self._stream:
                return
    
            def callback(outdata, frames, _time, status):
                if self._closed:
                    outdata[:] = np.zeros((frames, self.out_channels), dtype=np.int16)
                    return
                while True:
                    try:
                        chunk = self._q.get_nowait()
                    except queue.Empty:
                        break
                    with self._lock:
                        self._write_rb(chunk)
                with self._lock:
                    out = self._read_rb(frames)
                outdata[:] = out
    
            self._stream = sd.OutputStream(
                samplerate=self.samplerate,
                channels=self.out_channels,
                dtype="int16",
                blocksize=self.blocksize,
                callback=callback,
            )
            self._stream.start()
    
        def clear(self):
            """清空播放缓存,用于语音打断"""
            try:
                while True:
                    self._q.get_nowait()
            except queue.Empty:
                pass
            with self._lock:
                self._rb_w = 0
                self._rb_r = 0
                self._rb_len = 0
                self._rb[:] = 0
    
        def _write_rb(self, chunk: np.ndarray):
            n = int(chunk.shape[0])
            if n <= 0:
                return
            overflow = max(0, self._rb_len + n - self._rb_size)
            if overflow > 0:
                self._rb_r = (self._rb_r + overflow) % self._rb_size
                self._rb_len -= overflow
            end = self._rb_size - self._rb_w
            if n <= end:
                self._rb[self._rb_w:self._rb_w + n] = chunk
            else:
                self._rb[self._rb_w:] = chunk[:end]
                self._rb[:n - end] = chunk[end:]
            self._rb_w = (self._rb_w + n) % self._rb_size
            self._rb_len += n
    
        def _read_rb(self, frames: int) -> np.ndarray:
            if self._rb_len <= 0:
                return np.zeros((frames, self.out_channels), dtype=np.int16)
            n = min(frames, self._rb_len)
            out = np.zeros((frames, self.out_channels), dtype=np.int16)
            end = self._rb_size - self._rb_r
            if n <= end:
                out[:n] = self._rb[self._rb_r:self._rb_r + n]
            else:
                out[:end] = self._rb[self._rb_r:]
                out[end:n] = self._rb[:n - end]
            self._rb_r = (self._rb_r + n) % self._rb_size
            self._rb_len -= n
            return out
    
        async def push_frame(self, frame: AudioFrame):
            """接收音频帧,自动合并声道后入队"""
            if self._closed:
                return
            pcm = audioframe_to_s16_samples(frame)
            in_ch = pcm.shape[1]
            if self.out_channels == 1:
                if in_ch == 1:
                    out = pcm
                else:
                    out = np.mean(pcm.astype(np.int32), axis=1).astype(np.int16).reshape(-1, 1)
            else:
                if in_ch == self.out_channels:
                    out = pcm
                elif in_ch == 1 and self.out_channels == 2:
                    out = np.repeat(pcm, 2, axis=1)
                else:
                    out = pcm[:, :self.out_channels]
            try:
                self._q.put_nowait(out)
            except queue.Full:
                try:
                    self._q.get_nowait()
                except queue.Empty:
                    pass
                try:
                    self._q.put_nowait(out)
                except queue.Full:
                    pass
    
        async def close(self):
            self._closed = True
            if self._stream:
                self._stream.stop()
                self._stream.close()
                self._stream = None
    
    # --------------- main ---------------
    
    async def main():
        pc = RTCPeerConnection(RTCConfiguration(iceServers=[]))
    
        # 初始化音频播放器(单声道输出,5ms blocksize 低延迟)
        speaker = RemoteAudioPlayer(samplerate=48000, out_channels=1, blocksize=240, max_seconds=0.2)
        speaker.start()
    
        # 初始化麦克风(macOS avfoundation,Linux 请改为 pulse 或 alsa)
        mic = MediaPlayer("none:0", format="avfoundation",
                          options={"sample_rate": "48000", "channels": "1"})
        if not mic.audio:
            raise RuntimeError("未检测到麦克风,请检查 avfoundation 音频设备索引")
        pc.addTrack(mic.audio)
    
        # 客户端创建 DataChannel(名称可自定义),服务端会通过固定名为 "txt" 的通道推送事件
        pc.createDataChannel("oai-events")
    
        remote_dc = None
        got_first_txt_msg = False
    
        def make_session_update() -> dict:
            """构造 session.update 配置:音色、音频格式、VAD 策略、推理参数"""
            return {
                "type": "session.update",
                "session": {
                    "modalities": ["text", "audio"],
                    "voice": "Tina",
                    # 配置音频输入/输出格式和采样率
                    "audio": {
                        "input": {"format": {"type": "pcm", "sample_rate": 16000}},
                        "output": {"format": {"type": "pcm", "sample_rate": 24000}}
                    },
                    "instructions": "你是一个友好的AI助手。",
                    "turn_detection": {"type": "server_vad", "threshold": 0.5, "silence_duration_ms": 800},
                    "max_tokens": 16384,
                    "temperature": 0.9,
                },
            }
    
        # 处理服务端推送的 DataChannel 事件
        @pc.on("datachannel")
        def on_datachannel(ch):
            nonlocal remote_dc, got_first_txt_msg
            print(f"[DC] 收到服务端 DataChannel: {ch.label}")
            if ch.label == "txt":
                remote_dc = ch
    
            @ch.on("message")
            def on_msg(msg):
                nonlocal got_first_txt_msg
                try:
                    evt = json.loads(msg)
                except Exception:
                    return
                print(f"[{ch.label}] {evt.get('type')}")
    
                # 用户开始说话时清空播放缓存,实现语音打断
                if isinstance(evt, dict) and evt.get("type") == "input_audio_buffer.speech_started":
                    speaker.clear()
                    print("[播放] 检测到用户说话,清空播放缓存(打断)")
    
                # 收到 txt 通道首条消息后发送 session.update 配置会话
                if ch.label == "txt" and not got_first_txt_msg:
                    got_first_txt_msg = True
                    if remote_dc and remote_dc.readyState == "open":
                        remote_dc.send(json.dumps(make_session_update(), ensure_ascii=False))
                        print("[DC] 已发送 session.update")
    
        # 接收服务端音频并通过播放器低延迟输出
        @pc.on("track")
        async def on_track(track):
            if track.kind == "audio":
                async def _play():
                    try:
                        while True:
                            frame = await track.recv()
                            await speaker.push_frame(frame)
                    except Exception:
                        pass
                asyncio.create_task(_play())
    
        @pc.on("iceconnectionstatechange")
        def on_ice():
            print(f"[ICE] {pc.iceConnectionState}")
    
        @pc.on("connectionstatechange")
        async def on_conn():
            print(f"[连接] {pc.connectionState}")
            if pc.connectionState in ("failed", "closed", "disconnected"):
                await pc.close()
    
        # SDP 交换:创建 Offer 并 POST 到信令服务端,获取 Answer
        offer = await pc.createOffer()
        await pc.setLocalDescription(offer)
    
        async with aiohttp.ClientSession() as session:
            async with session.post(
                SIGNALING_URL,
                ssl=ssl.create_default_context(cafile=certifi.where()),
                data=offer.sdp.encode("utf-8"),
                headers={
                    "Content-Type": "application/sdp",
                    "Authorization": f"Bearer {API_KEY}",
                },
                timeout=aiohttp.ClientTimeout(total=10),
            ) as resp:
                if not resp.ok:
                    raise Exception(f"SDP 交换失败: {resp.status} {await resp.text()}")
                answer_sdp = await resp.text()
    
        await pc.setRemoteDescription(RTCSessionDescription(sdp=answer_sdp, type="answer"))
        print("SDP 交换完成,等待连接...")
    
        try:
            await asyncio.Event().wait()
        except (KeyboardInterrupt, asyncio.CancelledError):
            pass
        finally:
            print(f"\n退出。最终状态: 连接={pc.connectionState}, ICE={pc.iceConnectionState}")
            await speaker.close()
            try:
                if mic and mic.audio:
                    mic.audio.stop()
            except Exception:
                pass
            await pc.close()
    
    asyncio.run(main())
    

    運行 webrtc_demo.py,透過麥克風即可與 Qwen-Omni-Realtime 模型即時對話,系統會檢測您的音訊起始位置並自動發送到服務器,無需您手動發送。

JavaScript

  • 前提條件
    • 使用支援 WebRTC 的現代瀏覽器(Chrome、Edge、Firefox、Safari 等)。
    • 瀏覽器需要麥克風權限。
    • 瀏覽器無法直接向服務端發起建立連接的請求(受瀏覽器跨域安全策略限制),因此需要透過終端執行 curl 命令來完成連接建立。
  • 運行示例

    新建一個 HTML 文件,命名為 webrtc_demo.html,並將以下代碼複製到文件中:

    webrtc_demo.html

    <!DOCTYPE html>
    <html lang="zh-CN">
    <head>
        <meta charset="UTF-8" />
        <title>WebRTC Realtime 语音对话</title>
        <style>
            * { box-sizing: border-box; margin: 0; padding: 0; }
            body { font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif; background: #f5f7fa; color: #1d2129; padding: 24px; line-height: 1.6; }
    
            .container { max-width: 800px; margin: 0 auto; }
    
            h1 { font-size: 22px; font-weight: 600; margin-bottom: 20px; color: #1d2129; }
    
            /* 顶部吸顶栏 */
            .sticky-top { position: sticky; top: 0; z-index: 100; background: #f5f7fa; margin: 0 -24px 16px; padding: 12px 24px; border-bottom: 1px solid transparent; transition: border-color .2s; }
            .sticky-top.scrolled { border-bottom-color: #e5e6eb; }
    
            /* 控制栏 */
            .toolbar { display: flex; align-items: center; gap: 10px; flex-wrap: wrap; margin-bottom: 12px; }
            .toolbar label { display: flex; align-items: center; gap: 6px; font-size: 13px; color: #4e5969; cursor: pointer; }
    
            /* 按钮 */
            button { padding: 8px 18px; font-size: 13px; font-weight: 500; border: 1px solid #c9cdd4; border-radius: 6px; background: #fff; color: #1d2129; cursor: pointer; transition: all .15s; }
            button:hover:not(:disabled) { border-color: #165dff; color: #165dff; }
            button:disabled { opacity: .4; cursor: not-allowed; }
            .btn-primary { background: #165dff; border-color: #165dff; color: #fff; }
            .btn-primary:hover:not(:disabled) { background: #4080ff; border-color: #4080ff; color: #fff; }
            .btn-danger { border-color: #f53f3f; color: #f53f3f; }
            .btn-danger:hover:not(:disabled) { background: #f53f3f; color: #fff; }
    
            /* 状态指示 */
            .status-bar { display: flex; align-items: center; gap: 8px; padding: 10px 14px; border-radius: 8px; background: #fff; border: 1px solid #e5e6eb; font-size: 13px; }
            .status-dot { width: 8px; height: 8px; border-radius: 50%; background: #c9cdd4; flex-shrink: 0; }
            .status-dot.connected { background: #00b42a; }
            .status-dot.connecting { background: #ff7d00; animation: pulse 1s infinite; }
            .status-dot.error { background: #f53f3f; }
            @keyframes pulse { 0%,100% { opacity: 1; } 50% { opacity: .4; } }
    
            /* SDP 卡片 */
            .card { background: #fff; border: 1px solid #e5e6eb; border-radius: 10px; padding: 16px; margin-bottom: 16px; }
            .card-title { font-size: 13px; font-weight: 600; color: #4e5969; margin-bottom: 8px; }
            .step-num { display: inline-flex; align-items: center; justify-content: center; width: 20px; height: 20px; border-radius: 50%; background: #165dff; color: #fff; font-size: 11px; font-weight: 600; margin-right: 6px; }
            .card-hint { font-size: 12px; color: #86909c; margin-top: 6px; }
    
            textarea { width: 100%; font-family: "SF Mono", "Fira Code", "Fira Mono", Menlo, Consolas, monospace; font-size: 12px; padding: 10px; border: 1px solid #e5e6eb; border-radius: 6px; resize: vertical; background: #f7f8fa; color: #1d2129; transition: border-color .15s; }
            textarea:focus { outline: none; border-color: #165dff; background: #fff; }
    
            /* 视频 */
            .video-section { margin-bottom: 16px; }
            .video-label { font-size: 13px; color: #86909c; margin-bottom: 6px; }
            video { width: 320px; max-width: 100%; background: #000; border-radius: 8px; display: block; }
    
            /* 事件面板 */
            .events-title { font-size: 14px; font-weight: 600; color: #1d2129; margin-bottom: 10px; }
            .events-container { display: flex; flex-direction: column; gap: 6px; }
            .event-item { background: #fff; border: 1px solid #e5e6eb; border-radius: 8px; overflow: hidden; }
            .event-header { display: flex; align-items: center; gap: 8px; padding: 8px 12px; cursor: pointer; user-select: none; font-size: 12px; }
            .event-header:hover { background: #f7f8fa; }
            .event-arrow { font-size: 14px; font-weight: 700; width: 18px; text-align: center; }
            .event-arrow.server { color: #00b42a; }
            .event-arrow.client { color: #165dff; }
            .event-label { color: #4e5969; }
            .event-time { color: #c9cdd4; margin-left: auto; font-size: 11px; }
            .event-body { display: none; padding: 10px 12px; background: #f7f8fa; border-top: 1px solid #e5e6eb; }
            .event-body pre { margin: 0; font-size: 11px; font-family: "SF Mono", Menlo, Consolas, monospace; color: #4e5969; white-space: pre-wrap; word-break: break-all; }
            .events-empty { font-size: 13px; color: #c9cdd4; padding: 16px 0; text-align: center; }
        </style>
    </head>
    <body>
    <div class="container">
        <h1>WebRTC Realtime 语音对话</h1>
    
        <div class="sticky-top">
            <div class="toolbar">
                <button id="startBtn" class="btn-primary">开始会话</button>
                <button id="setAnswerBtn" disabled>设置 Answer</button>
                <button id="endBtn" class="btn-danger" disabled>结束会话</button>
                <button id="downloadBtn" disabled>下载远端音频</button>
                <label>
                    <input id="sendVideoCheckbox" type="checkbox" />
                    开启视频
                </label>
            </div>
    
            <div class="status-bar">
                <span class="status-dot" id="statusDot"></span>
                <span id="statusText">就绪</span>
            </div>
        </div>
    
        <div class="card">
            <div class="card-title"><span class="step-num">1</span>Offer SDP</div>
            <div style="margin-bottom: 8px;">
                <button id="copyOfferBtn" disabled>复制 Offer SDP</button>
            </div>
            <textarea id="offerBox" rows="6" readonly placeholder="点击“开始会话”后自动生成"></textarea>
            <div class="card-hint">ICE 收集完成后自动生成,复制后通过 curl 命令发送到服务端获取 Answer</div>
        </div>
    
        <div class="card">
            <div class="card-title"><span class="step-num">2</span>curl 命令</div>
            <div style="margin-bottom: 8px;">
                <button id="copyCurlBtn" disabled>复制 curl 命令</button>
            </div>
            <div class="card-hint" style="margin-bottom: 4px;">以下為華北2(北京)地區的 URL。請使用您的百鍊業務空間 ID,各地區的 URL 不同。</div>
            <textarea id="curlBox" rows="6" placeholder="Offer SDP 生成后自动填充 curl 命令"></textarea>
            <div class="card-hint">在產生的命令中填好業務空間 ID,再複製到終端機執行,並將傳回的 Answer SDP 貼到下方。</div>
        </div>
    
        <div class="card">
            <div class="card-title"><span class="step-num">3</span>Answer SDP</div>
            <textarea id="answerBox" rows="6" placeholder="将 curl 返回的 Answer SDP 粘贴到这里"></textarea>
            <div class="card-hint">粘贴后点击上方"设置 Answer"按钮建立连接</div>
        </div>
    
        <div class="video-section" id="videoSection" style="display:none;">
            <div class="video-label">本端视频预览</div>
            <video id="localVideo" autoplay playsinline muted></video>
        </div>
    
        <div class="events-title">事件(DataChannel)</div>
        <div id="events" class="events-container"></div>
    </div>
    
    <script>
        const eventsDiv = document.getElementById('events');
        const startBtn = document.getElementById('startBtn');
        const setAnswerBtn = document.getElementById('setAnswerBtn');
        const endBtn = document.getElementById('endBtn');
        const downloadBtn = document.getElementById('downloadBtn');
        const copyOfferBtn = document.getElementById('copyOfferBtn');
        const statusDot = document.getElementById('statusDot');
        const statusText = document.getElementById('statusText');
    
        const copyCurlBtn = document.getElementById('copyCurlBtn');
        const curlBox = document.getElementById('curlBox');
    
        const sendVideoCheckbox = document.getElementById('sendVideoCheckbox');
        const localVideo = document.getElementById('localVideo');
    
        const offerBox = document.getElementById('offerBox');
        const answerBox = document.getElementById('answerBox');
    
        let pc = null;
        let hiddenRemoteAudioEl = null;
    
        let mediaRecorder = null;
        let recordedChunks = [];
        let audioBlob = null;
    
        let localStream = null;
    
        let sendCanvas = null;
        let sendCanvasCtx = null;
        let sendCanvasStream = null;
        let sendRafId = 0;
    
        let gatedAudioTracks = [];
        let gatedVideoTracks = [];
        let audioSender = null;
        let videoSender = null;
        let audioTrack = null;
        let videoTrack = null;
    
        function setStatus(text, state) {
          statusText.textContent = text;
          statusDot.className = 'status-dot' + (state ? ' ' + state : '');
        }
    
        function gateMedia(on) {
          for (const t of gatedAudioTracks) t.enabled = !!on;
          for (const t of gatedVideoTracks) t.enabled = !!on;
        }
    
        function sendUpdate(channel) {
          const update = {
            event_id: `event_${Date.now()}`,
            type: "session.update",
            session: {
              // 配置音频输入/输出格式和采样率
              audio: {
                input: { format: { type: "pcm", sample_rate: 16000 } },
                output: { format: { type: "pcm", sample_rate: 24000 } }
              },
              input_audio_transcription: { model: "qwen3-asr-flash-realtime" },
              instructions: "You are a helpful assistant.",
              modalities: ["text", "audio"],
              smooth_output: false,
              turn_detection: {
                prefix_padding_ms: 500,
                silence_duration_ms: 800,
                threshold: 0.5,
                type: "server_vad",
              },
            },
          };
          if (channel && channel.readyState === "open") channel.send(JSON.stringify(update));
        }
    
        // ===== 事件面板 =====
        const events = [];
        function nowTs() { return new Date().toLocaleTimeString(); }
    
        function renderEvents() {
          eventsDiv.innerHTML = "";
          if (events.length === 0) {
            const empty = document.createElement("div");
            empty.className = "events-empty";
            empty.textContent = "等待事件...";
            eventsDiv.appendChild(empty);
            return;
          }
    
          for (const item of events) {
            const { event, timestamp } = item;
            const isClient = event?.type?.includes("update") || event?.type?.includes("create");
    
            const wrap = document.createElement("div");
            wrap.className = "event-item";
    
            const header = document.createElement("div");
            header.className = "event-header";
    
            const arrow = document.createElement("span");
            arrow.className = "event-arrow " + (isClient ? "client" : "server");
            arrow.textContent = isClient ? "↓" : "↑";
    
            const label = document.createElement("span");
            label.className = "event-label";
            const who = isClient ? "client" : "server";
            const type = event?.type ?? "message";
            label.textContent = `${who}: ${type}`;
    
            const time = document.createElement("span");
            time.className = "event-time";
            time.textContent = timestamp;
    
            const body = document.createElement("div");
            body.className = "event-body";
            const pre = document.createElement("pre");
            pre.textContent = JSON.stringify(event, null, 2);
            body.appendChild(pre);
    
            header.onclick = () => { body.style.display = body.style.display === "block" ? "none" : "block"; };
    
            header.appendChild(arrow);
            header.appendChild(label);
            header.appendChild(time);
            wrap.appendChild(header);
            wrap.appendChild(body);
    
            eventsDiv.appendChild(wrap);
          }
        }
    
        function clearUIEvents() { events.length = 0; renderEvents(); }
        function pushEventFromDataChannel(eventObj) {
          const ts = eventObj.timestamp || nowTs();
          if (!eventObj.timestamp) eventObj.timestamp = ts;
          events.unshift({ event: eventObj, timestamp: ts });
          renderEvents();
        }
    
        function normalizeSdpForSetRemote(sdp) {
          sdp = String(sdp).trim().replace(/\r?\n/g, "\r\n");
          if (!sdp.endsWith("\r\n")) sdp += "\r\n";
          return sdp;
        }
    
        // ===== WebRTC =====
        startBtn.onclick = () => startSession().catch(err => console.log("startSession error:", err));
        endBtn.onclick = () => endSession();
        setAnswerBtn.onclick = () => setRemoteAnswerFromUI().catch(err => console.log("setRemoteAnswer error:", err));
        copyOfferBtn.onclick = async () => {
          const txt = offerBox.value;
          if (!txt) return;
          await navigator.clipboard.writeText(txt);
          alert("Offer SDP 已复制");
        };
        copyCurlBtn.onclick = async () => {
          const txt = curlBox.value;
          if (!txt) return;
          await navigator.clipboard.writeText(txt);
          alert("curl 命令已复制,请在终端执行");
        };
        downloadBtn.onclick = () => {
          if (audioBlob) downloadBlob(audioBlob, 'remote-audio.webm');
          else alert('没有可下载的录音数据');
        };
    
        async function startSession() {
          if (pc) return;
    
          pc = new RTCPeerConnection({ iceServers: [] });
          clearUIEvents();
          setStatus('正在获取麦克风权限...', 'connecting');
    
          offerBox.value = "";
          answerBox.value = "";
          curlBox.value = "";
          setAnswerBtn.disabled = true;
          copyOfferBtn.disabled = true;
          copyCurlBtn.disabled = true;
    
          endBtn.disabled = false;
          downloadBtn.disabled = true;
    
          pc.onconnectionstatechange = () => {
            if (!pc) return;
            if (pc.connectionState === 'connected') {
              setStatus('已连接,请说话', 'connected');
            } else if (["failed", "closed", "disconnected"].includes(pc.connectionState)) {
              console.log("onconnectionstatechange:", pc.connectionState);
              endSession(true);
            }
          };
    
          pc.ontrack = async (e) => {
            const stream = e.streams[0];
            ensureHiddenAudioEl();
            hiddenRemoteAudioEl.srcObject = stream;
            try { await hiddenRemoteAudioEl.play(); } catch {}
            startRecordingRemoteStream(stream);
          };
    
          const wantVideo = !!sendVideoCheckbox.checked;
    
          const localPreviewFps = 30;
          const sendFps = 2;
    
          const constraints = wantVideo
            ? {
                audio: true,
                video: {
                  facingMode: { ideal: "user" },
                  frameRate: { ideal: localPreviewFps, max: localPreviewFps },
                  width: { ideal: 640 },
                  height: { ideal: 480 },
                }
              }
            : { audio: true };
    
          localStream = await navigator.mediaDevices.getUserMedia(constraints);
    
          const videoSection = document.getElementById('videoSection');
          if (wantVideo) {
            localVideo.srcObject = localStream;
            localVideo.style.display = "block";
            videoSection.style.display = "";
            try { await localVideo.play(); } catch {}
          } else {
            localVideo.srcObject = null;
            localVideo.style.display = "none";
            videoSection.style.display = "none";
          }
    
          gatedAudioTracks = [];
          gatedVideoTracks = [];
    
          localStream.getAudioTracks().forEach(t => {
            pc.addTrack(t, localStream);
            gatedAudioTracks.push(t);
          });
    
          if (wantVideo) {
            if (sendRafId) cancelAnimationFrame(sendRafId);
            sendRafId = 0;
            if (sendCanvasStream) sendCanvasStream.getTracks().forEach(t => t.stop());
            sendCanvasStream = null;
            sendCanvasCtx = null;
            sendCanvas = null;
    
            const settings = localStream.getVideoTracks()[0].getSettings();
            sendCanvas = document.createElement("canvas");
            sendCanvas.width = settings.width || 640;
            sendCanvas.height = settings.height || 480;
            sendCanvasCtx = sendCanvas.getContext("2d", { alpha: false });
    
            sendCanvasStream = sendCanvas.captureStream(sendFps);
            const lowFpsTrack = sendCanvasStream.getVideoTracks()[0];
            pc.addTrack(lowFpsTrack, sendCanvasStream);
            gatedVideoTracks.push(lowFpsTrack);
    
            const pump = () => {
              if (!sendCanvasCtx || !sendCanvas) return;
              try { sendCanvasCtx.drawImage(localVideo, 0, 0, sendCanvas.width, sendCanvas.height); } catch {}
              sendRafId = requestAnimationFrame(pump);
            };
            sendRafId = requestAnimationFrame(pump);
          }
    
          gateMedia(false);
    
          audioSender = pc.getSenders().find(s => s.track?.kind === 'audio');
          videoSender = pc.getSenders().find(s => s.track?.kind === 'video');
          audioTrack = audioSender?.track;
          videoTrack = videoSender?.track;
    
          await audioSender?.replaceTrack(null);
          await videoSender?.replaceTrack(videoTrack ? null : undefined);
    
          const dc = pc.createDataChannel('oai-events');
    
          dc.onopen = () => console.log("DC open");
          dc.onmessage = (e) => {
            handleDcMessage(e.data, dc);
          };
    
          pc.ondatachannel = (event) => {
            const ch = event.channel;
            ch.onmessage = (e) => {
                handleDcMessage(e.data, ch);
            };
          };
    
          function handleDcMessage(data, channel) {
              let obj;
              try { obj = JSON.parse(data); }
              catch (err) {
                pushEventFromDataChannel({ type: "raw", data: String(data), parseError: String(err) });
                return;
              }
              pushEventFromDataChannel(obj);
    
              if (obj?.type === "session.created") {
                console.log("Session created, opening media gate.");
                gateMedia(true);
                if(audioSender) audioSender.replaceTrack(audioTrack);
                if(videoSender && videoTrack) videoSender.replaceTrack(videoTrack);
    
                sendUpdate(channel);
              }
          }
    
          pc.onicegatheringstatechange = () => {
            if (!pc) return;
            if (pc.iceGatheringState === "complete" && pc.localDescription?.sdp) {
              const sdp = pc.localDescription.sdp;
              offerBox.value = sdp;
              copyOfferBtn.disabled = false;
              setAnswerBtn.disabled = false;
    
              const escapedSdp = sdp.replace(/'/g, "'\\''");
              // 以下為新加坡地區的 URL。請在範例中填寫實際的業務空間 ID,各地區的 URL 不同。
              curlBox.value = `curl -X POST 'https://{WorkspaceId}.ap-southeast-1.maas.aliyuncs.com/api/v1/webrtc/realtime?model=qwen3.8-omni-flash-realtime' \\\n  -H 'Content-Type: application/sdp' \\\n  -H "Authorization: Bearer $DASHSCOPE_API_KEY" \\\n  --data-binary '${escapedSdp}'`;
              copyCurlBtn.disabled = false;
    
              setStatus('Offer SDP 已生成,复制 curl 命令到终端获取 Answer SDP', 'connecting');
              console.log("ICE Gathering Complete. Ready to set remote description.");
            }
          };
    
          const offer = await pc.createOffer();
          await pc.setLocalDescription(offer);
        }
    
        async function setRemoteAnswerFromUI() {
          if (!pc) return alert('请先点击"开始会话"生成 Offer。');
          const txt = answerBox.value.trim();
          if (!txt) return alert("请粘贴 Answer SDP");
    
          const answerSdp = normalizeSdpForSetRemote(txt);
          try {
              await pc.setRemoteDescription({ type: 'answer', sdp: answerSdp });
              setStatus('正在建立连接...', 'connecting');
          } catch (e) {
              alert("设置 Answer 失败: " + e.message);
              console.error(e);
          }
        }
    
        function endSession(silent = false) {
          if (sendRafId) cancelAnimationFrame(sendRafId);
          sendRafId = 0;
    
          if (sendCanvasStream) {
            sendCanvasStream.getTracks().forEach(t => t.stop());
          }
          sendCanvasStream = null;
          sendCanvasCtx = null;
          sendCanvas = null;
    
          try { if (mediaRecorder && mediaRecorder.state !== "inactive") mediaRecorder.stop(); } catch {}
          mediaRecorder = null;
    
          if (localStream) {
            localStream.getTracks().forEach(t => t.stop());
            localStream = null;
          }
          localVideo.srcObject = null;
          localVideo.style.display = "none";
          document.getElementById('videoSection').style.display = "none";
    
          if (pc) {
            try { pc.close(); } catch {}
            pc = null;
          }
    
          gatedAudioTracks = [];
          gatedVideoTracks = [];
    
          if (hiddenRemoteAudioEl) {
            try { hiddenRemoteAudioEl.pause(); } catch {}
            hiddenRemoteAudioEl.srcObject = null;
            hiddenRemoteAudioEl.remove();
            hiddenRemoteAudioEl = null;
          }
    
          endBtn.disabled = true;
          setAnswerBtn.disabled = true;
          copyOfferBtn.disabled = true;
          copyCurlBtn.disabled = true;
          downloadBtn.disabled = !audioBlob;
    
          setStatus('已断开', '');
          if (!silent) console.log("session ended");
        }
    
        function ensureHiddenAudioEl() {
          if (hiddenRemoteAudioEl) return;
          hiddenRemoteAudioEl = document.createElement("audio");
          hiddenRemoteAudioEl.autoplay = true;
          hiddenRemoteAudioEl.playsInline = true;
          hiddenRemoteAudioEl.muted = false;
          hiddenRemoteAudioEl.style.display = "none";
          document.body.appendChild(hiddenRemoteAudioEl);
        }
    
        function startRecordingRemoteStream(remoteStream) {
          const audioTracks = remoteStream.getAudioTracks();
          if (!audioTracks.length) return;
    
          const audioStream = new MediaStream(audioTracks);
          recordedChunks = [];
          audioBlob = null;
          downloadBtn.disabled = true;
    
          try {
            mediaRecorder = new MediaRecorder(audioStream, { mimeType: 'audio/webm' });
          } catch (err) {
            console.log("MediaRecorder create failed:", err);
            return;
          }
    
          mediaRecorder.ondataavailable = (e) => {
            if (e.data && e.data.size > 0) recordedChunks.push(e.data);
          };
    
          mediaRecorder.onstop = () => {
            audioBlob = new Blob(recordedChunks, { type: 'audio/webm' });
            downloadBtn.disabled = !audioBlob || audioBlob.size === 0;
          };
    
          mediaRecorder.start();
        }
    
        function downloadBlob(blob, filename) {
          const url = URL.createObjectURL(blob);
          const a = document.createElement('a');
          a.style.display = 'none';
          a.href = url;
          a.download = filename;
          document.body.appendChild(a);
          a.click();
          URL.revokeObjectURL(url);
          a.remove();
        }
    
        renderEvents();
    
        const stickyTop = document.querySelector('.sticky-top');
        window.addEventListener('scroll', () => {
          stickyTop.classList.toggle('scrolled', window.scrollY > 10);
        }, { passive: true });
    </script>
    </body>
    </html>
    

    在瀏覽器中打開此文件,按以下步驟操作:

    1. 點擊开始会话,頁面會自動生成 Offer SDP 和對應的 curl 命令。
    2. 點擊复制 curl 命令,在終端中執行。命令返回的內容即為 Answer SDP。
    3. 將 Answer SDP 粘貼到頁面的 Answer SDP 文本框中,點擊设置 Answer即可建立連接並開始語音對話。

AOQ

使用 qwen3.8-omni-flash-realtime 的完整示例見即時通話或按鍵語音對話。

呼叫流程與會話管理

一次呼叫依次完成建連、會話配置、輸入、觸發響應和接收結果。

建立連接

以下 WebSocket 和 WebRTC 建連示例使用 qwen3.8-omni-flash-realtime。

WebSocket

WebSocket 原生連接

呼叫地址與鑒權見接入準備。

# pip install websocket-client
import json
import websocket
import os

API_KEY=os.getenv("DASHSCOPE_API_KEY")
# 以下為新加坡地區的 URL,請在範例中填寫實際的業務空間 ID,各地區的 URL 不同。
API_URL = "wss://{WorkspaceId}.ap-southeast-1.maas.aliyuncs.com/api-ws/v1/realtime?model=qwen3.8-omni-flash-realtime"

headers = [
    "Authorization: Bearer " + API_KEY
]

def on_open(ws):
    print(f"Connected to server: {API_URL}")
def on_message(ws, message):
    data = json.loads(message)
    print("Received event:", json.dumps(data, indent=2))
def on_error(ws, error):
    print("Error:", error)

ws = websocket.WebSocketApp(
    API_URL,
    header=headers,
    on_open=on_open,
    on_message=on_message,
    on_error=on_error
)

ws.run_forever()

DashScope Python SDK

# The Dashscope SDK version must be 2.0.1 or later.
import os
import json

import dashscope

# 以下為新加坡(ap-southeast-1)地區的設定。請在範例中填寫實際的業務空間 ID。region 的值因地區而異。
client = dashscope.Client(
    # If you have not set an environment variable, replace the following line with your Model Studio API key: api_key="sk-xxx",
    # API keys are region-specific. To get an API key, see https://www.alibabacloud.com/help/model-studio/get-api-key
    api_key=os.getenv("DASHSCOPE_API_KEY"),
    workspace="{WorkspaceId}",
    region="ap-southeast-1",
)

try:
    with client.audio.realtime.connect(
            model="qwen3.8-omni-flash-realtime",
    ) as session:
        print("Connected Successfully")
        print("Conversation started. Press Ctrl+C to exit.")
        # Receiver thread: iterate server events
        for event in session.events():
            print("Received event:")
            print(json.dumps(event, indent=2, ensure_ascii=False))
except KeyboardInterrupt:
    pass

DashScope Java SDK

// SDK 版本不低于 2.22.15
import com.alibaba.dashscope.audio.omni.*;
import com.alibaba.dashscope.exception.NoApiKeyException;
import com.google.gson.JsonObject;
import java.util.concurrent.CountDownLatch;

public class Main {
    public static void main(String[] args) throws InterruptedException, NoApiKeyException {
        CountDownLatch latch = new CountDownLatch(1);
        OmniRealtimeParam param = OmniRealtimeParam.builder()
                .model("qwen3.8-omni-flash-realtime")
                .apikey(System.getenv("DASHSCOPE_API_KEY"))
                // 以下為新加坡地區的 URL,請在範例中填寫實際的業務空間 ID,各地區的 URL 不同。
                .url("wss://{WorkspaceId}.ap-southeast-1.maas.aliyuncs.com/api-ws/v1/realtime")
                .build();

        OmniRealtimeConversation conversation = new OmniRealtimeConversation(param, new OmniRealtimeCallback() {
            @Override
            public void onOpen() {
                System.out.println("Connected Successfully");
            }
            @Override
            public void onEvent(JsonObject message) {
                System.out.println(message);
            }
            @Override
            public void onClose(int code, String reason) {
                System.out.println("connection closed code: " + code + ", reason: " + reason);
                latch.countDown();
            }
        });
        conversation.connect();
        latch.await();
        conversation.close(1000, "bye");
        System.exit(0);
    }
}

WebRTC

SDP 交換的地址與參數見接入準備。

WebRTC 建立連接分為兩個階段:

  1. SDP 交換(HTTP):用戶端先將自身的媒體能力和網絡地址(Offer SDP)透過 HTTP POST 發送給服務端,服務端返回自身資訊(Answer SDP),雙方完成能力協商。
  2. 建連(自動):協商完成後,WebRTC 底層自動建立音訊傳輸通道。

建連示例代碼:

# pip install aiortc aiohttp certifi
import asyncio, aiohttp, ssl, certifi
from aiortc import RTCPeerConnection, RTCConfiguration, RTCSessionDescription
from aiortc.mediastreams import AudioStreamTrack

API_KEY = "your-api-key"
MODEL = "qwen3.8-omni-flash-realtime"
# 以下為新加坡地區的 URL。請在範例中填寫實際的業務空間 ID,各地區的 URL 不同。
SIGNALING_URL = "https://{WorkspaceId}.ap-southeast-1.maas.aliyuncs.com/api/v1/webrtc/realtime?model=" + MODEL

async def connect():
    pc = RTCPeerConnection(RTCConfiguration(iceServers=[]))

    # 添加音频轨道,确保 Offer SDP 包含 m=audio(服务端必需)
    pc.addTrack(AudioStreamTrack())

    # 创建 DataChannel 以触发 SDP 协商(名称可自定义,服务端会通过名为 "txt" 的通道推送事件)
    pc.createDataChannel("oai-events")

    # SDP 交换:创建 Offer 并发送到服务端
    offer = await pc.createOffer()
    await pc.setLocalDescription(offer)

    async with aiohttp.ClientSession() as session:
        async with session.post(
            SIGNALING_URL,
            ssl=ssl.create_default_context(cafile=certifi.where()),
            data=offer.sdp.encode("utf-8"),
            headers={
                "Content-Type": "application/sdp",
                "Authorization": f"Bearer {API_KEY}",
            },
        ) as resp:
            if not resp.ok:
                raise Exception(f"SDP 交换失败: {resp.status} {await resp.text()}")
            answer_sdp = await resp.text()

    print("=== Offer SDP ===")
    print(offer.sdp)
    print("=== Answer SDP ===")
    print(answer_sdp)

    # ICE 建连自动完成
    await pc.setRemoteDescription(RTCSessionDescription(sdp=answer_sdp, type="answer"))
    print("WebRTC 连接已建立")
    return pc

瀏覽器端的 SDP 交換受跨域限制。請按即時對話示例中的“WebRTC → JavaScript”操作:在頁面生成 curl 命令,在終端執行,再將返回的 Answer SDP 粘貼到頁面完成連接。

AOQ

AOQ 先透過業務 AppServer 獲取連接參數,再由用戶端 SDK 建立媒體連接。鑒權步驟見AOQ 接入準備。

配置會話

在 session.update 的 session 對象中,用 instructions 設置業務背景和回覆要求。

透過用戶端事件 session.update 配置輸出模態、音色、音訊格式和 VAD。下面是 Qwen3.8-Omni-Flash-Realtime 的 WebSocket Manual 模式配置,與最小音訊問答示例一致。本例使用“龍安靈心”音色(longanlingxin):

{
  "type": "session.update",
  "session": {
    "modalities": [
      "text",
      "audio"
    ],
    "turn_detection": null,
    "audio": {
      "input": {
        "format": {
          "type": "pcm",
          "sample_rate": 16000,
          "sample_format": "s16le",
          "channels": 1,
          "packing": "interleaved",
          "channel_layout": "mono"
        }
      },
      "output": {
        "voice": "longanlingxin",
        "format": {
          "type": "pcm",
          "sample_rate": 24000
        }
      }
    }
  }
}

Qwen3.5-Omni-Realtime 配置示例

以下示例使用 Qwen3.5-Omni-Realtime、Ethan 音色和 semantic_vad 模式。

{
    // 该事件的id,由客户端生成
    "event_id": "event_ToPZqeobitzUJnt3QqtWg",
    // 事件类型,固定为session.update
    "type": "session.update",
    // 会话配置
    "session": {
        // 输出模态,支持设置为["text"](仅输出文本)或["text","audio"](输出文本与音频)。
        "modalities": [
            "text",
            "audio"
        ],
        // 输出音频的音色
        "voice": "Ethan",
        // 推荐写法:同时配置音频格式和采样率(仅 qwen3.5-omni-plus-realtime / qwen3.5-omni-flash-realtime 支持)
        // 输入格式:pcm / wav,默认 pcm;采样率:8000/16000/24000/48000,默认 16000
        // 输出格式:pcm / wav,默认 pcm;采样率:8000/16000/24000/48000,默认 24000
        "audio": {
            "input": {
                "format": {
                    "type": "pcm",
                    "sample_rate": 16000
                }
            },
            "output": {
                "format": {
                    "type": "pcm",
                    "sample_rate": 24000
                }
            }
        },
        // 系统消息,用于设定模型的目标或角色。
        "instructions": "你是某五星级酒店的AI客服专员,请准确且友好地解答客户关于房型、设施、价格、预订政策的咨询。请始终以专业和乐于助人的态度回应,杜绝提供未经证实或超出酒店服务范围的信息。",
        // 是否开启语音活动检测。若需启用,需传入一个配置对象,服务端将据此自动检测语音起止。
        // 设置为null表示由客户端决定何时发起模型响应。
        "turn_detection": {
            // VAD类型,取值为server_vad或semantic_vad。使用qwen3.5-omni-realtime系列模型时推荐设为semantic_vad。
            "type": "semantic_vad",
            // VAD检测阈值。建议在嘈杂的环境中增加,在安静的环境中降低。
            "threshold": 0.5,
            // 检测语音停止的静音持续时间,超过此值后会触发模型响应。取值范围 200-6000,默认 800。金融核实等短回答场景建议 500-600。
            "silence_duration_ms": 800
        }
    }
}

本段列舉的音訊格式與採樣率選項適用於 qwen3.5-omni-plus-realtime 和 qwen3.5-omni-flash-realtime 模型,歷史兼容欄位 input_audio_format / output_audio_format 仍有效,建議採用 audio.input.format / audio.output.format 欄位。

媒體格式和多通道配置見輸入與輸出功能,工具配置見工具呼叫與聯網搜索。

輪次控制與事件時序

VAD 模式

將session.update事件的session.turn_detection.type 設為"server_vad"或"semantic_vad"以啓用 VAD 模式。適用於語音通話場景。WebSocket 和 WebRTC 均支援 VAD 模式;音訊和圖片的傳輸方式及部分服務端事件不同。

使用 Qwen3.8-Omni-Flash-Realtime 系列模型進行客服或商務對話時,可將 session.turn_detection.type 設為 "semantic_vad_ec"。相比 semantic_vad,該模式更易於檢測到用戶說話並打斷模型回覆,同時仍可過濾背景音等無意義聲音。

WebRTC 僅支援 VAD 模式,不支援 Manual 模式。WebRTC 的音訊透過 RTP 直接傳輸,無需發送 input_audio_buffer.append 事件;圖片透過視訊軌道傳輸,不支援 input_image_buffer.append 事件。控制指令和服務端事件透過 DataChannel 傳輸,具體事件參見服務端事件。

交互流程如下:

  1. 用戶端發送音訊資料。WebSocket 透過 input_audio_buffer.append 事件發送;WebRTC 透過音訊軌道(RTP)自動傳輸,無需手動發送事件。
  2. 服務端檢測到語音開始,透過 DataChannel(WebRTC)或 WebSocket 發送 input_audio_buffer.speech_started 事件。
  3. 服務端檢測到語音結束,發送input_audio_buffer.speech_stopped 事件。
  4. 服務端自動提交音訊緩衝區,發送input_audio_buffer.committed 事件。
  5. 服務端開始生成響應,依次發送 response.created、conversation.item.created 等事件。模型的音訊回覆透過 WebSocket 的 response.audio.delta 事件增量返回,或透過 WebRTC 的音訊軌道(RTP)直接傳輸。
  6. 響應過程中,服務端透過 response.audio_transcript.delta 事件增量返回文字轉錄,最終發送 response.done 事件標誌響應完成。
生命週期用戶端事件服務端事件

會話初始化

session.update

會話配置

session.created

會話已建立

session.updated

會話配置已更新

用戶音訊輸入

input_audio_buffer.append

WebSocket:透過此事件添加音訊到緩衝區

input_image_buffer.append

WebSocket:透過此事件添加圖片到緩衝區

WebRTC:音訊透過 RTP 音訊軌道自動傳輸,圖片透過視訊軌道傳輸,無需發送上述事件。

input_audio_buffer.speech_started

檢測到語音開始

input_audio_buffer.speech_stopped

檢測到語音結束

input_audio_buffer.committed

服務器收到提交的音訊

服務器音訊輸出

無

response.created

服務端開始生成響應

response.output_item.added

響應時有新的輸出內容

conversation.item.created

對話項被建立

response.content_part.added

新的輸出內容添加到assistant message

response.audio_transcript.delta

增量生成的轉錄文字

response.audio.delta

WebSocket:模型增量生成的音訊透過此事件返回。WebRTC:音訊透過 RTP 音訊軌道直接傳輸,不返回此事件。

response.audio_transcript.done

文本轉錄完成

response.audio.done

音訊生成完成

response.content_part.done

Assistant message 的文本或音訊內容流式輸出完成

response.output_item.done

Assistant message 的整個輸出項流式傳輸完成

response.done

響應完成

conversation.item.input_audio_transcription.delta

用戶語音輸入的文字流式轉錄(需在 session.update 中啓用 input_audio_transcription)

conversation.item.input_audio_transcription.completed

用戶語音輸入的文字轉錄完成(需在 session.update 中啓用 input_audio_transcription)

Manual 模式

將session.update事件的session.turn_detection 設為 null 以啓用 Manual 模式。此模式下,用戶端透過顯式發送input_audio_buffer.commit 和response.create事件請求服務器響應。適用於按下即說場景,如聊天軟件中的發送語音。

交互流程如下:

  1. 用戶端隨時發送 input_audio_buffer.append與input_image_buffer.append事件追加音訊與圖片至緩衝區。

    發送 input_image_buffer.append 事件前,至少發送過一次 input_audio_buffer.append 事件。

  2. 用戶端發送input_audio_buffer.commit事件提交音訊緩衝區與圖像緩衝區,告知服務端本輪的用戶輸入(音訊及圖片)已全部發送完畢。

  3. 服務端響應 input_audio_buffer.committed事件。

  4. 用戶端發送response.create事件,等待服務端返回模型的輸出。

  5. 服務端響應conversation.item.created事件。

生命週期用戶端事件服務端事件

會話初始化

session.update

會話配置

session.created

會話已建立

session.updated

會話配置已更新

用戶音訊輸入

input_audio_buffer.append

添加音訊到緩衝區

input_image_buffer.append

添加圖片到緩衝區

input_audio_buffer.commit

提交音訊與圖片到服務器

response.create

建立模型響應

input_audio_buffer.committed

服務器收到提交的音訊

服務器音訊輸出

input_audio_buffer.clear

清除緩衝區的音訊

response.created

服務端開始生成響應

response.output_item.added

響應時有新的輸出內容

conversation.item.created

對話項被建立

response.content_part.added

新的輸出內容添加到assistant message 項

response.audio_transcript.delta

增量生成的轉錄文字

response.audio.delta

模型增量生成的音訊

response.audio_transcript.done

完成文本轉錄

response.audio.done

完成音訊生成

response.content_part.done

Assistant message 的文本或音訊內容流式輸出完成

response.output_item.done

Assistant message 的整個輸出項流式傳輸完成

response.done

響應完成

結束會話

每通會話結束後,應斷開連接。WebSocket 接入可呼叫 close() 關閉連接;WebRTC 接入需停止媒體並關閉 PeerConnection,具體步驟參見結束會話與資源清理。同一會話不關閉會導致上下文持續累積。

呼叫 close() 關閉 WebSocket 連接時,若連接剛建立即關閉(無會話交互),關閉耗時約 5-10 秒,此為正常現象。服務端需完成音訊流處理、上下文緩存及推理資源初始化後才能響應關閉請求。建議異步執行 close() 操作,避免阻塞主流程。

會話時長與上下文限制

qwen3.8-omni-flash-realtime 的全部輸入 Token 總數上限為 196608,輸入長度計算規則與 Qwen3.5-Omni-Realtime 一致。

  • 歷史對話與本輪輸入的 Token 總數超過最大輸入長度時,服務端會自動截斷歷史對話並繼續對話。文本和工具返回結果也計入輸入長度。

  • 本輪輸入本身超過最大輸入長度時,服務端會返回錯誤。

  • 單次會話最長可持續 120 分鐘,達到此上限後服務將主動關閉連接。

  • 模型會維護對話歷史上下文,當對話輪次或累計時長超過以下限制時,將自動丟棄更早的歷史資訊。最大時長指模型上下文中能保留的音訊或視訊(圖像幀)累計時長上限。該時長不是整個會話的持續時間。

    由於視訊以抽幀方式輸入(建議 1 幀/秒),視訊最大時長即模型能保留的圖像幀累計時長。例如 240 秒錶示模型最多保留最近 240 秒內收到的幀,超過後更早的幀將被丟棄。

    模型

    音訊最大輪次

    視訊最大輪次

    音訊最大時長

    視訊最大時長

    qwen3.8-omni-flash-realtime

    100輪

    50輪

    600秒

    240秒

    qwen3.5-omni-plus-realtime

    100輪

    50輪

    600秒

    240秒

    qwen3.5-omni-flash-realtime

    80輪

    50輪

    480秒

    120秒

輸入與輸出功能

按任務選擇音訊輸入、圖片與視訊、多通道音訊或音色。工具呼叫介紹 Function Calling、MCP 和聯網搜索配置。

音訊輸入

音訊是推薦的主要輸入方式,可按需搭配圖片。

WebSocket 透過 input_audio_buffer.append 追加 Base64 編碼音訊;WebRTC 透過建連時添加的音訊軌道(RTP)自動傳輸,無需該事件。音訊格式見會話配置,錄音與發送實現見即時對話示例。

圖片與視訊輸入

圖片可來自本地文件,也可從視訊流中即時採集。按協議選擇發送方式:

WebSocket

透過 input_image_buffer.append 發送 Base64 編碼圖片到服務端緩衝區。發送圖片前,至少發送過一次 input_audio_buffer.append。

  • VAD 模式:服務端在檢測到語音結束時自動提交音訊與圖片並觸發響應。本輪圖片應在 input_audio_buffer.speech_stopped 前發送。
  • Manual 模式:追加完本輪音訊與圖片後,發送 input_audio_buffer.commit,收到 input_audio_buffer.committed 後發送 response.create 請求回覆,見事件時序。

本地圖片示例:進入即時對話示例,選擇“WebSocket → Python 原生 WebSocket → Manual 模式”,展開 manual_mode.py,取消圖片輸入與發送兩部分代碼的注釋。

視訊通話場景可對視訊抽幀,建議以 1 張/秒的頻率透過 input_image_buffer.append 發送圖像。DashScope SDK 代碼見Omni-Realtime 示例代碼。

WebRTC

透過建連時添加的視訊軌道(RTP)發送畫面幀,不支援 input_image_buffer.append。WebRTC 僅支援 server_vad 或 semantic_vad,不支援 Manual 模式。

參見即時對話示例的“WebRTC → JavaScript”。示例中的視訊開關用於開啓或停止發送視訊畫面。

可選:視訊聚合

Qwen3.8-Omni-Flash-Realtime 的 session.video.input.representation_compact 預設為 none;設為 normal 可聚合視訊表徵,降低計算開銷,適用於不依賴細粒度視覺資訊的場景。需在首段音訊前配置,開始輸入後不得修改。

以下配置使用 Manual 模式、文本與音訊輸出、Tina 音色,並關閉輸入轉寫。conversation 為已連接的 SDK 會話。SDK 會同時發送音色、VAD 等配置,如需調整,在同一次呼叫中完整設置。

Python

使用 DashScope Python SDK 2.0.1 及以上版本,在即時對話示例中選擇“WebSocket → DashScope Python SDK → Manual 模式”:

  1. 在 manual_dash.py 中,將模型設為 qwen3.8-omni-flash-realtime。
  2. 用下方呼叫替換原 update_session 呼叫(input_video_representation_compact 即視訊聚合配置),在首段音訊前呼叫一次。
  3. 因本例關閉輸入轉寫,省略提交後的 ctx['transcription_done'].wait(timeout=10),繼續執行原有提交與發起回覆流程。
session.update_session(
    output_modalities=['audio', 'text'],
    voice="Tina",
    enable_turn_detection=False,
    enable_input_audio_transcription=False,
    input_video_representation_compact="normal",
)

Java

使用 DashScope Java SDK 2.22.15 及以上版本,在即時對話示例中選擇“WebSocket → DashScope Java SDK → Manual 模式”:

  1. 在 OmniWithoutServerVad.java 中,將模型設為 qwen3.8-omni-flash-realtime。
  2. 用下方透過 OmniRealtimeConfig.builder().parameters(...) 傳入視訊參數的配置和 updateSession 呼叫替換原會話配置,在首段音訊前呼叫一次。
  3. 因本例關閉輸入轉寫,省略提交後的 transcriptionDoneLatch.get().await(10, TimeUnit.SECONDS),繼續執行原有提交與發起回覆流程。

需導入 java.util.Arrays、java.util.Map、java.util.HashMap、com.alibaba.dashscope.audio.omni.OmniRealtimeConfig 和 com.alibaba.dashscope.audio.omni.OmniRealtimeModality。

Map<String, Object> videoInput = new HashMap<>();
videoInput.put("representation_compact", "normal");

Map<String, Object> video = new HashMap<>();
video.put("input", videoInput);

Map<String, Object> parameters = new HashMap<>();
parameters.put("video", video);

OmniRealtimeConfig config =
    OmniRealtimeConfig.builder()
        .modalities(Arrays.asList(OmniRealtimeModality.AUDIO, OmniRealtimeModality.TEXT))
        .voice("Tina")
        .enableTurnDetection(false)
        .enableInputAudioTranscription(false)
        .parameters(parameters)
        .build();
conversation.updateSession(config);

配置後按圖片與視訊輸入的步驟發送畫面,視訊 Token 計量見計費規則。

多通道音訊

以下配置適用於 Qwen3.8-Omni-Flash-Realtime 的 WebSocket 接入。

在首段音訊前設置 session.audio.input.format。支援 1、2、4 聲道;多通道必須為 PCM、16000 Hz、s16le、interleaved。2 聲道使用 raw_mic_array,4 聲道使用 foa_ambix。欄位約束和 JSON 片段見用戶端事件。

輸入 Token 的倍數見計費規則。

接收文本與音訊

模型的響應格式取決於配置的輸出模態。

WebSocket

WebRTC

  • 僅輸出文本

    透過 DataChannel 接收 response.text.delta 和 response.text.done 等流式文本事件。

  • 輸出文本+音訊
    • 文本:透過 DataChannel 接收 response.audio_transcript.delta 和 response.audio_transcript.done 等流式文本事件。
    • 音訊:透過 RTP 軌道即時接收和播放,無需透過 response.audio.delta 事件獲取音訊資料。

音色與聲音複刻

以下配置適用於 Qwen3.8-Omni-Flash-Realtime。

預設音色為 Tina。新接入使用 session.audio.output.voice;如果同時設置了它和兼容欄位 session.voice,輸出音訊使用 session.audio.output.voice 指定的音色。可選音色和試聽見Qwen3.8 音色,自定義音色見聲音複刻,其他型號見完整音色列表。

工具呼叫與聯網搜索

  • 聯網搜索和工具呼叫不兼容,不可同時開啓。

需要自定義工具時,參見Function Calling;接入遠程服務時,使用下方 MCP 入口;獲取即時資訊時,使用聯網搜索。

Function Calling 的工具結果透過 conversation.item.create 回填,item.type 設為 function_call_output。

MCP

以下配置適用於 Qwen3.8-Omni-Flash-Realtime。

在 session.tools 中配置公網 HTTPS MCP Streamable HTTP 服務;僅在沒有活動 Response 時更新 MCP 配置。預設需要審批。先確認工具發現完成,再發起需要該工具的 Response。執行結果、拒絕和失敗處理以及續答步驟見MCP 交互流程。

連網搜尋

連網搜尋功能使模型能夠基於即時檢索資料進行回複,適用於股票價格、天氣預報等需要即時資訊的情境。模型可自主判斷是否需要搜尋來回應使用者的即時問題。

連網搜尋僅 Qwen3.5-Omni-Realtime 系列模型支援,且預設關閉,需通過 session.update 事件啟用。

計費請參考計費說明中的agent策略。

啟用方式

在 session.update 事件中添加以下參數:

  • enable_search:設定為 true 啟用連網搜尋功能。
  • search_options.enable_source:設定為 true 返回搜尋結果來源列表。

參數詳情請參見session.update。

回覆與搜索執行情況

回覆仍透過文本與音訊事件接收。下方示例在收到 response.done 後讀取搜索計量中的 count 和 strategy,打印搜索次數與策略;計量欄位見搜索計量。

程式碼範例

以下樣本展示如何在即時對話中啟用連網搜尋功能。

DashScope Python SDK

在 update_session 調用中傳入 enable_search 和 search_options 參數:

import os
import base64
import time

import pyaudio
from concurrent.futures import ThreadPoolExecutor

import dashscope

# 以下為新加坡(ap-southeast-1)地區的設定。請在範例中填寫實際的業務空間 ID。region 的值因地區而異。
client = dashscope.Client(
    # If you have not set an environment variable, replace the following line with your Model Studio API key: api_key="sk-xxx",
    # API keys are region-specific. To get an API key, see https://www.alibabacloud.com/help/model-studio/get-api-key
    api_key=os.getenv("DASHSCOPE_API_KEY"),
    workspace="{WorkspaceId}",
    region="ap-southeast-1",
)
model = 'qwen3.8-omni-flash-realtime'
voice = 'Tina'

pya = pyaudio.PyAudio()
out = pya.open(format=pyaudio.paInt16, channels=1, rate=24000, output=True)

# Receiver thread: iterate server events
def receive_events(session):
    for response in session.events():
        if response['type'] == 'response.audio.delta':
            out.write(base64.b64decode(response['delta']))
        elif response['type'] == 'conversation.item.input_audio_transcription.delta':
            preview = response.get('text', '') + response.get('stash', '')
            print(f"\r[User] {preview}", end='', flush=True)
        elif response['type'] == 'conversation.item.input_audio_transcription.completed':
            print(f"\r[User] {response['transcript']}")
        elif response['type'] == 'response.audio_transcript.done':
            print(f"[LLM] {response['transcript']}")
        elif response['type'] == 'response.done':
            usage = response.get('response', {}).get('usage', {})
            plugins = usage.get('plugins', {})
            if plugins.get('search'):
                print(f"[Search] count={plugins['search']['count']}, strategy={plugins['search']['strategy']}")

with client.audio.realtime.connect(model=model) as session:
    session.update_session(
        output_modalities=['audio', 'text'],
        voice=voice,
        instructions="你是个人助理小云",
        enable_search=True,
        search_options={'enable_source': True}
    )
    mic = pya.open(format=pyaudio.paInt16, channels=1, rate=16000, input=True)
    print("联网搜索已启用,对着麦克风说话 (Ctrl+C 退出)...")
    try:
        with ThreadPoolExecutor(max_workers=1) as pool:
            receiver = pool.submit(receive_events, session)
            while True:
                audio_data = mic.read(3200, exception_on_overflow=False)
                session.append_audio(audio_data)
                time.sleep(0.01)
    except KeyboardInterrupt:
        session.close()
        mic.close()
        out.close()
        pya.terminate()
        print("\n对话结束")

DashScope Java SDK

在 updateSession 中通過 parameters 傳入連網搜尋配置:

import com.alibaba.dashscope.audio.omni.*;
import com.alibaba.dashscope.exception.NoApiKeyException;
import com.google.gson.JsonObject;
import javax.sound.sampled.*;
import java.nio.ByteBuffer;
import java.util.*;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.atomic.AtomicBoolean;

public class OmniSearch {
    static class SequentialAudioPlayer {
        private final SourceDataLine line;
        private final Queue<byte[]> audioQueue = new ConcurrentLinkedQueue<>();
        private final Thread playerThread;
        private final AtomicBoolean shouldStop = new AtomicBoolean(false);

        public SequentialAudioPlayer() throws LineUnavailableException {
            AudioFormat format = new AudioFormat(24000, 16, 1, true, false);
            line = AudioSystem.getSourceDataLine(format);
            line.open(format);
            line.start();
            playerThread = new Thread(() -> {
                while (!shouldStop.get()) {
                    byte[] audio = audioQueue.poll();
                    if (audio != null) {
                        line.write(audio, 0, audio.length);
                    } else {
                        try { Thread.sleep(10); } catch (InterruptedException ignored) {}
                    }
                }
            }, "AudioPlayer");
            playerThread.start();
        }

        public void play(String base64Audio) {
            audioQueue.add(Base64.getDecoder().decode(base64Audio));
        }
        public void close() {
            shouldStop.set(true);
            try { playerThread.join(1000); } catch (InterruptedException ignored) {}
            line.drain();
            line.close();
        }
    }

    public static void main(String[] args) {
        try {
            SequentialAudioPlayer player = new SequentialAudioPlayer();
            AtomicBoolean shouldStop = new AtomicBoolean(false);

            OmniRealtimeParam param = OmniRealtimeParam.builder()
                    .model("qwen3.8-omni-flash-realtime")
                    .apikey(System.getenv("DASHSCOPE_API_KEY"))
                    // 以下為新加坡地區的 URL,請在範例中填寫實際的業務空間 ID,各地區的 URL 不同。
                    .url("wss://{WorkspaceId}.ap-southeast-1.maas.aliyuncs.com/api-ws/v1/realtime")
                    .build();

            OmniRealtimeConversation conversation = new OmniRealtimeConversation(param, new OmniRealtimeCallback() {
                @Override public void onOpen() {
                    System.out.println("连接已建立");
                }
                @Override public void onClose(int code, String reason) {
                    System.out.println("连接已关闭");
                    shouldStop.set(true);
                }
                @Override public void onEvent(JsonObject event) {
                    String type = event.get("type").getAsString();
                    if ("response.audio.delta".equals(type)) {
                        player.play(event.get("delta").getAsString());
                    } else if ("response.audio_transcript.done".equals(type)) {
                        System.out.println("[LLM] " + event.get("transcript").getAsString());
                    } else if ("response.done".equals(type)) {
                        JsonObject response = event.getAsJsonObject("response");
                        if (response != null && response.has("usage")) {
                            JsonObject usage = response.getAsJsonObject("usage");
                            if (usage.has("plugins")) {
                                JsonObject plugins = usage.getAsJsonObject("plugins");
                                if (plugins.has("search")) {
                                    JsonObject search = plugins.getAsJsonObject("search");
                                    System.out.println("[Search] count=" + search.get("count").getAsInt()
                                            + ", strategy=" + search.get("strategy").getAsString());
                                }
                            }
                        }
                    }
                }
            });

            conversation.connect();
            conversation.updateSession(OmniRealtimeConfig.builder()
                    .modalities(Arrays.asList(OmniRealtimeModality.AUDIO, OmniRealtimeModality.TEXT))
                    .voice("Tina")
                    .enableTurnDetection(true)
                    .enableInputAudioTranscription(true)
                    .parameters(Map.of(
                            "instructions", "你是个人助理小云",
                            "enable_search", true,
                            "search_options", Map.of("enable_source", true)
                    ))
                    .build()
            );

            System.out.println("联网搜索已启用,请开始说话(按Ctrl+C退出)...");
            AudioFormat format = new AudioFormat(16000, 16, 1, true, false);
            TargetDataLine mic = AudioSystem.getTargetDataLine(format);
            mic.open(format);
            mic.start();

            ByteBuffer buffer = ByteBuffer.allocate(3200);
            while (!shouldStop.get()) {
                int bytesRead = mic.read(buffer.array(), 0, buffer.capacity());
                if (bytesRead > 0) {
                    byte[] chunk = new byte[bytesRead];
                    System.arraycopy(buffer.array(), 0, chunk, 0, bytesRead);
                    conversation.appendAudio(Base64.getEncoder().encodeToString(chunk));
                }
                Thread.sleep(20);
            }

            conversation.close(1000, "正常结束");
            player.close();
            mic.close();
        } catch (NoApiKeyException e) {
            System.err.println("未找到API KEY: 请设置环境变量 DASHSCOPE_API_KEY");
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

WebSocket(Python)

在 session.update 的 JSON 中添加 enable_search 和 search_options 欄位:

import json
import os
import websocket
import base64
import pyaudio
import threading

API_KEY = os.getenv("DASHSCOPE_API_KEY")
# 以下為新加坡地區的 URL,請在範例中填寫實際的業務空間 ID,各地區的 URL 不同。
API_URL = "wss://{WorkspaceId}.ap-southeast-1.maas.aliyuncs.com/api-ws/v1/realtime?model=qwen3.8-omni-flash-realtime"

pya = pyaudio.PyAudio()
out_stream = pya.open(format=pyaudio.paInt16, channels=1, rate=24000, output=True)

def on_open(ws):
    ws.send(json.dumps({
        "type": "session.update",
        "session": {
            "modalities": ["text", "audio"],
            "voice": "Tina",
            "instructions": "你是个人助理小云",
            # 配置音频输入/输出格式和采样率
            "audio": {
                "input": {"format": {"type": "pcm", "sample_rate": 16000}},
                "output": {"format": {"type": "pcm", "sample_rate": 24000}}
            },
            "enable_search": True,
            "search_options": {
                "enable_source": True
            }
        }
    }))
    print("联网搜索已启用,对着麦克风说话...")
    def send_audio():
        mic = pya.open(format=pyaudio.paInt16, channels=1, rate=16000, input=True)
        try:
            while True:
                audio = mic.read(3200, exception_on_overflow=False)
                ws.send(json.dumps({
                    "type": "input_audio_buffer.append",
                    "audio": base64.b64encode(audio).decode()
                }))
        except Exception:
            mic.close()
    threading.Thread(target=send_audio, daemon=True).start()

def on_message(ws, message):
    event = json.loads(message)
    if event["type"] == "response.audio.delta":
        out_stream.write(base64.b64decode(event["delta"]))
    elif event["type"] == "response.audio_transcript.done":
        print(f"[LLM] {event['transcript']}")
    elif event["type"] == "response.done":
        usage = event.get("response", {}).get("usage", {})
        plugins = usage.get("plugins", {})
        if plugins.get("search"):
            print(f"[Search] count={plugins['search']['count']}, strategy={plugins['search']['strategy']}")

def on_error(ws, error):
    print(f"Error: {error}")

headers = ["Authorization: Bearer " + API_KEY]
ws = websocket.WebSocketApp(API_URL, header=headers, on_open=on_open, on_message=on_message, on_error=on_error)
ws.run_forever()

響應格式

啟用連網搜尋後,response.done 事件中的 usage 會新增 plugins 欄位,用於記錄搜尋計量資訊:

{
    "usage": {
        "total_tokens": 2937,
        "input_tokens": 2554,
        "output_tokens": 383,
        "input_tokens_details": {
            "text_tokens": 2512,
            "audio_tokens": 42
        },
        "output_tokens_details": {
            "text_tokens": 90,
            "audio_tokens": 293
        },
        "plugins": {
            "search": {
                "count": 1,
                "strategy": "agent"
            }
        }
    }
}

計費與限流

計費規則

Qwen-Omni-Realtime 模型根據不同模態(音訊、圖像)對應的Token數計費。計費詳情請參見百鍊控制台。

輸出語音時,qwen3.8-omni-flash-realtime 的音訊及對應文字分別按音訊輸出和文字輸出單價計費;Qwen3.5-Omni-Realtime 系列僅對音訊計費,對應文字不計費。

MCP 工具呼叫不額外收費,模型推理仍按模型價格計費。MCP 的呼叫流程與使用限制見MCP 交互流程。

說明在多輪即時對話中,模型每次生成響應時,需要將上下文窗口內的所有歷史對話內容(包括之前各輪的音訊、圖片和文本)與本輪新增輸入一並作為輸入 Token 進行處理。因此,輸入 Token 會隨對話輪次的增加而逐輪累積,而非僅計算當前輪次的新增輸入。

例如,假設一段 10 秒的音訊輸入轉換為 70 個 Token(Qwen3.5-Omni-Realtime 系列模型),在第 3 輪對話時,該音訊仍在上下文窗口內,則它依然會被計入第 3 輪的輸入 Token。實際計費的輸入 Token 數 = 上下文窗口內所有歷史輪次的內容 Token 數 + 本輪新增輸入的 Token 數。

音訊、圖片、視訊轉換為Token數的規則

音頻

  • Qwen3.8-Omni-Flash-Realtime、Qwen3.5-Omni-Realtime 系列模型:

    • 輸入音訊計算公式:总 Token 数 = 音频时长(单位:秒)* 7
    • 輸出音訊計算公式:总 Tokens 数 = 音频时长(单位:秒)* 12.5
  • qwen3-omni-flash-realtime-2025-09-15:輸入與輸出音訊的計算公式均為总 Token 数 = 音频时长(单位:秒)* 12.5

    若音訊時長不足1秒,則按 1 秒計算。

qwen3.8-omni-flash-realtime 開啓空間音訊輸入時,音訊輸入 Token 數為普通音訊的 2 倍,2 通道和 4 通道的倍數相同。

圖片

  • qwen3.8-omni-flash-realtime、Qwen3.5-Omni-Realtime 系列模型:每 32x32 像素對應 1 個 Token
  • qwen3-omni-flash-realtime-2025-09-15:每32x32像素對應 1 個 Token

一張圖最少需要 4 個 Token,最多支援 1280 個 Token;可使用以下代碼,傳入圖像路徑和會話時長即可估算圖片消耗的 Token 總量:

# 使用以下命令安装Pillow库:pip install Pillow
from PIL import Image
import math

# Qwen3.8-Omni-Flash-Realtime、Qwen3.5-Omni-Realtime、qwen3-omni-flash-realtime-2025-09-15,缩放因子为32
factor = 32

def token_calculate(image_path='', duration=10):
    """
    :param image_path: 图像路径
    :param duration: 会话连接时长
    :return: 图像的Token数
    """
    if len(image_path) > 0:
        # 打开指定的PNG图片文件
        image = Image.open(image_path)
        # 获取图片的原始尺寸
        height = image.height
        width = image.width
        print(f"缩放前的图像尺寸为:高度为{height},宽度为{width}")
        # 将高度调整为factor的整数倍
        h_bar = round(height / factor) * factor
        # 将宽度调整为factor的整数倍
        w_bar = round(width / factor) * factor
        # 图像的Token下限:4个Token
        min_pixels = factor * factor * 4
        # 图像的Token上限:1280个Token
        max_pixels = 1280 * factor * factor
        # 对图像进行缩放处理,调整像素的总数在范围[min_pixels,max_pixels]内
        if h_bar * w_bar > max_pixels:
            # 计算缩放因子beta,使得缩放后的图像总像素数不超过max_pixels
            beta = math.sqrt((height * width) / max_pixels)
            # 重新计算调整后的高度,确保为factor的整数倍
            h_bar = math.floor(height / beta / factor) * factor
            # 重新计算调整后的宽度,确保为factor的整数倍
            w_bar = math.floor(width / beta / factor) * factor
        elif h_bar * w_bar < min_pixels:
            # 计算缩放因子beta,使得缩放后的图像总像素数不低于min_pixels
            beta = math.sqrt(min_pixels / (height * width))
            # 重新计算调整后的高度,确保为factor的整数倍
            h_bar = math.ceil(height * beta / factor) * factor
            # 重新计算调整后的宽度,确保为factor的整数倍
            w_bar = math.ceil(width * beta / factor) * factor
        print(f"缩放后的图像尺寸为:高度为{h_bar},宽度为{w_bar}")
        # 计算图像的Token数:总像素除以factor * factor
        token = int((h_bar * w_bar) / (factor * factor))
        print(f"缩放后的token数量为:{token}")
        total_token = token * math.ceil(duration / 2)
        print(f"总Token数为:{total_token}")
        return total_token
    else:
        print("错误:image_path参数为空,无法计算Token数")
        return 0

if __name__ == "__main__":
    total_token = token_calculate(image_path="xxx/test.jpg", duration=10)

視訊

qwen3.8-omni-flash-realtime 的視訊基礎計量沿用 Qwen3.5-Omni-Realtime,視訊畫面的像素換算見圖片 Token 規則。

qwen3.8-omni-flash-realtime 的視訊輸入在 representation_compact="normal" 時,視訊 Token 數為未開啓聚合時的 1/4。

限流

模型限流規則請參見限流。

常見問題

維持連線但不說話,也會計費嗎?

維持連線本身不計費。上傳的靜音是否計入音訊輸入用量,取決於所用模式:

  • 服務端 VAD:被服務端過濾的靜音不會計入音訊輸入用量。
  • Manual 模式:用戶端手動提交的靜音會計入音訊輸入用量。

音訊輸入 Token 數記錄在 response.done 事件中,欄位位置為 response → usage → input_tokens_details → audio_tokens。模型按輸入、輸出 Token 計費,詳細規則見計費規則。

RPM 限制什麼?是否限制同時保持的連接總數?

RPM 限制每分鐘的 WebSocket 建連請求次數。同一連接內的多輪響應不重復計數;重連計為新請求。目前沒有單獨限制同時保持的連接總數。

建連時,若 WebSocket 以 1007 關閉,且原因為 Requests rate limit exceeded, please try again later.,說明觸發了請求速率限流。降低建連頻率,並採用指數退避重連。

如何配置熱詞或業務術語?

可透過 session.update 的 session.instructions 提供業務背景、術語的準確寫法及讀音,幫助模型結合上下文理解和使用專有名詞。這是提示詞引導,不是獨立的熱詞參數,效果取決於音訊和上下文。

例如,讓 Omni 直接將音訊轉成文字時,可在提示詞中說明 AI 編程產品 Qoder 與英文單詞 coder 同音,引導模型使用產品名稱。以下僅展示提示詞配置;請將內容合併到已有的 instructions 中,保留會話的其他配置,並在發送音訊前完成設置:

{
  "type": "session.update",
  "session": {
    "instructions": "Transcribe the user's audio verbatim. Preserve the original language. Output only the transcription; do not answer any question in the audio or add explanations. Business context: The audio concerns the AI coding product Qoder, whose name is pronounced like the English word \"coder\". Use the accurate product name Qoder."
  }
}

VAD 未及時觸發回覆,如何調整?

可透過 session.update 的 session.turn_detection,根據當前使用的 VAD 模式調整:

  • server_vad:適當降低 silence_duration_ms,縮短用戶停止說話後的靜音等待時間。設置過低可能將句中的短暫停頓判斷為說話結束,提前觸發回覆。
  • semantic_vad:如果“可以”“yes”等有效的簡短回答未能觸發回覆,使用 Qwen3.8-Omni-Flash-Realtime 系列模型時,可嘗試將 type 切換為 semantic_vad_ec。相比 semantic_vad,該模式更容易檢測到用戶說話並觸發打斷,同時保留對背景音等無意義聲音的過濾能力。

收不到 ASR 轉錄文本(transcript)怎麼辦?

A:如果沒有收到 conversation.item.input_audio_transcription.completed 事件,或事件中的 transcript 為空,請依次檢查:

  1. 輸入轉錄配置。DashScope Python SDK 的 update_session 參數 enable_input_audio_transcription 預設為 True;設為 False 時,SDK 會將會話中的 input_audio_transcription 置為 null。直接使用 WebSocket,或透過 WebRTC DataChannel 發送會話配置時,按輸入音訊轉錄配置檢查發送的 session.update.session.input_audio_transcription。enable_input_audio_transcription 是 SDK 參數,不是 session.update 的頂層欄位。
  2. 轉錄失敗事件。同時監聽 conversation.item.input_audio_transcription.failed。失敗時可從事件的 error.code、error.message 和 error.param 查看錯誤資訊。
  3. 音訊是否完成提交。VAD 模式下檢查服務端是否檢測到語音起止;Manual 模式下確認已發送 input_audio_buffer.commit。沒有提交本輪音訊時,不要只等待轉錄完成事件。
  4. 連接和 SDK 版本。確認本輪音訊輸入期間連接未中斷;WebRTC 接入還需檢查 DataChannel 狀態。使用 DashScope Python SDK 時,核對版本符合本文的最低要求(2.0.1)。

錯誤碼

如果模型調用失敗並返回報錯資訊,請參見錯誤碼進行解決。

API 與 SDK 參考

透過 AOQ 接入的流程和示例,請參見AOQ 接入。協議選型和參考入口請參見API 參考。