Alibaba Cloud Model Studio：使用 AOQ 接入 qwen3.5-omni-plus-realtime 實現按鍵語音對話

通過 AOQ 接入 qwen3.5-omni-plus-realtime，由用戶端控制語音起止，實現按鍵通話和可選的拍照提問。用戶端代碼以 iOS Swift 為例。

方案概述

Qwen-Omni-Realtime 支援由服務端 VAD 自動劃分輪次，也支援由用戶端控制輪次的 Manual 模式。本教程將 session.turn_detection 設為 null：使用者按下按鈕時發送音頻，鬆開按鈕時提交音頻並顯式觸發模型回複。

Manual 模式適用於硬體按鍵對講、螢幕按住說話、雜訊環境下由業務自行判停，以及在一輪語音中按需附帶圖片等情境。音頻通過 AOQ Audio 軌傳輸，不需要發送 input_audio_buffer.append。

準備工作

1. 開通阿里雲百鍊，並按擷取與配置 API Key。API Key 只儲存在業務 AppServer，不要寫入用戶端代碼或提交到代碼倉庫。
2. 根據業務部署地區確認 AOQ Endpoint。地區和接入地址的選擇方法請參見選擇地區、服務部署範圍和接入網域名稱。
3. 從SDK 下載擷取最新版 AOQ Client SDK。
4. 搭建業務 AppServer，並按Token 鑒權實現服務端代理鑒權。每次建立新串連前，用戶端都應從 AppServer 擷取新的串連憑證。

匯入 SDK

根據開發平台匯入對應 SDK。後續用戶端代碼以 iOS Swift 為例，其他平台使用相同的介面設計和事件流程。本文以 PCM 音頻流為例。Opus 編碼由外掛程式提供；如果需要使用 Opus 編碼上行，請匯入 Opus 外掛程式。

android {
    defaultConfig {
        minSdk 21
        ndk { abiFilters 'armeabi-v7a', 'arm64-v8a' }
    }
}

dependencies {
    implementation fileTree(dir: 'libs', include: ['*.aar'])
}

<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />
<uses-permission android:name="android.permission.CAMERA" />

{
  "dependencies": {
    "@aoq/client-sdk": "file:./libs/AoqClientSdk.har"
  }
}

"requestPermissions": [
  { "name": "ohos.permission.INTERNET" },
  { "name": "ohos.permission.MICROPHONE",
    "reason": "$string:perm_mic_reason",
    "usedScene": { "abilities": ["EntryAbility"], "when": "inuse" } },
  { "name": "ohos.permission.CAMERA",
    "reason": "$string:perm_camera_reason",
    "usedScene": { "abilities": ["EntryAbility"], "when": "inuse" } }
]

export PYTHONPATH="$PWD/AoqClientSdk:$PYTHONPATH"
export LD_LIBRARY_PATH="$PWD/AoqClientSdk:$LD_LIBRARY_PATH"

實現流程

1. AppServer 通過 Realtime Token 地址擷取 qwen3.5-omni-plus-realtime 的 AOQ 串連參數。
2. 用戶端建立引擎，配置音頻編解碼與軌道；如需持續視覺理解，再配置 Video 軌。
3. 用戶端啟動本地採集和播放，預設關閉 Audio 軌發送，然後建立 AOQ 串連並發送 session.update。
4. 收到 session.updated 後，持續視頻方案開啟 Video 軌；Audio 軌仍保持關閉，直到使用者按下說話按鈕。
5. 使用者按下按鈕時開啟 Audio 軌；鬆開時先關閉 Audio 軌，再按需發送圖片，然後依次發送 input_audio_buffer.commit 和 response.create。
6. 收到 response.done 後可開始下一輪；結束使用時停止裝置、中斷連線並銷毀引擎。

視頻軌持續推流

發布 Video 軌並在 session.updated 後開啟視頻發送。模型持續看到最新畫面；每輪語音只需提交音頻並觸發回複。

Data 軌按需發圖

不發布 Video 軌。需要配圖時，在鬆開按鈕後先發送 input_image_buffer.append，再提交本輪音頻並觸發回複。

AppServer 擷取 Token

在 AppServer 設定 DASHSCOPE_API_KEY，並使用所選地區的 Endpoint 發送請求。clientIp 為用戶端的真實公網 IP；該欄位可選，但建議傳入，以便服務分配合適的 Relay 存取點。

curl -X POST \
  "https://{endpoint}/api/v1/webrtc/realtime?model=qwen3.5-omni-plus-realtime" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${DASHSCOPE_API_KEY}" \
  -H "x-dashscope-rtc-transport: moq" \
  -d "{\"clientIp\": \"${CLIENT_REAL_IP}\"}"

AppServer 將響應中的以下欄位返回用戶端。生產環境中不要把 API Key 返回用戶端。完整請求和響應欄位請參見Token 鑒權。

實現 iOS 用戶端

用戶端從 AppServer 擷取 AoqConnectConfig 後，按以下步驟實現 iOS 端按鍵語音對話。

1. 建立引擎並設定回調

建立 AOQ 單例引擎，並把業務對象註冊為回調接收方。客戶需要在回調中處理串連狀態、服務端事件、錯誤和警示。

let createConfig = AoqCreateConfig()
createConfig.workDir = workDir
engine = AoqClientEngine.createEngine(createConfig, delegate: self)

2. 啟動音視頻裝置

初始化音頻採集與播放。只有持續視頻軌方案需要啟動網路攝影機；客戶需要在調用前獲得麥克風和網路攝影機許可權。

let captureConfig = AoqAudioCaptureConfig()
captureConfig.channel = 1
captureConfig.isExternal = false
engine.startAudioCapture(captureConfig)

let playbackConfig = AoqAudioPlaybackConfig()
playbackConfig.channel = 1
playbackConfig.isExternal = false
playbackConfig.isDefaultSpeaker = true
engine.startAudioPlayer(playbackConfig)

3. 配置編解碼和軌道

根據接入模型和業務音頻格式配置音頻編解碼參數，並根據圖片輸入方式選擇軌道。以下音頻與視頻數值僅為樣本，請按模型要求和業務情境調整。串連前必須關閉 Audio 軌發送。

let audioEncoderConfig = AoqAudioCodecConfig()
audioEncoderConfig.trackType = .audio
audioEncoderConfig.codecType = .audioPCM
audioEncoderConfig.sampleRate = 16_000
audioEncoderConfig.channel = 1
engine.setAudioEncoderConfig(audioEncoderConfig)

let audioDecoderConfig = AoqAudioCodecConfig()
audioDecoderConfig.trackType = .audio
audioDecoderConfig.codecType = .audioPCM
audioDecoderConfig.sampleRate = 24_000
audioDecoderConfig.channel = 1
engine.setAudioDecoderConfig(audioDecoderConfig)

let videoEncoderConfig = AoqVideoCodecConfig()
videoEncoderConfig.trackType = .video
videoEncoderConfig.codecType = .videoJpeg
videoEncoderConfig.width = 960
videoEncoderConfig.height = 540
videoEncoderConfig.fps = 2
videoEncoderConfig.bitrate = 500_000
engine.setVideoEncoderConfig(videoEncoderConfig)

let publishAudioTrack = AoqTrackParam()
publishAudioTrack.trackType = .audio
let publishVideoTrack = AoqTrackParam()
publishVideoTrack.trackType = .video
let publishDataTrack = AoqTrackParam()
publishDataTrack.trackType = .data
let subscribeAudioTrack = AoqTrackParam()
subscribeAudioTrack.trackType = .audio
let subscribeDataTrack = AoqTrackParam()
subscribeDataTrack.trackType = .data
connectConfig.publishTracks = [publishAudioTrack, publishVideoTrack, publishDataTrack]
connectConfig.subscribeTracks = [subscribeAudioTrack, subscribeDataTrack]

let audioEncoderConfig = AoqAudioCodecConfig()
audioEncoderConfig.trackType = .audio
audioEncoderConfig.codecType = .audioPCM
audioEncoderConfig.sampleRate = 16_000
audioEncoderConfig.channel = 1
engine.setAudioEncoderConfig(audioEncoderConfig)

let audioDecoderConfig = AoqAudioCodecConfig()
audioDecoderConfig.trackType = .audio
audioDecoderConfig.codecType = .audioPCM
audioDecoderConfig.sampleRate = 24_000
audioDecoderConfig.channel = 1
engine.setAudioDecoderConfig(audioDecoderConfig)

let publishAudioTrack = AoqTrackParam()
publishAudioTrack.trackType = .audio
let publishDataTrack = AoqTrackParam()
publishDataTrack.trackType = .data
let subscribeAudioTrack = AoqTrackParam()
subscribeAudioTrack.trackType = .audio
let subscribeDataTrack = AoqTrackParam()
subscribeDataTrack.trackType = .data
connectConfig.publishTracks = [publishAudioTrack, publishDataTrack]
connectConfig.subscribeTracks = [subscribeAudioTrack, subscribeDataTrack]

4. 配置 Manual 會話

串連成功後，調用 sendDataMsg 發送 session.update 事件。客戶需要把 turn_detection 設為 null，並按業務選擇音色、系統指令和輸出模態。樣本中的音頻參數需要與 SDK 編解碼配置保持一致。完整欄位請參見用戶端事件。

private func sendSessionUpdate() {
    let event: [String: Any] = [
        "type": "session.update",
        "session": [
            "modalities": ["text", "audio"],
            "voice": "Ethan",
            "audio": [
                "input": ["format": ["type": "pcm", "sample_rate": 16_000]],
                "output": ["format": ["type": "pcm", "sample_rate": 24_000]]
            ],
            "turn_detection": NSNull()
        ]
    ]
    guard let data = try? JSONSerialization.data(withJSONObject: event) else { return }
    let dataMessage = AoqDataMsg()
    dataMessage.data = data
    engine.sendDataMsg(dataMessage)
}

5. 等待會話配置生效

在 onDataMsg 回調中處理 session.updated 事件，收到該事件後才能發送媒體。持續視頻軌方案此時調用 enableSendMediaStream 開啟 Video 軌；Audio 軌仍保持關閉，避免使用者按鍵前的音頻進入緩衝區。

func onDataMsg(_ msg: AoqDataMsg) {
    guard let event = try? JSONSerialization.jsonObject(with: msg.data) as? [String: Any],
          let type = event["type"] as? String else { return }
    if type == "session.updated", imageMode == .continuousVideo {
        engine.enableSendMediaStream(.video, enable: true)
    }
    // 按下通話按鈕前保持 Audio 軌關閉。
}

6. 實現按鍵語音互動

按下按鈕時調用 enableSendMediaStream 開啟 Audio 軌。鬆開按鈕時先調用 enableSendMediaStream 關閉 Audio 軌，確認本輪確有音頻，再按需發送圖片，並調用 sendDataMsg 依次發送 input_audio_buffer.commit 和 response.create 事件。

func onPushToTalkPressed() {
    hasAudioInCurrentTurn = true
    engine.enableSendMediaStream(.audio, enable: true)
}

func onPushToTalkReleased(base64Jpeg: String? = nil) {
    engine.enableSendMediaStream(.audio, enable: false)
    guard hasAudioInCurrentTurn else { return }
    if imageMode == .singleImage, let base64Jpeg {
        let imageEvent: [String: Any] = [
            "type": "input_image_buffer.append",
            "image": base64Jpeg
        ]
        if let data = try? JSONSerialization.data(withJSONObject: imageEvent) {
            let dataMessage = AoqDataMsg()
            dataMessage.data = data
            engine.sendDataMsg(dataMessage)
        }
    }
    for event in [
        ["type": "input_audio_buffer.commit"],
        ["type": "response.create"]
    ] {
        guard let data = try? JSONSerialization.data(withJSONObject: event) else { continue }
        let dataMessage = AoqDataMsg()
        dataMessage.data = data
        engine.sendDataMsg(dataMessage)
    }
    hasAudioInCurrentTurn = false
}

7. 選擇圖片輸入方式

持續視覺理解和偶發拍照提問使用不同的軌道配置與發送方式；客戶需要根據頻寬、功耗和互動方式選擇。

{
  "type": "input_image_buffer.append",
  "image": "<Base64-encoded JPEG data>"
}

8. 中斷連線並銷毀引擎

結束會話時中斷連線並銷毀引擎。disconnect 或 destroy 會自動關閉媒體裝置，無需額外調用停止介面。AoqClientEngine 為全域單例，只有 destroy 後才能重新建立。

engine.disconnect()
AoqClientEngine.destroy()

完整樣本

以下類接收已由 AppServer Token 響應轉換完成的 AoqConnectConfig。請在生產代碼中補充 UI 狀態、許可權、錯誤恢複和圖片壓縮邏輯。

import Foundation
import AoqClientSdk

final class ManualPushToTalkClient: NSObject, AoqEngineDelegate {
    enum ImageMode: Equatable {
        case none
        case continuousVideo
        case singleImage
    }

    private var engine: AoqClientEngine!
    private let imageMode: ImageMode
    private var hasAudioInCurrentTurn = false

    init(workDir: String, connectConfig: AoqConnectConfig, imageMode: ImageMode) {
        self.imageMode = imageMode
        super.init()
        let createConfig = AoqCreateConfig()
        createConfig.workDir = workDir
        self.engine = AoqClientEngine.createEngine(createConfig, delegate: self)

        // 樣本參數，請按接入模型和業務音頻格式調整。
        let audioEncoderConfig = AoqAudioCodecConfig()
        audioEncoderConfig.trackType = .audio
        audioEncoderConfig.codecType = .audioPCM
        audioEncoderConfig.sampleRate = 16_000
        audioEncoderConfig.channel = 1
        engine.setAudioEncoderConfig(audioEncoderConfig)

        let audioDecoderConfig = AoqAudioCodecConfig()
        audioDecoderConfig.trackType = .audio
        audioDecoderConfig.codecType = .audioPCM
        audioDecoderConfig.sampleRate = 24_000
        audioDecoderConfig.channel = 1
        engine.setAudioDecoderConfig(audioDecoderConfig)

        let publishAudioTrack = AoqTrackParam()
        publishAudioTrack.trackType = .audio
        let publishDataTrack = AoqTrackParam()
        publishDataTrack.trackType = .data
        let subscribeAudioTrack = AoqTrackParam()
        subscribeAudioTrack.trackType = .audio
        let subscribeDataTrack = AoqTrackParam()
        subscribeDataTrack.trackType = .data

        connectConfig.publishTracks = [publishAudioTrack, publishDataTrack]
        connectConfig.subscribeTracks = [subscribeAudioTrack, subscribeDataTrack]

        if imageMode == .continuousVideo {
            let videoEncoderConfig = AoqVideoCodecConfig()
            videoEncoderConfig.trackType = .video
            videoEncoderConfig.codecType = .videoJpeg
            videoEncoderConfig.width = 960
            videoEncoderConfig.height = 540
            videoEncoderConfig.fps = 2
            videoEncoderConfig.bitrate = 500_000
            engine.setVideoEncoderConfig(videoEncoderConfig)

            let publishVideoTrack = AoqTrackParam()
            publishVideoTrack.trackType = .video
            connectConfig.publishTracks = [publishAudioTrack, publishVideoTrack, publishDataTrack]
        }

        let captureConfig = AoqAudioCaptureConfig()
        captureConfig.channel = 1
        captureConfig.isExternal = false
        engine.startAudioCapture(captureConfig)

        let playbackConfig = AoqAudioPlaybackConfig()
        playbackConfig.channel = 1
        playbackConfig.isExternal = false
        playbackConfig.isDefaultSpeaker = true
        engine.startAudioPlayer(playbackConfig)

        if imageMode == .continuousVideo {
            let videoCaptureConfig = AoqVideoCaptureConfig()
            videoCaptureConfig.width = 1280
            videoCaptureConfig.height = 720
            videoCaptureConfig.fps = 15
            engine.startVideoCapture(videoCaptureConfig)
        }

        engine.enableSendMediaStream(.audio, enable: false)
        if imageMode == .continuousVideo {
            engine.enableSendMediaStream(.video, enable: false)
        }
        engine.connect(connectConfig)
    }

    func onPushToTalkPressed() {
        hasAudioInCurrentTurn = true
        engine.enableSendMediaStream(.audio, enable: true)
    }

    func onPushToTalkReleased(base64Jpeg: String? = nil) {
        engine.enableSendMediaStream(.audio, enable: false)
        guard hasAudioInCurrentTurn else { return }
        if imageMode == .singleImage, let base64Jpeg {
            let imageEvent: [String: Any] = [
                "type": "input_image_buffer.append",
                "image": base64Jpeg
            ]
            if let data = try? JSONSerialization.data(withJSONObject: imageEvent) {
                let dataMessage = AoqDataMsg()
                dataMessage.data = data
                engine.sendDataMsg(dataMessage)
            }
        }
        for event in [
            ["type": "input_audio_buffer.commit"],
            ["type": "response.create"]
        ] {
            guard let data = try? JSONSerialization.data(withJSONObject: event) else { continue }
            let dataMessage = AoqDataMsg()
            dataMessage.data = data
            engine.sendDataMsg(dataMessage)
        }
        hasAudioInCurrentTurn = false
    }

    private func sendSessionUpdate() {
        let event: [String: Any] = [
            "type": "session.update",
            "session": [
                "modalities": ["text", "audio"],
                "voice": "Ethan",
                "audio": [
                    "input": ["format": ["type": "pcm", "sample_rate": 16_000]],
                    "output": ["format": ["type": "pcm", "sample_rate": 24_000]]
                ],
                "turn_detection": NSNull()
            ]
        ]
        guard let data = try? JSONSerialization.data(withJSONObject: event) else { return }
        let dataMessage = AoqDataMsg()
        dataMessage.data = data
        engine.sendDataMsg(dataMessage)
    }

    func close() {
        engine.disconnect()
        AoqClientEngine.destroy()
    }

    func onConnectionStatusChange(_ status: AoqConnectionStatus) {
        if status == .connected { sendSessionUpdate() }
    }

    func onDataMsg(_ msg: AoqDataMsg) {
        guard let event = try? JSONSerialization.jsonObject(with: msg.data) as? [String: Any],
              let type = event["type"] as? String else { return }
        if type == "session.updated", imageMode == .continuousVideo {
            engine.enableSendMediaStream(.video, enable: true)
        }
    }

    func onError(_ code: Int, message: String) {}
    func onWarning(_ code: Int, message: String) {}
    func onStats(_ stats: AoqStats) {}
    func onAudioDeviceStateChanged(_ state: AoqAudioDeviceState) {}
    func onAudioDeviceRouteChanged(_ routeType: Int) {}
    func onAudioDeviceInterrupted(_ interrupt: Bool) {}
    func onAudioFileState(_ state: AoqAudioFileState) {}
    func onVideoDeviceStateChanged(_ state: AoqVideoDeviceState) {}
}

運行並驗證

分別完成一次純語音按鍵對話和一次帶圖片的按鍵對話，預期結果如下：

1. 按下按鈕前不發送 Audio 軌；按住按鈕時持續上行音頻。
2. 鬆開按鈕後依次收到 input_audio_buffer.committed、response.created 和 response.done，模型語音通過訂閱的 Audio 軌播放。
3. 選擇單張圖片方案時，模型結合本輪圖片與語音作答；選擇持續視頻方案時，模型使用最新視頻畫面。

服務端事件欄位和完整響應結構請參見服務端事件。

注意事項

1. AOQ 的 Audio 軌負責傳輸音頻，不要另外發送 input_audio_buffer.append。
2. input_audio_buffer.commit 只提交本輪輸入，不會觸發模型回複；必須隨後發送 response.create。
3. 空音頻緩衝區不要提交，否則服務端會返回錯誤。
4. 不要在收到 session.updated 前開啟媒體發送；Manual 模式下也不要在使用者按下按鈕前開啟 Audio 軌。

相關文檔

如需查看完整參數、事件欄位或其他平台介面，請參見：

• 擷取與配置 API Key
• 選擇地區、服務部署範圍和接入網域名稱
• AOQ Client SDK 簡介
• SDK 下載
• Token 鑒權
• Qwen-Omni-Realtime 使用者指南
• 通過 AOQ 實現 VAD 模式即時通話
• 用戶端事件
• 服務端事件