Alibaba Cloud Model Studio:Bangun percakapan suara push-to-talk dengan qwen3.5-omni-plus-realtime melalui AOQ

Gunakan AOQ untuk terhubung ke qwen3.5-omni-plus-realtime dan memungkinkan client mengontrol batas giliran (turn boundaries) untuk percakapan push-to-talk serta pertanyaan gambar opsional. Kode client menggunakan iOS Swift.

Solution overview

Qwen-Omni-Realtime mendukung VAD sisi server dan mode Manual yang dikontrol client. Tutorial ini mengatur session.turn_detection ke null. Client mengirim audio selama pengguna menahan tombol, lalu meng-commit audio tersebut dan secara eksplisit meminta respons saat pengguna melepas tombol.

Mode Manual cocok untuk tombol interkom perangkat keras, kontrol tekan-dan-tahan, lingkungan bising di mana aplikasi menentukan batas giliran, serta giliran yang secara opsional menyertakan gambar. Audio ditransmisikan melalui AOQ Audio track. Jangan kirim input_audio_buffer.append.

Prerequisites

1. Aktifkan Model Studio dan ikuti Obtain and configure an API key. Simpan Kunci API hanya di server aplikasi Anda. Jangan sertakan dalam kode client atau commit ke repositori kode.
2. Konfirmasi endpoint AOQ untuk wilayah tempat aplikasi Anda diterapkan. Untuk panduan pemilihan, lihat Select a region, deployment scope, and endpoint.
3. Unduh AOQ Client SDK versi terbaru sebagaimana dijelaskan dalam SDK download.
4. Bangun server aplikasi dan implementasikan autentikasi proxy sebagaimana dijelaskan dalam Token authentication. Sebelum setiap koneksi baru, client harus memperoleh kredensial koneksi baru dari server aplikasi.

Import the SDK

Impor SDK untuk platform pengembangan Anda. Implementasi client menggunakan iOS Swift. Platform lain menyediakan antarmuka dan alur event yang sama. Tutorial ini menggunakan aliran audio PCM. Pengkodean Opus disediakan oleh plugin. Impor plugin Opus jika uplink menggunakan Opus.

android {
    defaultConfig {
        minSdk 21
        ndk { abiFilters 'armeabi-v7a', 'arm64-v8a' }
    }
}

dependencies {
    implementation fileTree(dir: 'libs', include: ['*.aar'])
}

<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />
<uses-permission android:name="android.permission.CAMERA" />

{
  "dependencies": {
    "@aoq/client-sdk": "file:./libs/AoqClientSdk.har"
  }
}

"requestPermissions": [
  { "name": "ohos.permission.INTERNET" },
  { "name": "ohos.permission.MICROPHONE",
    "reason": "$string:perm_mic_reason",
    "usedScene": { "abilities": ["EntryAbility"], "when": "inuse" } },
  { "name": "ohos.permission.CAMERA",
    "reason": "$string:perm_camera_reason",
    "usedScene": { "abilities": ["EntryAbility"], "when": "inuse" } }
]

export PYTHONPATH="$PWD/AoqClientSdk:$PYTHONPATH"
export LD_LIBRARY_PATH="$PWD/AoqClientSdk:$LD_LIBRARY_PATH"

Implementation flow

1. Server aplikasi memperoleh parameter koneksi AOQ untuk qwen3.5-omni-plus-realtime dari URL token Realtime.
2. Client membuat engine dan mengonfigurasi codec audio serta track. Client juga mengonfigurasi Video track jika diperlukan pemahaman visual kontinu.
3. Client memulai capture dan pemutaran lokal, menonaktifkan pengiriman Audio-track secara default, terhubung ke AOQ, lalu mengirim session.update.
4. Setelah session.updated diterima, opsi continuous-video mengaktifkan Video track. Audio track tetap dinonaktifkan hingga pengguna menekan tombol bicara.
5. Saat pengguna menekan tombol, client mengaktifkan Audio track. Saat dilepas, client menonaktifkan Audio track, secara opsional mengirim gambar, lalu mengirim input_audio_buffer.commit dan response.create.
6. Setelah response.done diterima, giliran berikutnya dapat dimulai. Untuk mengakhiri, hentikan perangkat, putuskan koneksi, dan hapus engine.

Continuous Video track

Publikasikan Video track dan aktifkan pengiriman video setelah session.updated. Model terus-menerus melihat frame terbaru. Setiap giliran suara hanya perlu meng-commit audio dan meminta respons.

Send an image over the Data track

Jangan publikasikan Video track. Saat diperlukan gambar, kirim input_image_buffer.append setelah tombol dilepas, lalu commit audio dan minta respons.

Obtain a token from the application server

Atur DASHSCOPE_API_KEY di server aplikasi dan kirim permintaan ke endpoint untuk wilayah yang dipilih. clientIp adalah Alamat IP publik aktual dari client. Field ini opsional, tetapi menentukannya membantu layanan mengalokasikan endpoint relay yang sesuai.

curl -X POST \
  "https://{endpoint}/api/v1/webrtc/realtime?model=qwen3.5-omni-plus-realtime" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${DASHSCOPE_API_KEY}" \
  -H "x-dashscope-rtc-transport: moq" \
  -d "{\"clientIp\": \"${CLIENT_REAL_IP}\"}"

Server aplikasi mengembalikan field respons berikut ke client. Jangan pernah mengembalikan Kunci API ke client di lingkungan produksi. Untuk semua field permintaan dan respons, lihat Token authentication.

Implement the iOS client

Setelah client memperoleh AoqConnectConfig dari server aplikasi, ikuti langkah-langkah berikut untuk mengimplementasikan percakapan suara push-to-talk di iOS.

1. Create the engine and register callbacks

Buat engine AOQ singleton dan daftarkan objek aplikasi sebagai penerima callback. Tangani status koneksi, event server, error, dan peringatan dalam callback.

let createConfig = AoqCreateConfig()
createConfig.workDir = workDir
engine = AoqClientEngine.createEngine(createConfig, delegate: self)

2. Start audio and video devices

Inisialisasi capture dan pemutaran audio. Mulai kamera hanya untuk opsi continuous-video. Peroleh izin mikrofon dan kamera sebelum metode ini dipanggil.

let captureConfig = AoqAudioCaptureConfig()
captureConfig.channel = 1
captureConfig.isExternal = false
engine.startAudioCapture(captureConfig)

let playbackConfig = AoqAudioPlaybackConfig()
playbackConfig.channel = 1
playbackConfig.isExternal = false
playbackConfig.isDefaultSpeaker = true
engine.startAudioPlayer(playbackConfig)

3. Configure codecs and tracks

Konfigurasikan codec audio untuk model yang dipilih dan format audio aplikasi, serta pilih track untuk opsi input gambar. Nilai audio dan video berikut merupakan contoh. Sesuaikan dengan kebutuhan model dan skenario aplikasi. Nonaktifkan pengiriman Audio-track sebelum koneksi dibuat.

let audioEncoderConfig = AoqAudioCodecConfig()
audioEncoderConfig.trackType = .audio
audioEncoderConfig.codecType = .audioPCM
audioEncoderConfig.sampleRate = 16_000
audioEncoderConfig.channel = 1
engine.setAudioEncoderConfig(audioEncoderConfig)

let audioDecoderConfig = AoqAudioCodecConfig()
audioDecoderConfig.trackType = .audio
audioDecoderConfig.codecType = .audioPCM
audioDecoderConfig.sampleRate = 24_000
audioDecoderConfig.channel = 1
engine.setAudioDecoderConfig(audioDecoderConfig)

let videoEncoderConfig = AoqVideoCodecConfig()
videoEncoderConfig.trackType = .video
videoEncoderConfig.codecType = .videoJpeg
videoEncoderConfig.width = 960
videoEncoderConfig.height = 540
videoEncoderConfig.fps = 2
videoEncoderConfig.bitrate = 500_000
engine.setVideoEncoderConfig(videoEncoderConfig)

let publishAudioTrack = AoqTrackParam()
publishAudioTrack.trackType = .audio
let publishVideoTrack = AoqTrackParam()
publishVideoTrack.trackType = .video
let publishDataTrack = AoqTrackParam()
publishDataTrack.trackType = .data
let subscribeAudioTrack = AoqTrackParam()
subscribeAudioTrack.trackType = .audio
let subscribeDataTrack = AoqTrackParam()
subscribeDataTrack.trackType = .data
connectConfig.publishTracks = [publishAudioTrack, publishVideoTrack, publishDataTrack]
connectConfig.subscribeTracks = [subscribeAudioTrack, subscribeDataTrack]

let audioEncoderConfig = AoqAudioCodecConfig()
audioEncoderConfig.trackType = .audio
audioEncoderConfig.codecType = .audioPCM
audioEncoderConfig.sampleRate = 16_000
audioEncoderConfig.channel = 1
engine.setAudioEncoderConfig(audioEncoderConfig)

let audioDecoderConfig = AoqAudioCodecConfig()
audioDecoderConfig.trackType = .audio
audioDecoderConfig.codecType = .audioPCM
audioDecoderConfig.sampleRate = 24_000
audioDecoderConfig.channel = 1
engine.setAudioDecoderConfig(audioDecoderConfig)

let publishAudioTrack = AoqTrackParam()
publishAudioTrack.trackType = .audio
let publishDataTrack = AoqTrackParam()
publishDataTrack.trackType = .data
let subscribeAudioTrack = AoqTrackParam()
subscribeAudioTrack.trackType = .audio
let subscribeDataTrack = AoqTrackParam()
subscribeDataTrack.trackType = .data
connectConfig.publishTracks = [publishAudioTrack, publishDataTrack]
connectConfig.subscribeTracks = [subscribeAudioTrack, subscribeDataTrack]

4. Configure a Manual session

Setelah koneksi dibuat, panggil sendDataMsg untuk mengirim event session.update. Atur turn_detection ke null dan pilih modalities voice, instruksi, dan output untuk aplikasi Anda. Pertahankan parameter audio contoh agar konsisten dengan pengaturan codec SDK. Untuk semua field, lihat Client events.

private func sendSessionUpdate() {
    let event: [String: Any] = [
        "type": "session.update",
        "session": [
            "modalities": ["text", "audio"],
            "voice": "Ethan",
            "audio": [
                "input": ["format": ["type": "pcm", "sample_rate": 16_000]],
                "output": ["format": ["type": "pcm", "sample_rate": 24_000]]
            ],
            "turn_detection": NSNull()
        ]
    ]
    guard let data = try? JSONSerialization.data(withJSONObject: event) else { return }
    let dataMessage = AoqDataMsg()
    dataMessage.data = data
    engine.sendDataMsg(dataMessage)
}

5. Wait for the session configuration

Tangani event session.updated dalam callback onDataMsg. Jangan kirim media hingga event ini diterima. Untuk opsi continuous-video, panggil enableSendMediaStream untuk mengaktifkan Video track pada titik ini, tetapi tetap nonaktifkan Audio track agar audio sebelum pengguna menekan tombol tidak masuk ke buffer input.

func onDataMsg(_ msg: AoqDataMsg) {
    guard let event = try? JSONSerialization.jsonObject(with: msg.data) as? [String: Any],
          let type = event["type"] as? String else { return }
    if type == "session.updated", imageMode == .continuousVideo {
        engine.enableSendMediaStream(.video, enable: true)
    }
    // Keep Audio-track sending disabled until the talk button is pressed.
}

6. Implement push-to-talk interaction

Saat tombol ditekan, panggil enableSendMediaStream untuk mengaktifkan Audio track. Saat dilepas, panggil enableSendMediaStream untuk menonaktifkan Audio track, pastikan giliran tersebut berisi audio, secara opsional kirim gambar, lalu panggil sendDataMsg untuk mengirim input_audio_buffer.commit diikuti response.create.

func onPushToTalkPressed() {
    hasAudioInCurrentTurn = true
    engine.enableSendMediaStream(.audio, enable: true)
}

func onPushToTalkReleased(base64Jpeg: String? = nil) {
    engine.enableSendMediaStream(.audio, enable: false)
    guard hasAudioInCurrentTurn else { return }
    if imageMode == .singleImage, let base64Jpeg {
        let imageEvent: [String: Any] = [
            "type": "input_image_buffer.append",
            "image": base64Jpeg
        ]
        if let data = try? JSONSerialization.data(withJSONObject: imageEvent) {
            let dataMessage = AoqDataMsg()
            dataMessage.data = data
            engine.sendDataMsg(dataMessage)
        }
    }
    for event in [
        ["type": "input_audio_buffer.commit"],
        ["type": "response.create"]
    ] {
        guard let data = try? JSONSerialization.data(withJSONObject: event) else { continue }
        let dataMessage = AoqDataMsg()
        dataMessage.data = data
        engine.sendDataMsg(dataMessage)
    }
    hasAudioInCurrentTurn = false
}

7. Select an image-input option

Pemahaman visual kontinu dan pertanyaan gambar sesekali menggunakan konfigurasi track dan perilaku pengiriman yang berbeda. Pilih opsi berdasarkan bandwidth, konsumsi daya, dan desain interaksi.

{
  "type": "input_image_buffer.append",
  "image": "<Base64-encoded JPEG data>"
}

8. Disconnect and destroy the engine

Saat sesi berakhir, putuskan koneksi dan hapus engine. disconnect atau destroy secara otomatis menutup perangkat media, sehingga Anda tidak perlu memanggil metode stop secara terpisah. AoqClientEngine adalah singleton dan tidak dapat dibuat lagi hingga destroy dipanggil.

engine.disconnect()
AoqClientEngine.destroy()

Complete example

Kelas berikut menerima AoqConnectConfig yang dipetakan dari respons token server aplikasi. Tambahkan status UI, izin, pemulihan error, dan kompresi gambar di lingkungan produksi.

import Foundation
import AoqClientSdk

final class ManualPushToTalkClient: NSObject, AoqEngineDelegate {
    enum ImageMode: Equatable {
        case none
        case continuousVideo
        case singleImage
    }

    private var engine: AoqClientEngine!
    private let imageMode: ImageMode
    private var hasAudioInCurrentTurn = false

    init(workDir: String, connectConfig: AoqConnectConfig, imageMode: ImageMode) {
        self.imageMode = imageMode
        super.init()
        let createConfig = AoqCreateConfig()
        createConfig.workDir = workDir
        self.engine = AoqClientEngine.createEngine(createConfig, delegate: self)

        // Example values. Match these settings to the selected model and application format.
        let audioEncoderConfig = AoqAudioCodecConfig()
        audioEncoderConfig.trackType = .audio
        audioEncoderConfig.codecType = .audioPCM
        audioEncoderConfig.sampleRate = 16_000
        audioEncoderConfig.channel = 1
        engine.setAudioEncoderConfig(audioEncoderConfig)

        let audioDecoderConfig = AoqAudioCodecConfig()
        audioDecoderConfig.trackType = .audio
        audioDecoderConfig.codecType = .audioPCM
        audioDecoderConfig.sampleRate = 24_000
        audioDecoderConfig.channel = 1
        engine.setAudioDecoderConfig(audioDecoderConfig)

        let publishAudioTrack = AoqTrackParam()
        publishAudioTrack.trackType = .audio
        let publishDataTrack = AoqTrackParam()
        publishDataTrack.trackType = .data
        let subscribeAudioTrack = AoqTrackParam()
        subscribeAudioTrack.trackType = .audio
        let subscribeDataTrack = AoqTrackParam()
        subscribeDataTrack.trackType = .data

        connectConfig.publishTracks = [publishAudioTrack, publishDataTrack]
        connectConfig.subscribeTracks = [subscribeAudioTrack, subscribeDataTrack]

        if imageMode == .continuousVideo {
            let videoEncoderConfig = AoqVideoCodecConfig()
            videoEncoderConfig.trackType = .video
            videoEncoderConfig.codecType = .videoJpeg
            videoEncoderConfig.width = 960
            videoEncoderConfig.height = 540
            videoEncoderConfig.fps = 2
            videoEncoderConfig.bitrate = 500_000
            engine.setVideoEncoderConfig(videoEncoderConfig)

            let publishVideoTrack = AoqTrackParam()
            publishVideoTrack.trackType = .video
            connectConfig.publishTracks = [publishAudioTrack, publishVideoTrack, publishDataTrack]
        }

        let captureConfig = AoqAudioCaptureConfig()
        captureConfig.channel = 1
        captureConfig.isExternal = false
        engine.startAudioCapture(captureConfig)

        let playbackConfig = AoqAudioPlaybackConfig()
        playbackConfig.channel = 1
        playbackConfig.isExternal = false
        playbackConfig.isDefaultSpeaker = true
        engine.startAudioPlayer(playbackConfig)

        if imageMode == .continuousVideo {
            let videoCaptureConfig = AoqVideoCaptureConfig()
            videoCaptureConfig.width = 1280
            videoCaptureConfig.height = 720
            videoCaptureConfig.fps = 15
            engine.startVideoCapture(videoCaptureConfig)
        }

        engine.enableSendMediaStream(.audio, enable: false)
        if imageMode == .continuousVideo {
            engine.enableSendMediaStream(.video, enable: false)
        }
        engine.connect(connectConfig)
    }

    func onPushToTalkPressed() {
        hasAudioInCurrentTurn = true
        engine.enableSendMediaStream(.audio, enable: true)
    }

    func onPushToTalkReleased(base64Jpeg: String? = nil) {
        engine.enableSendMediaStream(.audio, enable: false)
        guard hasAudioInCurrentTurn else { return }
        if imageMode == .singleImage, let base64Jpeg {
            let imageEvent: [String: Any] = [
                "type": "input_image_buffer.append",
                "image": base64Jpeg
            ]
            if let data = try? JSONSerialization.data(withJSONObject: imageEvent) {
                let dataMessage = AoqDataMsg()
                dataMessage.data = data
                engine.sendDataMsg(dataMessage)
            }
        }
        for event in [
            ["type": "input_audio_buffer.commit"],
            ["type": "response.create"]
        ] {
            guard let data = try? JSONSerialization.data(withJSONObject: event) else { continue }
            let dataMessage = AoqDataMsg()
            dataMessage.data = data
            engine.sendDataMsg(dataMessage)
        }
        hasAudioInCurrentTurn = false
    }

    private func sendSessionUpdate() {
        let event: [String: Any] = [
            "type": "session.update",
            "session": [
                "modalities": ["text", "audio"],
                "voice": "Ethan",
                "audio": [
                    "input": ["format": ["type": "pcm", "sample_rate": 16_000]],
                    "output": ["format": ["type": "pcm", "sample_rate": 24_000]]
                ],
                "turn_detection": NSNull()
            ]
        ]
        guard let data = try? JSONSerialization.data(withJSONObject: event) else { return }
        let dataMessage = AoqDataMsg()
        dataMessage.data = data
        engine.sendDataMsg(dataMessage)
    }

    func close() {
        engine.disconnect()
        AoqClientEngine.destroy()
    }

    func onConnectionStatusChange(_ status: AoqConnectionStatus) {
        if status == .connected { sendSessionUpdate() }
    }

    func onDataMsg(_ msg: AoqDataMsg) {
        guard let event = try? JSONSerialization.jsonObject(with: msg.data) as? [String: Any],
              let type = event["type"] as? String else { return }
        if type == "session.updated", imageMode == .continuousVideo {
            engine.enableSendMediaStream(.video, enable: true)
        }
    }

    func onError(_ code: Int, message: String) {}
    func onWarning(_ code: Int, message: String) {}
    func onStats(_ stats: AoqStats) {}
    func onAudioDeviceStateChanged(_ state: AoqAudioDeviceState) {}
    func onAudioDeviceRouteChanged(_ routeType: Int) {}
    func onAudioDeviceInterrupted(_ interrupt: Bool) {}
    func onAudioFileState(_ state: AoqAudioFileState) {}
    func onVideoDeviceStateChanged(_ state: AoqVideoDeviceState) {}
}

Run and verify

Lengkapi satu giliran push-to-talk hanya audio dan satu giliran dengan gambar. Hasil yang diharapkan:

1. Audio track dinonaktifkan sebelum tombol ditekan dan mengirim audio secara kontinu selama tombol ditekan.
2. Setelah dilepas, input_audio_buffer.committed, response.created, dan response.done diterima secara berurutan, dan audio model diputar melalui Audio track yang berlangganan.
3. Dengan opsi single-image, model merespons menggunakan gambar dan audio dari giliran tersebut. Dengan video kontinu, model menggunakan frame video terbaru.

Untuk field event server dan skema respons lengkap, lihat Server events.

Important considerations

1. AOQ mentransmisikan audio melalui Audio track. Jangan kirim input_audio_buffer.append juga.
2. input_audio_buffer.commit hanya meng-commit giliran dan tidak memicu respons model. Kirim response.create setelahnya.
3. Jangan commit buffer audio kosong. Layanan akan mengembalikan error.
4. Jangan aktifkan pengiriman media sebelum session.updated. Di mode Manual, jangan aktifkan Audio track sebelum pengguna menekan tombol.

Related information

Untuk semua parameter, field event, dan antarmuka untuk platform lain, lihat:

• Obtain and configure an API key
• Select a region, deployment scope, and endpoint
• AOQ Client SDK overview
• SDK download
• Token authentication
• Qwen-Omni-Realtime user guide
• Build VAD-mode real-time conversations over AOQ
• Client events
• Server events