Alibaba Cloud Model Studio:Build push-to-talk voice conversations with qwen3.5-omni-plus-realtime over AOQ

Use AOQ to connect to qwen3.5-omni-plus-realtime and let the client control turn boundaries for push-to-talk conversations and optional image questions. The client code uses iOS Swift.

Solution overview

Qwen-Omni-Realtime supports server-side VAD and client-controlled Manual mode. This tutorial sets session.turn_detection to null. The client sends audio while the user holds a button, and commits the audio and explicitly requests a response when the user releases the button.

Manual mode is suitable for hardware intercom buttons, press-and-hold controls, noisy environments in which the application determines turn boundaries, and turns that optionally include an image. Audio is transported over the AOQ Audio track. Do not send input_audio_buffer.append.

Prerequisites

1. Activate Model Studio and follow Obtain and configure an API key. Store the API key only on your application server. Do not include it in client code or commit it to a code repository.
2. Confirm the AOQ endpoint for the region in which your application is deployed. For selection guidance, see Select a region, deployment scope, and endpoint.
3. Download the latest AOQ Client SDK as described in SDK download.
4. Build an application server and implement proxy authentication as described in Token authentication. Before each new connection, the client must obtain new connection credentials from the application server.

Import the SDK

Import the SDK for your development platform. The client implementation uses iOS Swift. Other platforms provide the same interfaces and event flow. This tutorial uses PCM audio streams. Opus encoding is provided by a plugin. Import the Opus plugin if the uplink uses Opus.

android {
    defaultConfig {
        minSdk 21
        ndk { abiFilters 'armeabi-v7a', 'arm64-v8a' }
    }
}

dependencies {
    implementation fileTree(dir: 'libs', include: ['*.aar'])
}

<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />
<uses-permission android:name="android.permission.CAMERA" />

{
  "dependencies": {
    "@aoq/client-sdk": "file:./libs/AoqClientSdk.har"
  }
}

"requestPermissions": [
  { "name": "ohos.permission.INTERNET" },
  { "name": "ohos.permission.MICROPHONE",
    "reason": "$string:perm_mic_reason",
    "usedScene": { "abilities": ["EntryAbility"], "when": "inuse" } },
  { "name": "ohos.permission.CAMERA",
    "reason": "$string:perm_camera_reason",
    "usedScene": { "abilities": ["EntryAbility"], "when": "inuse" } }
]

export PYTHONPATH="$PWD/AoqClientSdk:$PYTHONPATH"
export LD_LIBRARY_PATH="$PWD/AoqClientSdk:$LD_LIBRARY_PATH"

Implementation flow

1. The application server obtains AOQ connection parameters for qwen3.5-omni-plus-realtime from the Realtime token URL.
2. The client creates the engine and configures audio codecs and tracks. It also configures the Video track if continuous visual understanding is required.
3. The client starts local capture and playback, disables Audio-track sending by default, connects to AOQ, and sends session.update.
4. After session.updated is received, the continuous-video option enables the Video track. The Audio track remains disabled until the user presses the talk button.
5. When the user presses the button, the client enables the Audio track. On release, it disables the Audio track, optionally sends an image, and then sends input_audio_buffer.commit and response.create.
6. After response.done is received, another turn can start. To finish, stop the devices, disconnect, and destroy the engine.

Continuous Video track

Publish the Video track and enable video sending after session.updated. The model continuously sees the latest frames. Each voice turn only needs to commit audio and request a response.

Send an image over the Data track

Do not publish the Video track. When an image is required, send input_image_buffer.append after the button is released, and then commit the audio and request a response.

Obtain a token from the application server

Set DASHSCOPE_API_KEY on the application server and send the request to the endpoint for the selected region. clientIp is the actual public IP address of the client. This field is optional, but specifying it helps the service allocate an appropriate relay endpoint.

curl -X POST \
  "https://{endpoint}/api/v1/webrtc/realtime?model=qwen3.5-omni-plus-realtime" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${DASHSCOPE_API_KEY}" \
  -H "x-dashscope-rtc-transport: moq" \
  -d "{\"clientIp\": \"${CLIENT_REAL_IP}\"}"

The application server returns the following response fields to the client. Never return the API key to a client in production. For all request and response fields, see Token authentication.

Implement the iOS client

After the client obtains AoqConnectConfig from the application server, follow these steps to implement push-to-talk voice conversations on iOS.

1. Create the engine and register callbacks

Create the singleton AOQ engine and register the application object as the callback receiver. Handle connection states, server events, errors, and warnings in the callbacks.

let createConfig = AoqCreateConfig()
createConfig.workDir = workDir
engine = AoqClientEngine.createEngine(createConfig, delegate: self)

2. Start audio and video devices

Initialize audio capture and playback. Start the camera only for the continuous-video option. Obtain microphone and camera permissions before these methods are called.

let captureConfig = AoqAudioCaptureConfig()
captureConfig.channel = 1
captureConfig.isExternal = false
engine.startAudioCapture(captureConfig)

let playbackConfig = AoqAudioPlaybackConfig()
playbackConfig.channel = 1
playbackConfig.isExternal = false
playbackConfig.isDefaultSpeaker = true
engine.startAudioPlayer(playbackConfig)

3. Configure codecs and tracks

Configure the audio codecs for the selected model and the application's audio format, and select tracks for the image-input option. The following audio and video values are examples. Adjust them for the model requirements and application scenario. Disable Audio-track sending before the connection is established.

let audioEncoderConfig = AoqAudioCodecConfig()
audioEncoderConfig.trackType = .audio
audioEncoderConfig.codecType = .audioPCM
audioEncoderConfig.sampleRate = 16_000
audioEncoderConfig.channel = 1
engine.setAudioEncoderConfig(audioEncoderConfig)

let audioDecoderConfig = AoqAudioCodecConfig()
audioDecoderConfig.trackType = .audio
audioDecoderConfig.codecType = .audioPCM
audioDecoderConfig.sampleRate = 24_000
audioDecoderConfig.channel = 1
engine.setAudioDecoderConfig(audioDecoderConfig)

let videoEncoderConfig = AoqVideoCodecConfig()
videoEncoderConfig.trackType = .video
videoEncoderConfig.codecType = .videoJpeg
videoEncoderConfig.width = 960
videoEncoderConfig.height = 540
videoEncoderConfig.fps = 2
videoEncoderConfig.bitrate = 500_000
engine.setVideoEncoderConfig(videoEncoderConfig)

let publishAudioTrack = AoqTrackParam()
publishAudioTrack.trackType = .audio
let publishVideoTrack = AoqTrackParam()
publishVideoTrack.trackType = .video
let publishDataTrack = AoqTrackParam()
publishDataTrack.trackType = .data
let subscribeAudioTrack = AoqTrackParam()
subscribeAudioTrack.trackType = .audio
let subscribeDataTrack = AoqTrackParam()
subscribeDataTrack.trackType = .data
connectConfig.publishTracks = [publishAudioTrack, publishVideoTrack, publishDataTrack]
connectConfig.subscribeTracks = [subscribeAudioTrack, subscribeDataTrack]

let audioEncoderConfig = AoqAudioCodecConfig()
audioEncoderConfig.trackType = .audio
audioEncoderConfig.codecType = .audioPCM
audioEncoderConfig.sampleRate = 16_000
audioEncoderConfig.channel = 1
engine.setAudioEncoderConfig(audioEncoderConfig)

let audioDecoderConfig = AoqAudioCodecConfig()
audioDecoderConfig.trackType = .audio
audioDecoderConfig.codecType = .audioPCM
audioDecoderConfig.sampleRate = 24_000
audioDecoderConfig.channel = 1
engine.setAudioDecoderConfig(audioDecoderConfig)

let publishAudioTrack = AoqTrackParam()
publishAudioTrack.trackType = .audio
let publishDataTrack = AoqTrackParam()
publishDataTrack.trackType = .data
let subscribeAudioTrack = AoqTrackParam()
subscribeAudioTrack.trackType = .audio
let subscribeDataTrack = AoqTrackParam()
subscribeDataTrack.trackType = .data
connectConfig.publishTracks = [publishAudioTrack, publishDataTrack]
connectConfig.subscribeTracks = [subscribeAudioTrack, subscribeDataTrack]

4. Configure a Manual session

After the connection is established, call sendDataMsg to send a session.update event. Set turn_detection to null and select the voice, instructions, and output modalities for your application. Keep the example audio parameters consistent with the SDK codec settings. For all fields, see Client events.

private func sendSessionUpdate() {
    let event: [String: Any] = [
        "type": "session.update",
        "session": [
            "modalities": ["text", "audio"],
            "voice": "Ethan",
            "audio": [
                "input": ["format": ["type": "pcm", "sample_rate": 16_000]],
                "output": ["format": ["type": "pcm", "sample_rate": 24_000]]
            ],
            "turn_detection": NSNull()
        ]
    ]
    guard let data = try? JSONSerialization.data(withJSONObject: event) else { return }
    let dataMessage = AoqDataMsg()
    dataMessage.data = data
    engine.sendDataMsg(dataMessage)
}

5. Wait for the session configuration

Handle the session.updated event in the onDataMsg callback. Do not send media until this event is received. For the continuous-video option, call enableSendMediaStream to enable the Video track at this point, but keep the Audio track disabled so that audio before the user presses the button does not enter the input buffer.

func onDataMsg(_ msg: AoqDataMsg) {
    guard let event = try? JSONSerialization.jsonObject(with: msg.data) as? [String: Any],
          let type = event["type"] as? String else { return }
    if type == "session.updated", imageMode == .continuousVideo {
        engine.enableSendMediaStream(.video, enable: true)
    }
    // Keep Audio-track sending disabled until the talk button is pressed.
}

6. Implement push-to-talk interaction

When the button is pressed, call enableSendMediaStream to enable the Audio track. On release, call enableSendMediaStream to disable the Audio track, make sure that the turn contains audio, optionally send an image, and call sendDataMsg to send input_audio_buffer.commit followed by response.create.

func onPushToTalkPressed() {
    hasAudioInCurrentTurn = true
    engine.enableSendMediaStream(.audio, enable: true)
}

func onPushToTalkReleased(base64Jpeg: String? = nil) {
    engine.enableSendMediaStream(.audio, enable: false)
    guard hasAudioInCurrentTurn else { return }
    if imageMode == .singleImage, let base64Jpeg {
        let imageEvent: [String: Any] = [
            "type": "input_image_buffer.append",
            "image": base64Jpeg
        ]
        if let data = try? JSONSerialization.data(withJSONObject: imageEvent) {
            let dataMessage = AoqDataMsg()
            dataMessage.data = data
            engine.sendDataMsg(dataMessage)
        }
    }
    for event in [
        ["type": "input_audio_buffer.commit"],
        ["type": "response.create"]
    ] {
        guard let data = try? JSONSerialization.data(withJSONObject: event) else { continue }
        let dataMessage = AoqDataMsg()
        dataMessage.data = data
        engine.sendDataMsg(dataMessage)
    }
    hasAudioInCurrentTurn = false
}

7. Select an image-input option

Continuous visual understanding and occasional image questions use different track configurations and send behavior. Select an option based on bandwidth, power consumption, and interaction design.

{
  "type": "input_image_buffer.append",
  "image": "<Base64-encoded JPEG data>"
}

8. Disconnect and destroy the engine

When the session ends, disconnect and destroy the engine. disconnect or destroy automatically closes media devices, so you do not need to call stop methods separately. AoqClientEngine is a singleton and cannot be created again until destroy is called.

engine.disconnect()
AoqClientEngine.destroy()

Complete example

The following class accepts an AoqConnectConfig that was mapped from the application-server token response. Add UI state, permissions, error recovery, and image compression in production.

import Foundation
import AoqClientSdk

final class ManualPushToTalkClient: NSObject, AoqEngineDelegate {
    enum ImageMode: Equatable {
        case none
        case continuousVideo
        case singleImage
    }

    private var engine: AoqClientEngine!
    private let imageMode: ImageMode
    private var hasAudioInCurrentTurn = false

    init(workDir: String, connectConfig: AoqConnectConfig, imageMode: ImageMode) {
        self.imageMode = imageMode
        super.init()
        let createConfig = AoqCreateConfig()
        createConfig.workDir = workDir
        self.engine = AoqClientEngine.createEngine(createConfig, delegate: self)

        // Example values. Match these settings to the selected model and application format.
        let audioEncoderConfig = AoqAudioCodecConfig()
        audioEncoderConfig.trackType = .audio
        audioEncoderConfig.codecType = .audioPCM
        audioEncoderConfig.sampleRate = 16_000
        audioEncoderConfig.channel = 1
        engine.setAudioEncoderConfig(audioEncoderConfig)

        let audioDecoderConfig = AoqAudioCodecConfig()
        audioDecoderConfig.trackType = .audio
        audioDecoderConfig.codecType = .audioPCM
        audioDecoderConfig.sampleRate = 24_000
        audioDecoderConfig.channel = 1
        engine.setAudioDecoderConfig(audioDecoderConfig)

        let publishAudioTrack = AoqTrackParam()
        publishAudioTrack.trackType = .audio
        let publishDataTrack = AoqTrackParam()
        publishDataTrack.trackType = .data
        let subscribeAudioTrack = AoqTrackParam()
        subscribeAudioTrack.trackType = .audio
        let subscribeDataTrack = AoqTrackParam()
        subscribeDataTrack.trackType = .data

        connectConfig.publishTracks = [publishAudioTrack, publishDataTrack]
        connectConfig.subscribeTracks = [subscribeAudioTrack, subscribeDataTrack]

        if imageMode == .continuousVideo {
            let videoEncoderConfig = AoqVideoCodecConfig()
            videoEncoderConfig.trackType = .video
            videoEncoderConfig.codecType = .videoJpeg
            videoEncoderConfig.width = 960
            videoEncoderConfig.height = 540
            videoEncoderConfig.fps = 2
            videoEncoderConfig.bitrate = 500_000
            engine.setVideoEncoderConfig(videoEncoderConfig)

            let publishVideoTrack = AoqTrackParam()
            publishVideoTrack.trackType = .video
            connectConfig.publishTracks = [publishAudioTrack, publishVideoTrack, publishDataTrack]
        }

        let captureConfig = AoqAudioCaptureConfig()
        captureConfig.channel = 1
        captureConfig.isExternal = false
        engine.startAudioCapture(captureConfig)

        let playbackConfig = AoqAudioPlaybackConfig()
        playbackConfig.channel = 1
        playbackConfig.isExternal = false
        playbackConfig.isDefaultSpeaker = true
        engine.startAudioPlayer(playbackConfig)

        if imageMode == .continuousVideo {
            let videoCaptureConfig = AoqVideoCaptureConfig()
            videoCaptureConfig.width = 1280
            videoCaptureConfig.height = 720
            videoCaptureConfig.fps = 15
            engine.startVideoCapture(videoCaptureConfig)
        }

        engine.enableSendMediaStream(.audio, enable: false)
        if imageMode == .continuousVideo {
            engine.enableSendMediaStream(.video, enable: false)
        }
        engine.connect(connectConfig)
    }

    func onPushToTalkPressed() {
        hasAudioInCurrentTurn = true
        engine.enableSendMediaStream(.audio, enable: true)
    }

    func onPushToTalkReleased(base64Jpeg: String? = nil) {
        engine.enableSendMediaStream(.audio, enable: false)
        guard hasAudioInCurrentTurn else { return }
        if imageMode == .singleImage, let base64Jpeg {
            let imageEvent: [String: Any] = [
                "type": "input_image_buffer.append",
                "image": base64Jpeg
            ]
            if let data = try? JSONSerialization.data(withJSONObject: imageEvent) {
                let dataMessage = AoqDataMsg()
                dataMessage.data = data
                engine.sendDataMsg(dataMessage)
            }
        }
        for event in [
            ["type": "input_audio_buffer.commit"],
            ["type": "response.create"]
        ] {
            guard let data = try? JSONSerialization.data(withJSONObject: event) else { continue }
            let dataMessage = AoqDataMsg()
            dataMessage.data = data
            engine.sendDataMsg(dataMessage)
        }
        hasAudioInCurrentTurn = false
    }

    private func sendSessionUpdate() {
        let event: [String: Any] = [
            "type": "session.update",
            "session": [
                "modalities": ["text", "audio"],
                "voice": "Ethan",
                "audio": [
                    "input": ["format": ["type": "pcm", "sample_rate": 16_000]],
                    "output": ["format": ["type": "pcm", "sample_rate": 24_000]]
                ],
                "turn_detection": NSNull()
            ]
        ]
        guard let data = try? JSONSerialization.data(withJSONObject: event) else { return }
        let dataMessage = AoqDataMsg()
        dataMessage.data = data
        engine.sendDataMsg(dataMessage)
    }

    func close() {
        engine.disconnect()
        AoqClientEngine.destroy()
    }

    func onConnectionStatusChange(_ status: AoqConnectionStatus) {
        if status == .connected { sendSessionUpdate() }
    }

    func onDataMsg(_ msg: AoqDataMsg) {
        guard let event = try? JSONSerialization.jsonObject(with: msg.data) as? [String: Any],
              let type = event["type"] as? String else { return }
        if type == "session.updated", imageMode == .continuousVideo {
            engine.enableSendMediaStream(.video, enable: true)
        }
    }

    func onError(_ code: Int, message: String) {}
    func onWarning(_ code: Int, message: String) {}
    func onStats(_ stats: AoqStats) {}
    func onAudioDeviceStateChanged(_ state: AoqAudioDeviceState) {}
    func onAudioDeviceRouteChanged(_ routeType: Int) {}
    func onAudioDeviceInterrupted(_ interrupt: Bool) {}
    func onAudioFileState(_ state: AoqAudioFileState) {}
    func onVideoDeviceStateChanged(_ state: AoqVideoDeviceState) {}
}

Run and verify

Complete one audio-only push-to-talk turn and one turn with an image. Expected results:

1. The Audio track is disabled before the button is pressed and sends audio continuously while the button is held.
2. After release, input_audio_buffer.committed, response.created, and response.done are received in sequence, and model audio is played over the subscribed Audio track.
3. With the single-image option, the model responds using the image and audio from the turn. With continuous video, it uses the latest video frames.

For server event fields and complete response schemas, see Server events.

Important considerations

1. AOQ transports audio over the Audio track. Do not also send input_audio_buffer.append.
2. input_audio_buffer.commit only commits the turn and does not trigger a model response. Send response.create afterward.
3. Do not commit an empty audio buffer. The service returns an error.
4. Do not enable media sending before session.updated. In Manual mode, do not enable the Audio track before the user presses the button.

Related information

For all parameters, event fields, and interfaces for other platforms, see:

• Obtain and configure an API key
• Select a region, deployment scope, and endpoint
• AOQ Client SDK overview
• SDK download
• Token authentication
• Qwen-Omni-Realtime user guide
• Build VAD-mode real-time conversations over AOQ
• Client events
• Server events