Using AOQ with qwen3.5-omni-plus-realtime for real-time calls - Alibaba Cloud Model Studio

Describes how to integrate the AOQ Client SDK on Android, iOS, and HarmonyOS to implement audio/video calls using AOQ and qwen3.5-omni-plus-realtime.

Get the SDK

For the AOQ Client SDK and the Opus audio plug-in, see SDK download. The Opus codec is provided as a separate plug-in. Include it as needed for your use case.

Import the SDK

Copy the core SDK files to your project's dependency directory and declare the required permissions in your project configuration.

Android

Place AoqClientSdk-release.aar in your project's app/libs/ directory. Place libPluginOpus.so by ABI into app/libs/armeabi-v7a/ and app/libs/arm64-v8a/. In app/build.gradle:

android {
    defaultConfig {
        minSdk 21
        ndk { abiFilters 'armeabi-v7a', 'arm64-v8a' }
    }
    sourceSets { main { jniLibs.srcDirs = ['libs'] } }
    packagingOptions {
        // Avoid conflicts with same-named .so files in the host project
        pickFirsts += ['lib/*/*.so']
    }
}

dependencies {
    implementation fileTree(dir: 'libs', include: ['*.aar'])
}

Declare permissions in AndroidManifest.xml:

<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />
<uses-permission android:name="android.permission.CAMERA" />

RECORD_AUDIO and CAMERA are runtime permissions. Your app must call Android's ActivityCompat.requestPermissions() at runtime to request user authorization.

iOS (framework)

Drag AoqClientSdk.framework and PluginOpus.framework into your Xcode project. Under Target > General > Frameworks, Libraries, and Embedded Content, select Embed & Sign.
Declare permissions: in Xcode, select your Target > Info > Custom iOS Target Properties, and add the following two permission usage descriptions:

Key
Value
NSMicrophoneUsageDescription
For real-time voice calls
NSCameraUsageDescription
For real-time video calls
Swift project: import AoqClientSdk. Objective-C project: #import <AoqClientSdk/AoqClientSdk.h>.

Key	Value
`NSMicrophoneUsageDescription`	For real-time voice calls
`NSCameraUsageDescription`	For real-time video calls

HarmonyOS (har)

Place aoq-client-sdk.har in your project's libs/ directory. Place libPluginOpus.so by ABI into entry/libs/armeabi-v7a/ and entry/libs/arm64-v8a/. Declare the dependency in entry/oh-package.json5.
Add permissions in entry/src/main/module.json5:

"requestPermissions": [
  { "name": "ohos.permission.INTERNET" },
  { "name": "ohos.permission.MICROPHONE",
    "reason": "$string:perm_mic_reason",
    "usedScene": { "abilities": ["EntryAbility"], "when": "inuse" } },
  { "name": "ohos.permission.CAMERA",
    "reason": "$string:perm_camera_reason",
    "usedScene": { "abilities": ["EntryAbility"], "when": "inuse" } }
]

In EntryAbility, trigger runtime authorization via abilityAccessCtrl.createAtManager().requestPermissionsFromUser.

Try the demo

Use the Android demo from Alibaba Cloud Model Studio to quickly verify AOQ connectivity. Download the APK and configure the API key and workspaceId to try selected models.

Scan the following QR code to download and install the Android demo:

Get a token from the AppServer

Follow the AOQ section in Token authentication to set up an AppServer that obtains tokens. Before each call, the client must request a token from your AppServer.

Implement AI audio/video calls

Create engine and set callbacks

Call the createEngine API to create an AoqClientEngine instance.

iOS:

let config = AoqCreateConfig()
config.workDir = workDir
engine = AoqClientEngine.createEngine(config, delegate: self)

Implement the AoqEngineDelegate protocol to listen for onConnectionStatusChange, onDataMsg, onError, and other callbacks.

Android:

AoqCreateConfig config = new AoqCreateConfig();
config.workDir = appCtx.getFilesDir().getAbsolutePath();
engine = AoqClientEngine.createEngine(appCtx, config, this);

HarmonyOS:

const config: AoqCreateConfig = { workDir: context.filesDir, extras: '' };
engine = AoqClientEngine.createEngine(config, this, context);

Start audio/video capture and playback

Call startAudioCapture and startAudioPlayer to start local audio capture and playback. Call startVideoCapture to start the camera, and use setLocalView to bind the SDK's render target to your preview control.

iOS:

// Audio capture
let capCfg = AoqAudioCaptureConfig()
capCfg.channel = 1; capCfg.isExternal = false
engine.startAudioCapture(capCfg)

// Audio playback
let playCfg = AoqAudioPlaybackConfig()
playCfg.channel = 1; playCfg.isExternal = false
engine.startAudioPlayer(playCfg)

// Video capture
let vidCfg = AoqVideoCaptureConfig()
vidCfg.width = 720; vidCfg.height = 1280; vidCfg.fps = 15
engine.startVideoCapture(vidCfg)

// Set render view for local video preview
let canvas = AoqVideoCanvas()
canvas.view = localPreview
canvas.renderMode = .crop
engine.setLocalView(.video, canvas: canvas)

Android:

// Audio capture
AoqAudioCaptureConfig capCfg = new AoqAudioCaptureConfig();
capCfg.channel = 1; capCfg.isExternal = false;
engine.startAudioCapture(capCfg);

// Audio playback
AoqAudioPlaybackConfig playCfg = new AoqAudioPlaybackConfig();
playCfg.channel = 1; playCfg.isExternal = false;
engine.startAudioPlayer(playCfg);

// Video capture
AoqVideoCaptureConfig vidCfg = new AoqVideoCaptureConfig();
vidCfg.width = 720; vidCfg.height = 1280; vidCfg.fps = 15;
engine.startVideoCapture(vidCfg);

// Set render view for local video preview
AoqVideoCanvas canvas = new AoqVideoCanvas();
canvas.view = localPreview;
canvas.renderMode = AoqRenderMode.AoqRenderModeCrop;
engine.setLocalView(AoqTrackType.AoqTrackTypeVideo, canvas);

HarmonyOS:

// Audio capture
const capCfg: AoqAudioCaptureConfig = { channel: 1, isExternal: false };
engine.startAudioCapture(capCfg);

// Audio playback
const playCfg: AoqAudioPlaybackConfig = { channel: 1, isExternal: false };
engine.startAudioPlayer(playCfg);

// Video capture
const vidCfg: AoqVideoCaptureConfig = { width: 720, height: 1280, fps: 15, isExternal: false };
engine.startVideoCapture(vidCfg);

// Set render view for local video preview
const canvas: AoqVideoCanvas = { view: localCtrl, renderMode: AoqRenderMode.AoqRenderModeCrop };
engine.setLocalView(AoqTrackType.AoqTrackTypeVideo, canvas);

Get connection credentials

Have your AppServer proxy the request to Alibaba Cloud Model Studio. See Token authentication.

Configure codecs and establish connection

Set codec parameters and call connect.

Note: qwen3.5-omni-plus-realtime requires the client to start sending media data only after receiving session.updated from the server. To avoid accidentally sending media during the window between a successful connect and the arrival of session.updated, call enableSendMediaStream(trackType, false) for each upstream track before calling connect. For WebSocket event details, see Client events.

iOS:

// Audio codec configuration
let encCfg = AoqAudioCodecConfig()
encCfg.codecType = .audioPCM; encCfg.sampleRate = 16000; encCfg.channel = 1
engine.setAudioEncoderConfig(encCfg)
engine.setAudioDecoderConfig(encCfg)

// Disable media sending before connect; enable after receiving session.updated
engine.enableSendMediaStream(.audio, enable: false)
engine.enableSendMediaStream(.video, enable: false)

// Establish connection
let conn = AoqConnectConfig()
conn.token = token; conn.sid = sid; conn.certFingerprint = cert
conn.relayEndpoints = endpoints; conn.workspaceIdHash = workspaceIdHash

let aTrack = AoqTrackParam(); aTrack.trackType = .audio
let vTrack = AoqTrackParam(); vTrack.trackType = .video
let dTrack = AoqTrackParam(); dTrack.trackType = .data
conn.publishTracks   = [aTrack, vTrack, dTrack]
conn.subscribeTracks = [aTrack, dTrack]
engine.connect(conn)

Android:

// Audio codec configuration
AoqAudioCodecConfig encCfg = new AoqAudioCodecConfig();
encCfg.codecType = AoqEncoderType.AoqEncoderTypeAudioPCM;
encCfg.sampleRate = 16000; encCfg.channel = 1;
engine.setAudioEncoderConfig(encCfg);
engine.setAudioDecoderConfig(encCfg);

// Disable media sending before connect; enable after receiving session.updated
engine.enableSendMediaStream(AoqTrackType.AoqTrackTypeAudio, false);
engine.enableSendMediaStream(AoqTrackType.AoqTrackTypeVideo, false);

// Establish connection
AoqConnectConfig conn = new AoqConnectConfig();
conn.token = token; conn.sid = sid; conn.certFingerprint = cert;
conn.relayEndpoints.addAll(endpoints); conn.workspaceIdHash = workspaceIdHash;

AoqTrackParam aTrack = new AoqTrackParam(); aTrack.trackType = AoqTrackType.AoqTrackTypeAudio;
AoqTrackParam vTrack = new AoqTrackParam(); vTrack.trackType = AoqTrackType.AoqTrackTypeVideo;
AoqTrackParam dTrack = new AoqTrackParam(); dTrack.trackType = AoqTrackType.AoqTrackTypeData;
conn.publishTracks.add(aTrack);
conn.publishTracks.add(vTrack);
conn.publishTracks.add(dTrack);
conn.subscribeTracks.add(aTrack);
conn.subscribeTracks.add(dTrack);
engine.connect(conn);

HarmonyOS:

// Audio codec configuration
const encCfg: AoqAudioCodecConfig = {
  codecType: AoqEncoderType.AoqEncoderTypeAudioPCM,
  sampleRate: 16000, channel: 1
};
engine.setAudioEncoderConfig(encCfg);
engine.setAudioDecoderConfig(encCfg);

// Disable media sending before connect; enable after receiving session.updated
engine.enableSendMediaStream(AoqTrackType.AoqTrackTypeAudio, false);
engine.enableSendMediaStream(AoqTrackType.AoqTrackTypeVideo, false);

// Establish connection
const conn: AoqConnectConfig = {
  token, sid, certFingerprint: cert,
  relayEndpoints: endpoints,
  workspaceIdHash,
  publishTracks: [
    { trackType: AoqTrackType.AoqTrackTypeAudio },
    { trackType: AoqTrackType.AoqTrackTypeVideo },
    { trackType: AoqTrackType.AoqTrackTypeData }
  ],
  subscribeTracks: [
    { trackType: AoqTrackType.AoqTrackTypeAudio },
    { trackType: AoqTrackType.AoqTrackTypeData }
  ]
};
engine.connect(conn);

Important: The AOQ SDK starts sending media data by default after connecting. This example demonstrates how to disable media sending when connecting to a model.

Configure AI session

In the onConnectionStatusChange(Connected) callback, send a session.update message via sendDataMsg. This message contains session parameters such as modalities, voice, instructions, and turn_detection, completing the session handshake. For WebSocket event details, see Client events.

iOS:

func onConnectionStatusChange(_ status: AoqConnectionStatus) {
    if status == .connected { sendSessionUpdate() }
}

private func sendSessionUpdate() {
    let json = """
    {
      // Event ID, generated by the client
      "event_id": "event_ToPZqeobitzUJnt3QqtWg",
      // Event type, always session.update
      "type": "session.update",
      // Session configuration
      "session": {
          // Output modalities: ["text"] (text only) or ["text","audio"] (text and audio)
          "modalities": [
              "text",
              "audio"
          ],
          // Output audio voice
          "voice": "Ethan",
          // Input audio format; currently only "pcm" is supported. Input audio is 16 kHz PCM.
          "input_audio_format": "pcm",
          // Output audio format; currently only "pcm" is supported. Output audio is 24 kHz PCM.
          "output_audio_format": "pcm",
          // System message for setting the model's objective or role.
          "instructions": "You are an AI customer service agent for a five-star hotel. Accurately and helpfully answer customer inquiries about room types, facilities, pricing, and reservation policies. Always respond professionally and helpfully. Do not provide unverified information or information outside the scope of hotel services.",
          // Whether to enable voice activity detection. If enabled, pass a config object and the server will automatically detect voice start and end.
          // Set to null to let the client control when to trigger a model response.
          "turn_detection": {
              // VAD type: server_vad or semantic_vad. Recommended: semantic_vad for qwen3.5-omni-realtime.
              "type": "semantic_vad",
              // VAD detection threshold. Increase in noisy environments; decrease in quiet ones.
              "threshold": 0.5,
              // Duration of silence after which a model response is triggered
              "silence_duration_ms": 800
          }
      }
    }
    """
    let msg = AoqDataMsg()
    msg.data = json.data(using: .utf8)!
    engine.send(msg)
}

Android:

@Override
public void onConnectionStatusChange(AoqConnectionStatus status) {
    if (status == AoqConnectionStatus.AoqConnectionStatusConnected) {
        sendSessionUpdate();
    }
}

private void sendSessionUpdate() {
    String sessionUpdateJson = /* Same session.update JSON as in the Swift example above */;
    AoqDataMsg msg = new AoqDataMsg();
    msg.data = sessionUpdateJson.getBytes(StandardCharsets.UTF_8);
    engine.sendDataMsg(msg);
}

HarmonyOS:

onConnectionStatusChange(status: AoqConnectionStatus): void {
  if (status === AoqConnectionStatus.AoqConnectionStatusConnected) {
    this.sendSessionUpdate();
  }
}

private sendSessionUpdate(): void {
  const sessionUpdateJson = /* Same session.update JSON as in the Swift example above */;
  const msg: AoqDataMsg = { data: new TextEncoder().encode(sessionUpdateJson).buffer };
  this.engine.sendDataMsg(msg);
}

Enable media sending after receiving session.updated

In the onDataMsg callback, parse incoming messages. When session.updated is received from the model, call enableSendMediaStream(trackType, true) for each track type that was disabled earlier. For WebSocket event details, see Server events.

iOS:

func onDataMsg(_ msg: AoqDataMsg) {
    guard let obj = try? JSONSerialization.jsonObject(with: msg.data) as? [String: Any],
          let type = obj["type"] as? String else { return }
    if type == "session.updated" {
        engine.enableSendMediaStream(.audio, enable: true)
        engine.enableSendMediaStream(.video, enable: true)
    }
}

Android:

@Override
public void onDataMsg(AoqDataMsg msg) {
    if (msg == null || msg.data == null) return;
    try {
        JSONObject obj = new JSONObject(new String(msg.data, StandardCharsets.UTF_8));
        if ("session.updated".equals(obj.optString("type"))) {
            engine.enableSendMediaStream(AoqTrackType.AoqTrackTypeAudio, true);
            engine.enableSendMediaStream(AoqTrackType.AoqTrackTypeVideo, true);
        }
    } catch (JSONException ignored) {}
}

HarmonyOS:

onDataMsg(msg: AoqDataMsg): void {
  if (!msg?.data) return;
  try {
    const text = new TextDecoder('utf-8').decode(new Uint8Array(msg.data));
    const obj = JSON.parse(text) as { type?: string };
    if (obj.type === 'session.updated') {
      this.engine.enableSendMediaStream(AoqTrackType.AoqTrackTypeAudio, true);
      this.engine.enableSendMediaStream(AoqTrackType.AoqTrackTypeVideo, true);
    }
  } catch (_) { /* Not JSON, ignore */ }
}

Important

Start sending media streams only after receiving session.updated. The AI may not be ready to receive data before this event.
The audio and video tracks added during connection (AOQ media channels) automatically deliver data to the server.
1. Audio: transmitted directly through the audio track — no need to send input_audio_buffer.append events.
2. Video: frames are sent through the video track — no need to send input_image_buffer.append events.

Disconnect and destroy engine

engine.disconnect()
AoqClientEngine.destroy()

Common scenarios

Barge-in

The SDK is deeply integrated with Alibaba Cloud Model Studio. Barge-in messages from the model interrupt the previous turn when a new turn begins.
The SDK provides the interruptAudioPlayer interface for local playback interruption. Call this API when the user wants to stop playback.

// iOS
engine.interruptAudioPlayer(.audio, fadeMs: 100)

Mute / unmute

After muting, the SDK continues to capture audio but sends only silent frames. The session is not interrupted.

engine.muteAudioCapture(true);   // Mute microphone (capture continues, but only silent frames are sent)
engine.muteAudioCapture(false);  // Unmute

Switch between front and rear cameras

// Pass the desired camera direction enum
engine.switchCamera(AoqCameraDirection.AoqCameraDirectionFront);
engine.switchCamera(AoqCameraDirection.AoqCameraDirectionBack);

Call subtitles and ASR results

The server pushes ASR results and AI text responses through downstream data messages. In your app, route them by the type field in the onDataMsg callback. For WebSocket event details, see Server events.

Important notes

Singleton semantics: createEngine is a singleton — repeated calls return the same instance. The engine can only be recreated after destroy is called. For multi-page use, manage the engine lifecycle at the Application or Ability level.
Local preview view types:
- Android: SurfaceView or TextureView. Other types are not supported.
- iOS: Any UIView subclass.
- HarmonyOS: See the SDK documentation.
Audio route changes: Events such as headphone insertion/removal or Bluetooth connection trigger onAudioDeviceRouteChanged. No handling is usually required in your app. If your UI includes a speaker/earpiece toggle, sync the state based on this callback.
Background audio continuation: For audio to continue when the call moves to the background, Info.plist must have UIBackgroundModes = audio enabled and AVAudioSession must be correctly activated in the foreground. The SDK handles most cases; use setAudioSessionRestriction: for fine-grained control.

iOS demo source code

To learn how to implement AOQ on iOS, download the sample source code: aoqdemo.zip.

Detailed AOQ Client SDK API: SDK overview
qwen3.5-omni-plus-realtime model client events: Client events
qwen3.5-omni-plus-realtime model server events: Server events

Get the SDK

Import the SDK

Android

iOS (framework)

HarmonyOS (har)

Try the demo

Get a token from the AppServer

Implement AI audio/video calls

Create engine and set callbacks

Start audio/video capture and playback

Get connection credentials

Configure codecs and establish connection

Configure AI session

Enable media sending after receiving session.updated

Disconnect and destroy engine

Common scenarios

Barge-in

Mute / unmute

Switch between front and rear cameras

Call subtitles and ASR results

Important notes

iOS demo source code

Related documents