Menggunakan AOQ dengan qwen3.5-omni-plus-realtime untuk panggilan real-time - Alibaba Cloud Model Studio

Menjelaskan cara mengintegrasikan AOQ Client SDK pada Android, iOS, dan HarmonyOS untuk mengimplementasikan panggilan audio/video menggunakan AOQ dan qwen3.5-omni-plus-realtime.

Get the SDK

Untuk AOQ Client SDK dan plug-in audio Opus, lihat SDK download. Kodek Opus disediakan sebagai plug-in terpisah. Sertakan sesuai kebutuhan kasus penggunaan Anda.

Import the SDK

Salin file SDK inti ke direktori dependensi proyek Anda dan deklarasikan izin yang diperlukan dalam konfigurasi proyek.

Android

Letakkan AoqClientSdk-release.aar di direktori app/libs/ proyek Anda. Letakkan libPluginOpus.so berdasarkan ABI ke dalam app/libs/armeabi-v7a/ dan app/libs/arm64-v8a/. Di app/build.gradle:

android {
    defaultConfig {
        minSdk 21
        ndk { abiFilters 'armeabi-v7a', 'arm64-v8a' }
    }
    sourceSets { main { jniLibs.srcDirs = ['libs'] } }
    packagingOptions {
        // Avoid conflicts with same-named .so files in the host project
        pickFirsts += ['lib/*/*.so']
    }
}

dependencies {
    implementation fileTree(dir: 'libs', include: ['*.aar'])
}

Deklarasikan izin di AndroidManifest.xml:

<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />
<uses-permission android:name="android.permission.CAMERA" />

RECORD_AUDIO dan CAMERA adalah izin runtime. Aplikasi Anda harus memanggil ActivityCompat.requestPermissions() milik Android saat runtime untuk meminta otorisasi pengguna.

iOS (framework)

Tarik AoqClientSdk.framework dan PluginOpus.framework ke dalam proyek Xcode Anda. Di bawah Target > General > Frameworks, Libraries, and Embedded Content, pilih Embed & Sign.
Deklarasikan izin: di Xcode, pilih Target Anda > Info > Custom iOS Target Properties, lalu tambahkan dua deskripsi penggunaan izin berikut:

Key
Value
NSMicrophoneUsageDescription
For real-time voice calls
NSCameraUsageDescription
For real-time video calls
Proyek Swift: import AoqClientSdk. Proyek Objective-C: #import <AoqClientSdk/AoqClientSdk.h>.

Key	Value
`NSMicrophoneUsageDescription`	For real-time voice calls
`NSCameraUsageDescription`	For real-time video calls

HarmonyOS (har)

Letakkan aoq-client-sdk.har di direktori libs/ proyek Anda. Letakkan libPluginOpus.so berdasarkan ABI ke dalam entry/libs/armeabi-v7a/ dan entry/libs/arm64-v8a/. Deklarasikan dependensi di entry/oh-package.json5.
Tambahkan izin di entry/src/main/module.json5:

"requestPermissions": [
  { "name": "ohos.permission.INTERNET" },
  { "name": "ohos.permission.MICROPHONE",
    "reason": "$string:perm_mic_reason",
    "usedScene": { "abilities": ["EntryAbility"], "when": "inuse" } },
  { "name": "ohos.permission.CAMERA",
    "reason": "$string:perm_camera_reason",
    "usedScene": { "abilities": ["EntryAbility"], "when": "inuse" } }
]

Di EntryAbility, picu otorisasi runtime melalui abilityAccessCtrl.createAtManager().requestPermissionsFromUser.

Try the demo

Gunakan demo Android dari Alibaba Cloud Model Studio untuk memverifikasi konektivitas AOQ secara cepat. Unduh APK dan konfigurasikan Kunci API serta workspaceId untuk mencoba model yang dipilih.

Pindai kode QR berikut untuk mengunduh dan menginstal demo Android:

Get a token from the AppServer

Ikuti bagian AOQ dalam Token authentication untuk menyiapkan AppServer yang mendapatkan token. Sebelum setiap panggilan, klien harus meminta token dari AppServer Anda.

Implement AI audio/video calls

Create engine and set callbacks

Panggil API createEngine untuk membuat instans AoqClientEngine.

iOS:

let config = AoqCreateConfig()
config.workDir = workDir
engine = AoqClientEngine.createEngine(config, delegate: self)

Implementasikan protokol AoqEngineDelegate untuk mendengarkan callback seperti onConnectionStatusChange, onDataMsg, onError, dan lainnya.

Android:

AoqCreateConfig config = new AoqCreateConfig();
config.workDir = appCtx.getFilesDir().getAbsolutePath();
engine = AoqClientEngine.createEngine(appCtx, config, this);

HarmonyOS:

const config: AoqCreateConfig = { workDir: context.filesDir, extras: '' };
engine = AoqClientEngine.createEngine(config, this, context);

Start audio/video capture and playback

Panggil startAudioCapture dan startAudioPlayer untuk memulai tangkapan dan pemutaran audio lokal. Panggil startVideoCapture untuk mengaktifkan kamera, dan gunakan setLocalView untuk mengikat target render SDK ke kontrol pratinjau Anda.

iOS:

// Audio capture
let capCfg = AoqAudioCaptureConfig()
capCfg.channel = 1; capCfg.isExternal = false
engine.startAudioCapture(capCfg)

// Audio playback
let playCfg = AoqAudioPlaybackConfig()
playCfg.channel = 1; playCfg.isExternal = false
engine.startAudioPlayer(playCfg)

// Video capture
let vidCfg = AoqVideoCaptureConfig()
vidCfg.width = 720; vidCfg.height = 1280; vidCfg.fps = 15
engine.startVideoCapture(vidCfg)

// Set render view for local video preview
let canvas = AoqVideoCanvas()
canvas.view = localPreview
canvas.renderMode = .crop
engine.setLocalView(.video, canvas: canvas)

Android:

// Audio capture
AoqAudioCaptureConfig capCfg = new AoqAudioCaptureConfig();
capCfg.channel = 1; capCfg.isExternal = false;
engine.startAudioCapture(capCfg);

// Audio playback
AoqAudioPlaybackConfig playCfg = new AoqAudioPlaybackConfig();
playCfg.channel = 1; playCfg.isExternal = false;
engine.startAudioPlayer(playCfg);

// Video capture
AoqVideoCaptureConfig vidCfg = new AoqVideoCaptureConfig();
vidCfg.width = 720; vidCfg.height = 1280; vidCfg.fps = 15;
engine.startVideoCapture(vidCfg);

// Set render view for local video preview
AoqVideoCanvas canvas = new AoqVideoCanvas();
canvas.view = localPreview;
canvas.renderMode = AoqRenderMode.AoqRenderModeCrop;
engine.setLocalView(AoqTrackType.AoqTrackTypeVideo, canvas);

HarmonyOS:

// Audio capture
const capCfg: AoqAudioCaptureConfig = { channel: 1, isExternal: false };
engine.startAudioCapture(capCfg);

// Audio playback
const playCfg: AoqAudioPlaybackConfig = { channel: 1, isExternal: false };
engine.startAudioPlayer(playCfg);

// Video capture
const vidCfg: AoqVideoCaptureConfig = { width: 720, height: 1280, fps: 15, isExternal: false };
engine.startVideoCapture(vidCfg);

// Set render view for local video preview
const canvas: AoqVideoCanvas = { view: localCtrl, renderMode: AoqRenderMode.AoqRenderModeCrop };
engine.setLocalView(AoqTrackType.AoqTrackTypeVideo, canvas);

Get connection credentials

Minta AppServer Anda untuk meneruskan permintaan ke Alibaba Cloud Model Studio. Lihat Token authentication.

Configure codecs and establish connection

Atur parameter kodek dan panggil connect.

Catatan: qwen3.5-omni-plus-realtime mengharuskan klien mulai mengirim data media hanya setelah menerima session.updated dari server. Untuk menghindari pengiriman media secara tidak sengaja selama jendela antara connect yang berhasil dan kedatangan session.updated, panggil enableSendMediaStream(trackType, false) untuk setiap track upstream sebelum memanggil connect. Untuk detail event WebSocket, lihat Client events.

iOS:

// Audio codec configuration
let encCfg = AoqAudioCodecConfig()
encCfg.codecType = .audioPCM; encCfg.sampleRate = 16000; encCfg.channel = 1
engine.setAudioEncoderConfig(encCfg)
engine.setAudioDecoderConfig(encCfg)

// Disable media sending before connect; enable after receiving session.updated
engine.enableSendMediaStream(.audio, enable: false)
engine.enableSendMediaStream(.video, enable: false)

// Establish connection
let conn = AoqConnectConfig()
conn.token = token; conn.sid = sid; conn.certFingerprint = cert
conn.relayEndpoints = endpoints; conn.workspaceIdHash = workspaceIdHash

let aTrack = AoqTrackParam(); aTrack.trackType = .audio
let vTrack = AoqTrackParam(); vTrack.trackType = .video
let dTrack = AoqTrackParam(); dTrack.trackType = .data
conn.publishTracks   = [aTrack, vTrack, dTrack]
conn.subscribeTracks = [aTrack, dTrack]
engine.connect(conn)

Android:

// Audio codec configuration
AoqAudioCodecConfig encCfg = new AoqAudioCodecConfig();
encCfg.codecType = AoqEncoderType.AoqEncoderTypeAudioPCM;
encCfg.sampleRate = 16000; encCfg.channel = 1;
engine.setAudioEncoderConfig(encCfg);
engine.setAudioDecoderConfig(encCfg);

// Disable media sending before connect; enable after receiving session.updated
engine.enableSendMediaStream(AoqTrackType.AoqTrackTypeAudio, false);
engine.enableSendMediaStream(AoqTrackType.AoqTrackTypeVideo, false);

// Establish connection
AoqConnectConfig conn = new AoqConnectConfig();
conn.token = token; conn.sid = sid; conn.certFingerprint = cert;
conn.relayEndpoints.addAll(endpoints); conn.workspaceIdHash = workspaceIdHash;

AoqTrackParam aTrack = new AoqTrackParam(); aTrack.trackType = AoqTrackType.AoqTrackTypeAudio;
AoqTrackParam vTrack = new AoqTrackParam(); vTrack.trackType = AoqTrackType.AoqTrackTypeVideo;
AoqTrackParam dTrack = new AoqTrackParam(); dTrack.trackType = AoqTrackType.AoqTrackTypeData;
conn.publishTracks.add(aTrack);
conn.publishTracks.add(vTrack);
conn.publishTracks.add(dTrack);
conn.subscribeTracks.add(aTrack);
conn.subscribeTracks.add(dTrack);
engine.connect(conn);

HarmonyOS:

// Audio codec configuration
const encCfg: AoqAudioCodecConfig = {
  codecType: AoqEncoderType.AoqEncoderTypeAudioPCM,
  sampleRate: 16000, channel: 1
};
engine.setAudioEncoderConfig(encCfg);
engine.setAudioDecoderConfig(encCfg);

// Disable media sending before connect; enable after receiving session.updated
engine.enableSendMediaStream(AoqTrackType.AoqTrackTypeAudio, false);
engine.enableSendMediaStream(AoqTrackType.AoqTrackTypeVideo, false);

// Establish connection
const conn: AoqConnectConfig = {
  token, sid, certFingerprint: cert,
  relayEndpoints: endpoints,
  workspaceIdHash,
  publishTracks: [
    { trackType: AoqTrackType.AoqTrackTypeAudio },
    { trackType: AoqTrackType.AoqTrackTypeVideo },
    { trackType: AoqTrackType.AoqTrackTypeData }
  ],
  subscribeTracks: [
    { trackType: AoqTrackType.AoqTrackTypeAudio },
    { trackType: AoqTrackType.AoqTrackTypeData }
  ]
};
engine.connect(conn);

Important: AOQ SDK secara default mulai mengirim data media setelah terhubung. Contoh ini menunjukkan cara menonaktifkan pengiriman media saat terhubung ke model.

Configure AI session

Dalam callback onConnectionStatusChange(Connected), kirim pesan session.update melalui sendDataMsg. Pesan ini berisi parameter sesi seperti modalities, voice, instructions, dan turn_detection, yang melengkapi handshake sesi. Untuk detail event WebSocket, lihat Client events.

iOS:

func onConnectionStatusChange(_ status: AoqConnectionStatus) {
    if status == .connected { sendSessionUpdate() }
}

private func sendSessionUpdate() {
    let json = """
    {
      // Event ID, generated by the client
      "event_id": "event_ToPZqeobitzUJnt3QqtWg",
      // Event type, always session.update
      "type": "session.update",
      // Session configuration
      "session": {
          // Output modalities: ["text"] (text only) or ["text","audio"] (text and audio)
          "modalities": [
              "text",
              "audio"
          ],
          // Output audio voice
          "voice": "Ethan",
          // Input audio format; currently only "pcm" is supported. Input audio is 16 kHz PCM.
          "input_audio_format": "pcm",
          // Output audio format; currently only "pcm" is supported. Output audio is 24 kHz PCM.
          "output_audio_format": "pcm",
          // System message for setting the model's objective or role.
          "instructions": "You are an AI customer service agent for a five-star hotel. Accurately and helpfully answer customer inquiries about room types, facilities, pricing, and reservation policies. Always respond professionally and helpfully. Do not provide unverified information or information outside the scope of hotel services.",
          // Whether to enable voice activity detection. If enabled, pass a config object and the server will automatically detect voice start and end.
          // Set to null to let the client control when to trigger a model response.
          "turn_detection": {
              // VAD type: server_vad or semantic_vad. Recommended: semantic_vad for qwen3.5-omni-realtime.
              "type": "semantic_vad",
              // VAD detection threshold. Increase in noisy environments; decrease in quiet ones.
              "threshold": 0.5,
              // Duration of silence after which a model response is triggered
              "silence_duration_ms": 800
          }
      }
    }
    """
    let msg = AoqDataMsg()
    msg.data = json.data(using: .utf8)!
    engine.send(msg)
}

Android:

@Override
public void onConnectionStatusChange(AoqConnectionStatus status) {
    if (status == AoqConnectionStatus.AoqConnectionStatusConnected) {
        sendSessionUpdate();
    }
}

private void sendSessionUpdate() {
    String sessionUpdateJson = /* Same session.update JSON as in the Swift example above */;
    AoqDataMsg msg = new AoqDataMsg();
    msg.data = sessionUpdateJson.getBytes(StandardCharsets.UTF_8);
    engine.sendDataMsg(msg);
}

HarmonyOS:

onConnectionStatusChange(status: AoqConnectionStatus): void {
  if (status === AoqConnectionStatus.AoqConnectionStatusConnected) {
    this.sendSessionUpdate();
  }
}

private sendSessionUpdate(): void {
  const sessionUpdateJson = /* Same session.update JSON as in the Swift example above */;
  const msg: AoqDataMsg = { data: new TextEncoder().encode(sessionUpdateJson).buffer };
  this.engine.sendDataMsg(msg);
}

Enable media sending after receiving session.updated

Dalam callback onDataMsg, uraikan pesan masuk. Saat menerima session.updated dari model, panggil enableSendMediaStream(trackType, true) untuk setiap jenis track yang sebelumnya dinonaktifkan. Untuk detail event WebSocket, lihat Server events.

iOS:

func onDataMsg(_ msg: AoqDataMsg) {
    guard let obj = try? JSONSerialization.jsonObject(with: msg.data) as? [String: Any],
          let type = obj["type"] as? String else { return }
    if type == "session.updated" {
        engine.enableSendMediaStream(.audio, enable: true)
        engine.enableSendMediaStream(.video, enable: true)
    }
}

Android:

@Override
public void onDataMsg(AoqDataMsg msg) {
    if (msg == null || msg.data == null) return;
    try {
        JSONObject obj = new JSONObject(new String(msg.data, StandardCharsets.UTF_8));
        if ("session.updated".equals(obj.optString("type"))) {
            engine.enableSendMediaStream(AoqTrackType.AoqTrackTypeAudio, true);
            engine.enableSendMediaStream(AoqTrackType.AoqTrackTypeVideo, true);
        }
    } catch (JSONException ignored) {}
}

HarmonyOS:

onDataMsg(msg: AoqDataMsg): void {
  if (!msg?.data) return;
  try {
    const text = new TextDecoder('utf-8').decode(new Uint8Array(msg.data));
    const obj = JSON.parse(text) as { type?: string };
    if (obj.type === 'session.updated') {
      this.engine.enableSendMediaStream(AoqTrackType.AoqTrackTypeAudio, true);
      this.engine.enableSendMediaStream(AoqTrackType.AoqTrackTypeVideo, true);
    }
  } catch (_) { /* Not JSON, ignore */ }
}

Important

Mulai mengirim aliran media hanya setelah menerima session.updated. AI mungkin belum siap menerima data sebelum event ini.
Track audio dan video yang ditambahkan selama koneksi (channel media AOQ) secara otomatis mengirimkan data ke server.
1. Audio: dikirim langsung melalui track audio — tidak perlu mengirim event input_audio_buffer.append.
2. Video: frame dikirim melalui track video — tidak perlu mengirim event input_image_buffer.append.

Disconnect and destroy engine

engine.disconnect()
AoqClientEngine.destroy()

Common scenarios

Barge-in

SDK terintegrasi secara mendalam dengan Alibaba Cloud Model Studio. Pesan barge-in dari model akan menginterupsi giliran sebelumnya ketika giliran baru dimulai.
SDK menyediakan antarmuka interruptAudioPlayer untuk interupsi pemutaran lokal. Panggil API ini ketika pengguna ingin menghentikan pemutaran.

// iOS
engine.interruptAudioPlayer(.audio, fadeMs: 100)

Mute / unmute

Setelah dimatikan suaranya, SDK tetap melakukan tangkapan audio tetapi hanya mengirim frame diam. session tidak terganggu.

engine.muteAudioCapture(true);   // Mute microphone (capture continues, but only silent frames are sent)
engine.muteAudioCapture(false);  // Unmute

Switch between front and rear cameras

// Pass the desired camera direction enum
engine.switchCamera(AoqCameraDirection.AoqCameraDirectionFront);
engine.switchCamera(AoqCameraDirection.AoqCameraDirectionBack);

Call subtitles and ASR results

Server mendorong hasil ASR dan respons teks AI melalui pesan data downstream. Di aplikasi Anda, arahkan berdasarkan bidang type dalam callback onDataMsg. Untuk detail event WebSocket, lihat Server events.

Important notes

Singleton semantics: createEngine bersifat singleton — pemanggilan berulang mengembalikan instans yang sama. Engine hanya dapat dibuat ulang setelah destroy dipanggil. Untuk penggunaan multi-halaman, kelola siklus hidup engine di tingkat Application atau Ability.
Local preview view types:
- Android: SurfaceView atau TextureView. Jenis lain tidak didukung.
- iOS: Subkelas UIView apa pun.
- HarmonyOS: Lihat dokumentasi SDK.
Audio route changes: Event seperti pemasangan/pencabutan headphone atau koneksi Bluetooth memicu onAudioDeviceRouteChanged. Biasanya tidak diperlukan penanganan di aplikasi Anda. Jika UI Anda menyertakan toggle speaker/earpiece, sinkronkan status berdasarkan callback ini.
Background audio continuation: Agar audio tetap berlanjut saat panggilan berpindah ke latar belakang, Info.plist harus memiliki UIBackgroundModes = audio diaktifkan dan AVAudioSession harus diaktifkan dengan benar di latar depan. SDK menangani sebagian besar kasus; gunakan setAudioSessionRestriction: untuk kontrol detail halus.

iOS demo source code

Untuk mempelajari cara mengimplementasikan AOQ di iOS, unduh kode sumber contoh: aoqdemo.zip.

API lengkap AOQ Client SDK: SDK overview
Event klien model qwen3.5-omni-plus-realtime: Client events
Event server model qwen3.5-omni-plus-realtime: Server events

Get the SDK

Import the SDK

Android

iOS (framework)

HarmonyOS (har)

Try the demo

Get a token from the AppServer

Implement AI audio/video calls

Create engine and set callbacks

Start audio/video capture and playback

Get connection credentials

Configure codecs and establish connection

Configure AI session

Enable media sending after receiving session.updated

Disconnect and destroy engine

Common scenarios

Barge-in

Mute / unmute

Switch between front and rear cameras

Call subtitles and ASR results

Important notes

iOS demo source code

Related documents