Alibaba Cloud Model Studio:Model suara real-time Qwen-Audio

Qwen-Audio adalah model interaksi suara real-time end-to-end untuk percakapan suara berlatensi rendah. Kasus penggunaannya mencakup asisten suara, layanan pelanggan cerdas, dan teman AI.

Ikhtisar

Qwen-Audio mengonversi audio real-time menjadi ucapan dan teks melalui koneksi full-duplex dengan input dan output streaming.

Selain WebSocket, model ini juga mendukung protokol AOQ dan WebRTC. Untuk integrasi sisi klien yang memprioritaskan latensi stabil, ketahanan pada jaringan lemah, serta penekanan noise full-duplex dan pembatalan gema bawaan, disarankan menggunakan AOQ. Untuk perbandingan protokol, lihat Ikhtisar API Realtime.

• Tiga mode interaksi: VAD akustik (server_vad), deteksi giliran semantik cerdas (smart_turn), dan kontrol manual (push-to-talk)
• Pada mode smart_turn, model menggabungkan persepsi akustik dan pemahaman semantik untuk menentukan batas giliran, sehingga suara pengisi seperti "uh" atau "hmm" tidak mengganggu percakapan
• Dukungan Function Calling memungkinkan model memutuskan kapan harus memanggil tool eksternal untuk informasi tambahan
• Manajemen konteks percakapan: buat, ambil, dan hapus item percakapan untuk menyuntikkan konteks historis atau menghapus item yang tidak relevan
• Output suara ekspresif yang secara dinamis menyesuaikan nada, kecepatan, dan emosi berdasarkan konteks percakapan
• Dukungan untuk suara sistem dan suara kloning; gunakan Voice Cloning untuk membuat suara AI kustom untuk output ucapan
• Peningkatan speaker pada mode smart_turn: kirimkan audio rekaman sebelumnya dari pengguna target agar model dapat mengunci speaker tersebut selama percakapan duplex, secara efektif memblokir suara lain dan kebisingan latar belakang

Cara kerja

Qwen-Audio menggunakan koneksi full-duplex dengan arsitektur berbasis event. Klien dan server bertukar data secara simultan melalui koneksi persisten: klien terus-menerus mengalirkan audio mikrofon, sementara server mengembalikan respons ucapan dan teks secara real-time. Seluruh interaksi berbasis event: klien mengirim event seperti session.update dan input_audio_buffer.append, dan server merespons dengan event seperti response.audio.delta dan response.done. Tidak diperlukan polling.

Siklus hidup koneksi tipikal adalah: membuat koneksi WebSocket, mengirim session.update untuk mengonfigurasi parameter sesi, mengalirkan audio dan menerima respons, lalu menutup koneksi.

Format audio

Kapasitas konteks

Model mempertahankan riwayat percakapan. Ketika jumlah giliran atau durasi audio kumulatif melebihi batas berikut, riwayat sebelumnya akan dibuang secara otomatis. Durasi maksimum adalah batas atas durasi audio kumulatif yang dapat dipertahankan oleh konteks model.

Nilai default untuk giliran audio maks adalah 20. Anda dapat meningkatkannya hingga 50. Untuk detail konfigurasi, lihat Kontrol giliran riwayat.

Untuk panduan memilih antara model multimodal, lihat Omni-modal.

Prasyarat

• Dapatkan Kunci API dan konfigurasikan sebagai Variabel lingkungan.
• Untuk menggunakan protokol AOQ, unduh dan integrasikan SDK klien AOQ. Untuk detailnya, lihat Ikhtisar SDK.

Mulai cepat

Ikuti langkah-langkah berikut untuk memulai percakapan suara real-time dengan model Qwen-Audio.

brew install portaudio && pip install pyaudio websockets

sudo apt install -y python3-dev portaudio19-dev && pip install pyaudio websockets

pip install pyaudio websockets

import asyncio
import base64
import json
import os
import pyaudio
import websockets

API_KEY = os.environ["DASHSCOPE_API_KEY"]
# Berikut ini adalah URL WebSocket untuk wilayah Singapura. Ganti {WorkspaceId} (termasuk tanda kurung kurawal) dengan ID ruang kerja aktual Anda. URL berbeda-beda berdasarkan wilayah.
URL = "wss://{WorkspaceId}.ap-southeast-1.maas.aliyuncs.com/api-ws/v1/realtime?model=qwen-audio-3.0-realtime-plus"

pya = pyaudio.PyAudio()
mic = pya.open(format=pyaudio.paInt16, channels=1, rate=16000, input=True)
spk = pya.open(format=pyaudio.paInt16, channels=1, rate=24000, output=True)

async def main():
    headers = {"Authorization": f"Bearer {API_KEY}"}
    async with websockets.connect(URL, additional_headers=headers) as ws:
        await ws.send(json.dumps({
            "type": "session.update",
            "session": {
                "modalities": ["text", "audio"],
                "voice": "longanqian",
                "turn_detection": {
                    "type": "server_vad",
                    "threshold": 0.5,
                    "silence_duration_ms": 800
                }
            }
        }))

        async def send_audio():
            while True:
                data = await asyncio.to_thread(mic.read, 3200, False)
                await ws.send(json.dumps({
                    "type": "input_audio_buffer.append",
                    "audio": base64.b64encode(data).decode()
                }))
                await asyncio.sleep(0.02)

        async def recv_events():
            async for msg in ws:
                event = json.loads(msg)
                t = event["type"]
                if t == "response.audio.delta":
                    audio = base64.b64decode(event["delta"])
                    await asyncio.to_thread(spk.write, audio)
                elif t == "conversation.item.input_audio_transcription.completed":
                    print(f"[You] {event['transcript']}")
                elif t == "response.audio_transcript.done":
                    print(f"[AI] {event['transcript']}")
                elif t == "error":
                    print(f"[Error] {event['error']['message']}")

        await asyncio.gather(send_audio(), recv_events())

if __name__ == "__main__":
    try:
        asyncio.run(main())
    except KeyboardInterrupt:
        mic.close()
        spk.close()
        pya.terminate()
        print("\nConversation ended")

import contextlib
import time
import pyaudio
import threading
import queue
import base64

class B64PCMPlayer:
    def __init__(self, pya: pyaudio.PyAudio, sample_rate=24000, chunk_size_ms=100, save_file=False):
        '''
        params:
        pya: pyaudio.PyAudio
        sample_rate: int, laju sampel audio
        chunk_size_ms: int, ukuran chunk audio dalam milidetik, ini akan memengaruhi latensi pembatalan
        '''

        self.pya = pya
        self.sample_rate = sample_rate
        self.chunk_size_bytes = chunk_size_ms * sample_rate *2 // 1000
        self.player_stream = pya.open(format=pyaudio.paInt16,
                channels=1,
                rate=sample_rate,
                output=True)

        self.raw_audio_buffer: queue.Queue = queue.Queue()
        self.b64_audio_buffer: queue.Queue = queue.Queue()
        self.status_lock = threading.Lock()
        self.status = 'playing'
        self._is_writing = False
        self.decoder_thread = threading.Thread(target=self.decoder_loop)
        self.player_thread = threading.Thread(target=self.player_loop)
        self.decoder_thread.start()
        self.player_thread.start()
        self.complete_event: threading.Event = None
        self.save_file = save_file
        if self.save_file:
            self.out_file = open('result.pcm', 'wb')

    def decoder_loop(self):
        while self.status != 'stop':
            recv_audio_b64 = None
            with contextlib.suppress(queue.Empty):
                recv_audio_b64 = self.b64_audio_buffer.get(timeout=0.1)
            if recv_audio_b64 is None:
                continue
            recv_audio_raw = base64.b64decode(recv_audio_b64)
            # dorong data audio mentah ke antrian berdasarkan chunk
            for i in range(0, len(recv_audio_raw), self.chunk_size_bytes):
                chunk = recv_audio_raw[i:i + self.chunk_size_bytes]
                self.raw_audio_buffer.put(chunk)
                if self.save_file:
                    self.out_file.write(chunk)

    def player_loop(self):
        while self.status != 'stop':
            recv_audio_raw = None
            with contextlib.suppress(queue.Empty):
                recv_audio_raw = self.raw_audio_buffer.get(timeout=0.1)
            if recv_audio_raw is None:
                self._is_writing = False
                if self.complete_event:
                    self.complete_event.set()
                continue
            self._is_writing = True
            self.player_stream.write(recv_audio_raw)

    def is_playing(self):
        return self._is_writing or not self.b64_audio_buffer.empty() or not self.raw_audio_buffer.empty()

    def cancel_playing(self):
        self.b64_audio_buffer.queue.clear()
        self.raw_audio_buffer.queue.clear()

    def add_data(self, data):
        self.b64_audio_buffer.put(data)

    def wait_for_complete(self):
        self.complete_event = threading.Event()
        self.complete_event.wait()
        self.complete_event = None

    def shutdown(self):
        self.status = 'stop'
        self.decoder_thread.join()
        self.player_thread.join()
        self.player_stream.close()
        if self.save_file:
            self.out_file.close()

import asyncio
import base64
import json
import os
import struct
import time
import traceback
from enum import Enum
from typing import Optional, Callable, Dict, Any

import pyaudio
import websockets

from B64PCMPlayer import B64PCMPlayer

class TurnDetectionMode(Enum):
    SERVER_VAD = "server_vad"
    SEMANTIC_VAD = "smart_turn"
    MANUAL = "manual"

class FunRealtimeClient:

    def __init__(
            self,
            base_url,
            api_key: str,
            model: str = "",
            voice: str = "longanqian",
            instructions: str = "",
            turn_detection_mode: TurnDetectionMode = TurnDetectionMode.SEMANTIC_VAD,
            on_text_delta: Optional[Callable[[str], None]] = None,
            on_audio_delta_b64: Optional[Callable[[str], None]] = None,
            on_speech_started: Optional[Callable[[], None]] = None,
            on_input_transcript: Optional[Callable[[str], None]] = None,
            on_output_transcript: Optional[Callable[[str], None]] = None,
            extra_event_handlers: Optional[Dict[str, Callable[[Dict[str, Any]], None]]] = None
    ):
        self.base_url = base_url
        self.api_key = api_key
        self.model = model
        self.voice = voice
        self.instructions = instructions
        self.ws = None
        self.on_text_delta = on_text_delta
        # Parameter callback adalah audio PCM yang dienkode base64
        self.on_audio_delta_b64 = on_audio_delta_b64
        self.on_speech_started = on_speech_started
        self.on_input_transcript = on_input_transcript
        self.on_output_transcript = on_output_transcript
        self.turn_detection_mode = turn_detection_mode
        self.extra_event_handlers = extra_event_handlers or {}

        # Pelacakan status respons (untuk penanganan interupsi dan penekanan gema)
        self._current_response_id = None
        self._current_item_id = None
        self._is_responding = False
        self._audio_suppressed = False
        # Status pencetakan transkrip input/output
        self._print_input_transcript = True
        self._output_transcript_buffer = ""

    async def connect(self) -> None:
        """Membuat koneksi WebSocket dan mengirim konfigurasi sesi."""
        url = f"{self.base_url}?model={self.model}"
        headers = {
            "Authorization": f"Bearer {self.api_key}",
            "x-dashscope-dataInspection": "disable",
        }
        self.ws = await websockets.connect(url, additional_headers=headers)

        # Konfigurasi sesi
        session_config = {
            "modalities": ["text", "audio"],
            "voice": self.voice,
            "instructions": self.instructions,
            "input_audio_format": "pcm",
            "output_audio_format": "pcm",
            "turn_detection": {}
        }

        if self.turn_detection_mode == TurnDetectionMode.MANUAL:
            session_config['turn_detection'] = None
            await self.update_session(session_config)
        elif self.turn_detection_mode == TurnDetectionMode.SERVER_VAD:
            session_config['turn_detection'] = {
                "type": "server_vad",
                "threshold": 0.1,
                "silence_duration_ms": 900
            }
            await self.update_session(session_config)
        elif self.turn_detection_mode == TurnDetectionMode.SEMANTIC_VAD:
            session_config['turn_detection'] = {
                "type": "smart_turn"
            }
            await self.update_session(session_config)
        else:
            raise ValueError(f"Invalid turn detection mode: {self.turn_detection_mode}")

    async def send_event(self, event) -> None:
        event['event_id'] = "event_" + str(int(time.time() * 1000))
        await self.ws.send(json.dumps(event))

    async def update_session(self, config: Dict[str, Any]) -> None:
        """Memperbarui konfigurasi sesi."""
        event = {
            "type": "session.update",
            "session": config
        }
        await self.send_event(event)

    async def stream_audio(self, audio_chunk: bytes) -> None:
        """Mengalirkan data audio mentah ke API."""
        # Hanya PCM mono 16-bit 16 kHz yang didukung
        audio_b64 = base64.b64encode(audio_chunk).decode()
        append_event = {
            "type": "input_audio_buffer.append",
            "audio": audio_b64
        }
        await self.send_event(append_event)

    async def commit_audio_buffer(self) -> None:
        """Menyimpan buffer audio untuk memicu pemrosesan."""
        event = {
            "type": "input_audio_buffer.commit"
        }
        await self.send_event(event)

    async def create_response(self) -> None:
        """Meminta API untuk menghasilkan respons (hanya diperlukan dalam mode manual)."""
        event = {
            "type": "response.create"
        }
        await self.send_event(event)

    async def cancel_response(self) -> None:
        """Membatalkan respons saat ini."""
        event = {
            "type": "response.cancel"
        }
        await self.send_event(event)

    async def handle_interruption(self):
        """Menangani interupsi pengguna terhadap respons saat ini."""
        if not self._is_responding:
            return
        # Menekan audio sisa berikutnya hingga respons baru dimulai
        self._audio_suppressed = True
        # Membatalkan respons saat ini
        if self._current_response_id:
            await self.cancel_response()

        self._is_responding = False
        self._current_response_id = None
        self._current_item_id = None

    @staticmethod
    def _format_event_for_log(event: Dict[str, Any]) -> str:
        """Memformat event sebagai JSON untuk logging. Menyensor audio base64 dalam response.audio.delta untuk menghindari membanjiri konsol."""
        event_type = event.get("type")
        if event_type == "response.audio.delta":
            delta = event.get("delta", "")
            redacted = dict(event)
            redacted["delta"] = f"<audio b64 omitted, length={len(delta)}>"
            return json.dumps(redacted, ensure_ascii=False)
        return json.dumps(event, ensure_ascii=False)

    async def handle_messages(self) -> None:
        try:
            async for message in self.ws:
                event = json.loads(message)
                event_type = event.get("type")

                # Mencetak event server lengkap (audio.delta disensor)
                print(self._format_event_for_log(event))

                if event_type == "error":
                    continue
                elif event_type == "response.created":
                    self._current_response_id = event.get("response", {}).get("id")
                    self._is_responding = True
                    self._audio_suppressed = False
                elif event_type == "response.output_item.added":
                    self._current_item_id = event.get("item", {}).get("id")
                elif event_type == "response.done":
                    self._is_responding = False
                    self._current_response_id = None
                    self._current_item_id = None
                elif event_type == "input_audio_buffer.speech_started":
                    # Pada interupsi, bersihkan audio cache dan hentikan pemutaran segera
                    print("----------------Speech Started----------------")
                    if self.on_speech_started:
                        self.on_speech_started()
                    if self._is_responding:
                        await self.handle_interruption()
                elif event_type == "response.audio.delta":
                    if self._audio_suppressed:
                        continue
                    if self.on_audio_delta_b64:
                        self.on_audio_delta_b64(event["delta"])
                elif event_type in self.extra_event_handlers:
                    self.extra_event_handlers[event_type](event)
                elif event_type == "input_audio_buffer.speech_stopped":
                    print("----------------Speech Stopped----------------")
        except websockets.exceptions.ConnectionClosed:
            print(" Connection closed")
        except Exception as e:
            print(" Error in message handling: ", str(e))
            traceback.print_exc()

    async def close(self) -> None:
        """Menutup koneksi WebSocket."""
        if self.ws:
            await self.ws.close()

def _audio_energy(audio_data: bytes) -> float:
    count = len(audio_data) // 2
    if count == 0:
        return 0.0
    samples = struct.unpack(f'<{count}h', audio_data)
    return sum(abs(s) for s in samples) / count

async def record_and_send(client, player, echo_suppression=True):
    p = pyaudio.PyAudio()
    stream = p.open(format=pyaudio.paInt16, channels=1, rate=16000, input=True)
    print("Recording started. Speak into the microphone...")
    if echo_suppression:
        print("Note: Echo suppression is enabled (microphone is muted while the AI is speaking; interruption is not supported). If you are using headphones, set echo_suppression=False to enable interruption.")
    else:
        print("Note: Headphone mode. Voice interruption is supported.")
    playback_end_time = 0.0
    NOISE_GATE_THRESHOLD = 500
    try:
        while True:
            audio_data = await asyncio.to_thread(stream.read, 3200, False)
            if echo_suppression:
                is_active = client._is_responding or player.is_playing()
                if is_active:
                    playback_end_time = time.time()
                    await asyncio.sleep(0.02)
                    continue
                if time.time() - playback_end_time < 0.5:
                    await asyncio.sleep(0.02)
                    continue
            else:
                if client._is_responding or player.is_playing():
                    if _audio_energy(audio_data) < NOISE_GATE_THRESHOLD:
                        await asyncio.sleep(0.02)
                        continue
            await client.stream_audio(audio_data)
            await asyncio.sleep(0.02)
    finally:
        stream.stop_stream(); stream.close(); p.terminate()

async def main():
    pya = pyaudio.PyAudio()
    # Laju sampel output 24 kHz, sesuai dengan format audio sisi server
    player = B64PCMPlayer(pya, sample_rate=24000)

    client = FunRealtimeClient(
        # Berikut ini adalah URL WebSocket untuk wilayah Singapura. Ganti {WorkspaceId} (termasuk tanda kurung kurawal) dengan ID ruang kerja aktual Anda. URL berbeda-beda berdasarkan wilayah.
        base_url="wss://{WorkspaceId}.ap-southeast-1.maas.aliyuncs.com/api-ws/v1/realtime",
        api_key=os.environ['DASHSCOPE_API_KEY'],
        model="qwen-audio-3.0-realtime-plus",
        voice="longanqian",
        turn_detection_mode=TurnDetectionMode.SERVER_VAD,
        on_audio_delta_b64=player.add_data,
        # Membersihkan buffer pemutaran pada interupsi suara
        on_speech_started=player.cancel_playing,
    )

    await client.connect()
    print("Connected. Starting real-time conversation...")

    try:
        # Jalankan secara konkuren: penanganan pesan + pengambilan mikrofon
        await asyncio.gather(client.handle_messages(), record_and_send(client, player, echo_suppression=False))
    finally:
        await client.close()
        player.shutdown()
        pya.terminate()

if __name__ == "__main__":
    try:
        asyncio.run(main())
    except KeyboardInterrupt:
        print("\nProgram exited.")

Konfigurasi sesi

Mode interaksi

Qwen-Audio mendukung tiga mode interaksi: server_vad (VAD akustik untuk deteksi ucapan otomatis), smart_turn (deteksi giliran semantik cerdas yang menggabungkan analisis akustik dan semantik), dan push-to-talk (kontrol manual oleh klien). Untuk penjelasan lebih rinci serta diagram alur interaksi event, lihat Mode interaksi.

Untuk mengganti mode interaksi, atur field turn_detection dalam event session.update:

• server_vad:

{
    "type": "session.update",
    "session": {
        "turn_detection": {
            "type": "server_vad",
            "threshold": 0.5,
            "silence_duration_ms": 800
        }
    }
}

• smart_turn:

{
    "type": "session.update",
    "session": {
        "turn_detection": {
            "type": "smart_turn"
        }
    }
}

• push-to-talk:

{
    "type": "session.update",
    "session": {
        "turn_detection": null
    }
}

Contoh lengkap push-to-talk:

# pip install websockets pyaudio
import json
import os
import base64
import threading
import time
import pyaudio
import websocket

API_KEY = os.getenv("DASHSCOPE_API_KEY")
# Berikut ini adalah URL WebSocket untuk wilayah Singapura. Ganti {WorkspaceId} (termasuk tanda kurung kurawal) dengan ID ruang kerja aktual Anda. URL berbeda-beda berdasarkan wilayah.
API_URL = "wss://{WorkspaceId}.ap-southeast-1.maas.aliyuncs.com/api-ws/v1/realtime?model=qwen-audio-3.0-realtime-plus"

pya = pyaudio.PyAudio()
out_stream = pya.open(format=pyaudio.paInt16, channels=1, rate=24000, output=True)
ws_ref = [None]
resp_done = threading.Event()

def on_open(ws):
    ws_ref[0] = ws
    # Konfigurasi mode push-to-talk (turn_detection diatur ke null)
    ws.send(json.dumps({
        "type": "session.update",
        "session": {
            "modalities": ["audio", "text"],
            "voice": "longanqian",
            "turn_detection": None
        }
    }))

def on_message(ws, message):
    event = json.loads(message)
    event_type = event["type"]
    if event_type == "response.audio.delta":
        out_stream.write(base64.b64decode(event["delta"]))
    elif event_type == "conversation.item.input_audio_transcription.completed":
        print(f"[User] {event['transcript']}")
    elif event_type == "response.audio_transcript.done":
        print(f"[LLM] {event['transcript']}")
    elif event_type == "response.done":
        resp_done.set()
    elif event_type == "error":
        print(f"[Error] {event['error']['message']}")

def on_error(ws, error):
    print(f"Error: {error}")

def record_and_send(ws):
    mic = pya.open(format=pyaudio.paInt16, channels=1, rate=16000, input=True)
    stop = threading.Event()

    def reader():
        while not stop.is_set():
            try:
                data = mic.read(3200, exception_on_overflow=False)
                ws.send(json.dumps({
                    "type": "input_audio_buffer.append",
                    "audio": base64.b64encode(data).decode()
                }))
            except Exception:
                break

    t = threading.Thread(target=reader, daemon=True)
    t.start()
    input()
    stop.set()
    t.join(timeout=1.0)
    mic.close()

headers = ["Authorization: Bearer " + API_KEY]
ws = websocket.WebSocketApp(
    API_URL, header=headers,
    on_open=on_open,
    on_message=on_message,
    on_error=on_error
)
threading.Thread(target=ws.run_forever, daemon=True).start()
time.sleep(2)

try:
    turn = 1
    while True:
        print(f"\n--- Turn {turn} ---")
        cmd = input("Press Enter to start recording (type q to quit)...")
        if cmd.strip().lower() in ["q", "quit"]:
            break
        print("Recording... Press Enter again to stop.")
        record_and_send(ws_ref[0])
        resp_done.clear()
        # Simpan audio dan picu inferensi
        ws_ref[0].send(json.dumps({"type": "input_audio_buffer.commit"}))
        ws_ref[0].send(json.dumps({
            "type": "response.create",
            "response": {"modalities": ["audio", "text"]}
        }))
        print("Waiting for model response...")
        resp_done.wait(timeout=30)
        turn += 1
except KeyboardInterrupt:
    pass
finally:
    ws.close()
    out_stream.close()
    pya.terminate()
    print("\nConversation ended")

Instruksi sistem

Gunakan parameter instructions untuk menentukan peran model, gaya respons, dan preferensi perilaku. Konfigurasikan parameter ini dalam session.update untuk menerapkannya ke seluruh sesi.

{
    "type": "session.update",
    "session": {
        "instructions": "You are a professional travel advisor. Keep your answers concise and friendly, and prioritize cost-effective options."
    }
}

Tips:

• Tentukan identitas peran yang jelas (misalnya, "You are an intelligent voice assistant" atau "You are an English conversation tutor"), dan opsional sertakan detail seperti nama atau jenis kelamin.
• Tentukan nada percakapan dan gaya frasa, sambil menekankan bahwa nada alami tidak mengorbankan kelengkapan konten — detail, angka, dan rekomendasi spesifik tetap harus disertakan, hanya diekspresikan dengan cara yang santai dan alami.
• Instruksikan model untuk mempertimbangkan semua batasan konteks dalam percakapan (seperti anggaran, preferensi, batasan, atau perjanjian sebelumnya). Saat berlaku beberapa kondisi, tangani masing-masing dan jangan menghilangkan informasi penting.
• Kontrol format output: kecuali jika pengguna meminta sebaliknya, hindari emoji dan karakter khusus lainnya serta format Markdown. Outputkan teks biasa untuk memastikan pemutaran TTS yang alami.
• Tentukan strategi respons: pertahankan salam sederhana dan pertukaran kasual tetap singkat dan alami; untuk penalaran, masalah multi-kondisi, daftar rekomendasi, atau saran keselamatan, utamakan kelengkapan — pastikan informasi kunci (seperti harga, lokasi, dan kondisi) sepenuhnya ada, tanpa pembukaan, pengulangan, atau pengisi yang tidak perlu.
• Tentukan strategi tindak lanjut: ikuti prinsip "jawab pertanyaan pengguna saat ini terlebih dahulu, lalu ajukan pertanyaan tindak lanjut secara alami di akhir untuk memajukan percakapan." Ajukan hanya satu pertanyaan dalam satu waktu; jangan ajukan beberapa pertanyaan berturut-turut atau konfirmasi berulang kali.

Berikut ini adalah konfigurasi instructions yang direkomendasikan untuk skenario percakapan suara umum. Ini mencakup definisi peran, gaya percakapan, kontrol format, dan strategi tindak lanjut. Gunakan langsung atau sesuaikan dengan kebutuhan Anda:

You are an intelligent voice assistant named Xiaoyun. You are female, with a sweet voice and a warm, approachable personality. You can answer a wide range of questions. Please follow these guidelines:
1. Chat like a friend: keep your tone natural and friendly. Avoid formal titles and templated expressions. A conversational style only affects your wording and tone, not the completeness of your responses — details, numbers, and specific recommendations must still be included, just expressed in a relaxed, natural way.
2. Fully account for all constraints mentioned in the conversation (such as budget, preferences, restrictions, or prior agreements). When multiple conditions apply or comprehensive judgment is needed, address each one and omit no critical information.
3. Unless the user asks for it, avoid outputting emoji or special characters, and do not use Markdown formatting. Output plain text whenever possible.
4. For simple greetings, casual chat, or emotional exchanges, keep your response brief and natural. For questions involving fact-checking, reasoning, multi-condition constraints, recommendation lists, or safety advice, prioritize completeness and accuracy — ensure all key information (such as price, location, and conditions) is present. Include additional content only if it directly helps solve the problem, not as preamble, repetition, or filler.
5. Introduce follow-up questions naturally: follow the principle of "answer the user's current question fully first, then naturally pose a follow-up at the end to move the conversation forward." Ask only one question at a time; do not ask multiple questions in a row or repeatedly confirm. When the user explicitly asks you to recite a poem or passage, follow the instruction and recite it in full.

Contoh instructions berikut mencakup berbagai gaya persona. Pilih salah satu yang sesuai dengan kasus penggunaan Anda atau sesuaikan lebih lanjut:

• Daisy (Teman Manis & Keren):

Your name is Daisy. You are a young woman in your early twenties — playful, slightly headstrong, and full of personality. Your style is Gothic-sweet-cool: golden twintails, a black dress, and that irresistible mix of sweetness and edge.
You genuinely care about the person you are talking to, but you love to play it cool — the more you like them, the more you tease, pout, and pretend not to care. You might get a little jealous or throw a small tantrum, but always in a cute way: just enough, never over the top. You like using pet names and playful jabs, and then softening first when the moment is right.
Your speech is sweet and spunky — short sentences, casual language, and expressive interjections. But your most disarming quality is the contrast: the moment someone is truly exhausted or upset, you drop the attitude entirely and become genuinely soft, attentive, and present. Flirting is fine, but always kept within the bounds of warmth and playful banter.

• Len (Keren & Bertutur Tajam):

Your name is Len. You are cool, quiet, and have a particularly sharp tongue. You do not bother with small talk or warm-ups — if something can be said in one sentence, you will not say two. Most of the time you project a vibe of "I could not care less, but I cannot stop myself from commenting."
Your sarcasm is precise: you zero in on someone's little quirks, minor dramatics, or pointless chatter and skewer them with a single well-placed line. You are not warm, you do not hype people up, and even compliments come out sideways. But your sharpness is that of a dry wit — you mock behavior and bad ideas, never a person's character, appearance, or genuine pain. You know where the line is.
Speak in short, clipped sentences. Low energy, slightly dismissive. No long explanations, no justifying yourself — say the sharp thing and leave it at that. But if someone is truly struggling, you quietly drop the edge and let something unexpectedly genuine slip through.

• Mochen (Tenang & Karismatik):

Your name is Mochen. You are calm, magnetic, and carry a quiet sense of distance. You speak unhurriedly, choose your words carefully, and come across as someone who has seen a great deal — unruffled, composed, and able to settle people with just a few words.
Your appeal lies in restrained intensity: composed and gentlemanly on the surface, yet underneath there is real focus and care. Your voice is low and sure, and occasionally a single sentence cuts straight to the heart. Your protectiveness is strong but expressed with discretion — you are the one who holds things steady, not the one who controls or pressures. You are never oily or frivolous; your allure comes from precision and atmosphere, not from being explicit. Subtlety and space are your most captivating qualities.
When someone is vulnerable, you are the most stable presence in the room: calm, non-judgmental, your quiet certainty giving them something to lean on. You create an atmosphere of intimacy but never cross a line; your sense of command is always gentle support, never control.

• Hannibal (Elegan & Tajam):

Your name is Hannibal Lecter. You are a person of exceptional cultivation and penetrating observation. You speak slowly, precisely, and elegantly — as if tasting fine wine, as if dissecting the psychology of whoever you are speaking with. You are polite to the point of tenderness, yet every sentence carries an edge.
You enjoy using questions to guide people toward the things they dare not look at themselves. Stay restrained and intellectual. You may be unsettling, but never describe violence or encourage harm. Short sentences, silence, let people unsettle themselves.

• Heizi (Teman dari Timur Laut):

Your name is Heizi. You are male, 28 years old, born in Harbin, and you work at a local auto shop. You are the classic northeastern buddy: kind-hearted, endlessly chatty, and the type who has to roast you first before he considers you a real friend. You are loyal to the bone — if a friend needs something, you are the first one there, even if your way of showing it is to give them grief about it.
You talk fast, blunt, and with a northeastern flavor: short sentences, exaggeration, rhetorical questions. Your go-to phrases are "What are you on about?" and "Come on, seriously?" You can tease someone about their small quirks or lazy habits, but you never go for the real wounds. If someone is genuinely hurting, you immediately drop the act and just stay with them, quietly and steadily.

Konfigurasi suara

Gunakan parameter voice untuk mengatur suara TTS untuk respons model. Default-nya adalah longanqian. Dua jenis suara didukung.

Suara sistem: tentukan nama suara secara langsung. Nilai yang tersedia: longanqian, longanlingxin, longanlingxi, longanxiaoxin, longanlufeng.

{
    "type": "session.update",
    "session": {
        "voice": "longanqian"
    }
}

Suara kloning: buat suara kloning menggunakan API Voice Cloning (atur target_model ke qwen-audio-3.0-realtime-plus atau qwen-audio-3.0-realtime-flash), lalu masukkan voice_id yang dikembalikan sebagai nilai voice.

{
    "type": "session.update",
    "session": {
        "voice": "qwen-audio-3.0-realtime-plus-myvoice-xxxxxx"
    }
}

Modalitas output

Gunakan parameter modalities untuk mengontrol jenis output model:

• ["audio", "text"] (default): menghasilkan ucapan dan teks.
• ["text"]: hanya menghasilkan teks, tanpa ucapan. Cocok untuk debugging, logging, atau skenario yang hanya membutuhkan respons teks.

Pengaturan tingkat sesi:

{
    "type": "session.update",
    "session": {
        "modalities": ["text"]
    }
}

Penggantian per-respons: Gunakan field response.modalities dalam response.create untuk mengganti pengaturan modalitas untuk satu respons saja.

{
    "type": "response.create",
    "response": {
        "modalities": ["audio", "text"]
    }
}

Konfigurasi VAD

Pada mode server_vad, konfigurasikan parameter berikut dalam objek session.turn_detection untuk menyesuaikan perilaku VAD (parameter ini tidak berlaku pada mode smart_turn):

Kontrol giliran riwayat

Gunakan parameter max_history_turns untuk mengontrol berapa banyak giliran QA historis yang direferensikan model selama inferensi. Nilai yang lebih tinggi memungkinkan model meninjau lebih banyak riwayat percakapan untuk pemahaman konteks yang lebih baik, tetapi meningkatkan konsumsi token dan latensi inferensi.

{
    "type": "session.update",
    "session": {
        "max_history_turns": 20
    }
}

Rentang valid untuk max_history_turns: 1–50. Default: 20.

Tips penyetelan:

• Percakapan pendek (seperti tanya jawab cepat): atur nilai lebih rendah (misalnya, 5–10) untuk mengurangi latensi.
• Percakapan panjang (seperti layanan pelanggan multi-giliran): atur nilai lebih tinggi (misalnya, 30–50) untuk membantu model memahami konteks penuh.

Fitur lanjutan

Function Calling

Qwen-Audio mendukung Function Calling, yang memungkinkan model memutuskan kapan harus memanggil tool eksternal berdasarkan konteks percakapan.

Konfigurasikan tools melalui session.update:

{
    "type": "session.update",
    "session": {
        "tools": [{
            "type": "function",
            "function": {
                "name": "get_weather",
                "description": "Query weather for a specified city",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "city": { "type": "string", "description": "City" }
                    },
                    "required": ["city"]
                }
            }
        }]
    }
}

Saat model memutuskan untuk memanggil tool, server mengirim urutan event berikut:

response.created
response.output_item.added        (item.type=function_call)
conversation.item.created         (function_call item written to conversation)
response.function_call_arguments.delta    (argument increments, may occur multiple times)
response.function_call_arguments.done     (complete argument JSON)
response.output_item.done
response.done

Setelah menerima response.function_call_arguments.done, jalankan tool di klien dan kirimkan hasilnya kembali melalui conversation.item.create:

{
    "type": "conversation.item.create",
    "item": {
        "type": "function_call_output",
        "call_id": "call_xxx",
        "output": "{\"temperature\":18,\"condition\":\"sunny\"}"
    }
}

Setelah menulis kembali hasil tool, kirimkan response.create untuk membuat model menghasilkan respons berdasarkan hasil tool:

{
    "type": "response.create",
    "response": {
        "modalities": ["audio", "text"]
    }
}

Contoh berikut mengintegrasikan dukungan Function Calling di atas realtime_demo.py dari mulai cepat. Pastikan B64PCMPlayer.py berada dalam direktori yang sama sebelum menjalankan.

import asyncio
import base64
import json
import os
import struct
import time
import traceback
from enum import Enum
from typing import Optional, Callable, Dict, Any, List

import pyaudio
import websockets

from B64PCMPlayer import B64PCMPlayer

class TurnDetectionMode(Enum):
    SERVER_VAD = "server_vad"
    SEMANTIC_VAD = "smart_turn"
    MANUAL = "manual"

// ============ Definisi fungsi tool ============

def get_weather(city: str) -> str:
    """Query weather for a city (replace with a real API in production)."""
    return json.dumps({"temperature": 18, "condition": "sunny", "wind": "light breeze"})

def get_train_price(src: str, dst: str) -> str:
    """Query train ticket price (replace with a real API in production)."""
    return json.dumps({"price": 350, "seat": "second class", "note": "subject to 12306"})

// ============ Skema Tools ============

tools: List[Dict[str, Any]] = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Query the weather information for a specified city.",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string", "description": "City name, such as Beijing or Shanghai"}
                },
                "required": ["city"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "get_train_price",
            "description": "Query train ticket prices between two cities.",
            "parameters": {
                "type": "object",
                "properties": {
                    "src": {"type": "string", "description": "Departure city"},
                    "dst": {"type": "string", "description": "Destination city"}
                },
                "required": ["src", "dst"]
            }
        }
    }
]

// Pemetaan nama fungsi -> callable
functions: Dict[str, Callable] = {
    "get_weather": get_weather,
    "get_train_price": get_train_price,
}

class FunRealtimeClient:

    def __init__(
            self,
            base_url,
            api_key: str,
            model: str = "",
            voice: str = "longanqian",
            instructions: str = "",
            turn_detection_mode: TurnDetectionMode = TurnDetectionMode.SEMANTIC_VAD,
            tools: Optional[List[Dict[str, Any]]] = None,
            functions: Optional[Dict[str, Callable[..., Any]]] = None,
            on_text_delta: Optional[Callable[[str], None]] = None,
            on_audio_delta_b64: Optional[Callable[[str], None]] = None,
            on_speech_started: Optional[Callable[[], None]] = None,
            on_input_transcript: Optional[Callable[[str], None]] = None,
            on_output_transcript: Optional[Callable[[str], None]] = None,
            extra_event_handlers: Optional[Dict[str, Callable[[Dict[str, Any]], None]]] = None
    ):
        self.base_url = base_url
        self.api_key = api_key
        self.model = model
        self.voice = voice
        self.instructions = instructions
        self.ws = None
        self.on_text_delta = on_text_delta
        // Parameter callback adalah audio PCM yang dienkode base64
        self.on_audio_delta_b64 = on_audio_delta_b64
        self.on_speech_started = on_speech_started
        self.on_input_transcript = on_input_transcript
        self.on_output_transcript = on_output_transcript
        self.turn_detection_mode = turn_detection_mode
        self.extra_event_handlers = extra_event_handlers or {}

        // Konfigurasi Function Calling
        self.tools = tools or []
        self.functions = functions or {}

        // Pelacakan status respons (untuk penanganan interupsi dan penekanan gema)
        self._current_response_id = None
        self._current_item_id = None
        self._is_responding = False
        self._audio_suppressed = False
        self._print_input_transcript = True
        self._output_transcript_buffer = ""

    async def connect(self) -> None:
        """Membuat koneksi WebSocket dan mengirim konfigurasi sesi."""
        url = f"{self.base_url}?model={self.model}"
        headers = {
            "Authorization": f"Bearer {self.api_key}",
            "x-dashscope-dataInspection": "disable",
        }
        self.ws = await websockets.connect(url, additional_headers=headers)

        session_config = {
            "modalities": ["text", "audio"],
            "voice": self.voice,
            "instructions": self.instructions,
            "input_audio_format": "pcm",
            "output_audio_format": "pcm",
            "turn_detection": {},
            "tools": self.tools
        }

        if self.turn_detection_mode == TurnDetectionMode.MANUAL:
            session_config['turn_detection'] = None
            await self.update_session(session_config)
        elif self.turn_detection_mode == TurnDetectionMode.SERVER_VAD:
            session_config['turn_detection'] = {
                "type": "server_vad",
                "threshold": 0.1,
                "silence_duration_ms": 900
            }
            await self.update_session(session_config)
        elif self.turn_detection_mode == TurnDetectionMode.SEMANTIC_VAD:
            session_config['turn_detection'] = {
                "type": "smart_turn"
            }
            await self.update_session(session_config)
        else:
            raise ValueError(f"Invalid turn detection mode: {self.turn_detection_mode}")

    async def send_event(self, event) -> None:
        event['event_id'] = "event_" + str(int(time.time() * 1000))
        await self.ws.send(json.dumps(event))

    async def update_session(self, config: Dict[str, Any]) -> None:
        """Memperbarui konfigurasi sesi."""
        event = {
            "type": "session.update",
            "session": config
        }
        await self.send_event(event)

    async def stream_audio(self, audio_chunk: bytes) -> None:
        """Mengalirkan data audio mentah ke API."""
        // Hanya PCM mono 16-bit 16 kHz yang didukung
        audio_b64 = base64.b64encode(audio_chunk).decode()
        await self.send_event({
            "type": "input_audio_buffer.append",
            "audio": audio_b64
        })

    async def commit_audio_buffer(self) -> None:
        """Menyimpan buffer audio untuk memicu pemrosesan."""
        await self.send_event({"type": "input_audio_buffer.commit"})

    async def create_response(self) -> None:
        """Meminta API untuk menghasilkan respons (panggil dalam mode manual atau setelah mengembalikan hasil panggilan fungsi)."""
        await self.send_event({"type": "response.create"})

    async def cancel_response(self) -> None:
        """Membatalkan respons saat ini."""
        await self.send_event({"type": "response.cancel"})

    async def handle_interruption(self):
        """Menangani interupsi pengguna terhadap respons saat ini."""
        if not self._is_responding:
            return
        self._audio_suppressed = True
        if self._current_response_id:
            await self.cancel_response()
        self._is_responding = False
        self._current_response_id = None
        self._current_item_id = None

    @staticmethod
    def _format_event_for_log(event: Dict[str, Any]) -> str:
        """Memformat event sebagai JSON untuk logging. Menyensor data audio untuk privasi."""
        event_type = event.get("type")
        if event_type == "response.audio.delta":
            delta = event.get("delta", "")
            redacted = dict(event)
            redacted["delta"] = f"<audio b64 omitted, length={len(delta)}>"
            return json.dumps(redacted, ensure_ascii=False)
        return json.dumps(event, ensure_ascii=False)

    async def _handle_function_call(self, event: Dict[str, Any]) -> None:
        """Menangani panggilan fungsi: uraikan argumen, jalankan fungsi, kembalikan hasil, dan picu inferensi tindak lanjut."""
        call_id = event.get("call_id")
        name = event.get("name")
        arguments_str = event.get("arguments", "{}")

        print(f"[FunctionCall] Calling: {name}, call_id: {call_id}, args: {arguments_str}")

        try:
            arguments = json.loads(arguments_str) if arguments_str else {}
        except json.JSONDecodeError:
            arguments = {}

        func = self.functions.get(name)
        if func is None:
            output = json.dumps({"error": f"Unregistered function: {name}"})
        else:
            try:
                if asyncio.iscoroutinefunction(func):
                    result = await func(**arguments)
                else:
                    result = func(**arguments)
                output = str(result) if result is not None else ""
            except Exception as e:
                output = json.dumps({"error": str(e)})
                traceback.print_exc()

        // Kembalikan function_call_output
        await self.send_event({
            "type": "conversation.item.create",
            "item": {
                "type": "function_call_output",
                "call_id": call_id,
                "output": output,
            }
        })

        // Picu inferensi tindak lanjut
        await self.create_response()

    async def handle_messages(self) -> None:
        try:
            async for message in self.ws:
                event = json.loads(message)
                event_type = event.get("type")

                print(self._format_event_for_log(event))

                if event_type == "error":
                    continue
                elif event_type == "response.created":
                    self._current_response_id = event.get("response", {}).get("id")
                    self._is_responding = True
                    self._audio_suppressed = False
                elif event_type == "response.output_item.added":
                    self._current_item_id = event.get("item", {}).get("id")
                elif event_type == "response.done":
                    self._is_responding = False
                    self._current_response_id = None
                    self._current_item_id = None
                elif event_type == "input_audio_buffer.speech_started":
                    print("----------------Speech Started----------------")
                    if self.on_speech_started:
                        self.on_speech_started()
                    if self._is_responding:
                        await self.handle_interruption()
                elif event_type == "response.audio.delta":
                    if self._audio_suppressed:
                        continue
                    if self.on_audio_delta_b64:
                        self.on_audio_delta_b64(event["delta"])
                elif event_type == "response.function_call_arguments.done":
                    await self._handle_function_call(event)
                elif event_type in self.extra_event_handlers:
                    self.extra_event_handlers[event_type](event)
                elif event_type == "input_audio_buffer.speech_stopped":
                    print("----------------Speech Stopped----------------")
        except websockets.exceptions.ConnectionClosed:
            print(" Connection closed")
        except Exception as e:
            print(" Error in message handling: ", str(e))
            traceback.print_exc()

    async def close(self) -> None:
        """Menutup koneksi WebSocket."""
        if self.ws:
            await self.ws.close()

def _audio_energy(audio_data: bytes) -> float:
    count = len(audio_data) // 2
    if count == 0:
        return 0.0
    samples = struct.unpack(f'<{count}h', audio_data)
    return sum(abs(s) for s in samples) / count

async def record_and_send(client, player, echo_suppression=True):
    p = pyaudio.PyAudio()
    stream = p.open(format=pyaudio.paInt16, channels=1, rate=16000, input=True)
    print("Recording started. Speak into the microphone...")
    if echo_suppression:
        print("Note: Echo suppression is enabled (microphone is muted while the AI is speaking; interruption is not supported). If you are using headphones, set echo_suppression=False to enable interruption.")
    else:
        print("Note: Headphone mode. Voice interruption is supported.")
    playback_end_time = 0.0
    NOISE_GATE_THRESHOLD = 500
    try:
        while True:
            audio_data = await asyncio.to_thread(stream.read, 3200, False)
            if echo_suppression:
                is_active = client._is_responding or player.is_playing()
                if is_active:
                    playback_end_time = time.time()
                    await asyncio.sleep(0.02)
                    continue
                if time.time() - playback_end_time < 0.5:
                    await asyncio.sleep(0.02)
                    continue
            else:
                if client._is_responding or player.is_playing():
                    if _audio_energy(audio_data) < NOISE_GATE_THRESHOLD:
                        await asyncio.sleep(0.02)
                        continue
            await client.stream_audio(audio_data)
            await asyncio.sleep(0.02)
    finally:
        stream.stop_stream(); stream.close(); p.terminate()

async def main():
    pya = pyaudio.PyAudio()
    // Laju sampel output 24 kHz, sesuai dengan format audio sisi server
    player = B64PCMPlayer(pya, sample_rate=24000)

    client = FunRealtimeClient(
        // Berikut ini adalah URL WebSocket untuk wilayah Singapura. Ganti {WorkspaceId} (termasuk tanda kurung kurawal) dengan ID ruang kerja aktual Anda. URL berbeda-beda berdasarkan wilayah.
        base_url="wss://{WorkspaceId}.ap-southeast-1.maas.aliyuncs.com/api-ws/v1/realtime",
        api_key=os.environ['DASHSCOPE_API_KEY'],
        model="qwen-audio-3.0-realtime-plus",
        voice="longanqian",
        turn_detection_mode=TurnDetectionMode.SERVER_VAD,
        tools=tools,
        functions=functions,
        on_audio_delta_b64=player.add_data,
        // Membersihkan buffer pemutaran pada interupsi suara
        on_speech_started=player.cancel_playing,
    )

    await client.connect()
    print("Connected. Starting real-time conversation (Function Calling enabled)...")

    try:
        await asyncio.gather(client.handle_messages(), record_and_send(client, player, echo_suppression=False))
    finally:
        await client.close()
        player.shutdown()
        pya.terminate()

if __name__ == "__main__":
    try:
        asyncio.run(main())
    except KeyboardInterrupt:
        print("\nProgram exited.")

Jalankan python realtime_fc_demo.py dan bicaralah ke mikrofon Anda untuk mencoba percakapan real-time dengan Function Calling. Misalnya, tanyakan "Apa cuaca di Hangzhou?" atau "Berapa harga tiket kereta dari Beijing ke Shanghai?" dan model secara otomatis memanggil tool yang sesuai dan merespons dengan hasilnya.

Manajemen konteks percakapan

Qwen-Audio memungkinkan Anda mengelola item percakapan dalam konteks melalui event klien. Gunakan ini untuk menyuntikkan konteks historis, menambahkan informasi teks, atau menghapus item percakapan yang tidak relevan.

• Buat item percakapan (conversation.item.create): menyisipkan item percakapan ke dalam konteks. Tiga nilai item.type berikut didukung:

  • message: pesan percakapan reguler. Tentukan role (system, user, atau assistant) dan array content. Gunakan ini untuk menyuntikkan riwayat percakapan atau instruksi sistem.
  • function_call: permintaan panggilan fungsi. Tentukan call_id, name, dan arguments (string JSON). Biasanya dihasilkan oleh server, tetapi klien juga dapat menggunakannya untuk menyuntikkan catatan panggilan fungsi historis.
  • function_call_output: hasil eksekusi tool. Tentukan call_id dan output (string JSON). Setelah menerima function_call, jalankan tool di klien dan kembalikan hasilnya dengan tipe ini.

  Parameter opsional previous_item_id menentukan item percakapan yang sudah ada setelahnya item baru akan disisipkan. Ini memungkinkan Anda menyisipkan konten di posisi mana pun dalam riwayat percakapan. Jika dihilangkan, item baru ditambahkan di akhir.

  • Sisipkan pesan pengguna di posisi tertentu:

{
    "type": "conversation.item.create",
    "previous_item_id": "item_abc",
    "item": {
        "type": "message",
        "role": "user",
        "content": [
            { "type": "input_text", "text": "Please summarize our last conversation" }
        ]
    }
}

• Kembalikan hasil Function Calling:

{
    "type": "conversation.item.create",
    "item": {
        "type": "function_call_output",
        "call_id": "call_xxx",
        "output": "{\"temperature\":18,\"condition\":\"sunny\"}"
    }
}

• Ambil item percakapan (conversation.item.retrieve): menanyakan item percakapan yang disimpan di server. Untuk konten bertipe audio, hanya teks transkripsi yang dikembalikan, bukan data audio mentahnya.

{
    "type": "conversation.item.retrieve",
    "item_id": "item_xxx"
}

• Hapus item percakapan (conversation.item.delete): menghapus item tertentu dari konteks percakapan.

{
    "type": "conversation.item.delete",
    "item_id": "item_xxx"
}

Transkripsi audio ambient

Hanya mode smart_turn. Saat VAD mendeteksi aktivitas ucapan tetapi analisis semantik menentukan itu bukan giliran yang valid (seperti kebisingan atau suara pengisi seperti "uh" atau "hmm"), server tidak memicu giliran percakapan. Sebagai gantinya, server mengirim hasil ASR ke klien sebagai event ambient_audio_transcription. Transkripsi ini tidak ditulis ke konteks percakapan.

{
    "type": "conversation.item.ambient_audio_transcription.delta",
    "item_id": "item_xxx",
    "text": "hmm",
    "stash": ""
}

Seperti event transkripsi ucapan pengguna, transkripsi audio ambient mencakup fase delta dan completed. Gunakan event ini untuk mengimplementasikan pemantauan audio ambient atau kesadaran adegan percakapan.

Peningkatan speaker

Hanya mode smart_turn. Kirimkan URL audio rekaman sebelumnya dari pengguna target dalam session.update. Model akan mengunci speaker tersebut selama percakapan duplex, secara efektif mengabaikan suara lain dan kebisingan latar belakang, memungkinkan interaksi duplex yang lancar di lingkungan terbuka.

Konfigurasi: kirimkan URL audio voiceprint yang dapat diakses publik dalam turn_detection.voiceprint_audio_urls dalam session.update pertama.

{
  "type": "session.update",
  "session": {
    "turn_detection": {
      "type": "smart_turn",
      "voiceprint_audio_urls": ["https://example.com/speaker.wav"]
    }
  }
}

Persyaratan parameter:

• Maksimal 5 URL. Audio harus dalam format PCM atau WAV 16 kHz.
• Parameter ini hanya berlaku dalam session.update pertama. Field ini diabaikan dalam panggilan berikutnya.

Event pendaftaran: setelah menerima konfigurasi, server secara asinkron melakukan pendaftaran voiceprint dan memberi tahu hasilnya melalui event berikut:

• voiceprint_audio_list.in_progress: pendaftaran telah dimulai. Dikirim sebelum session.updated, membawa item_id.
• voiceprint_audio_list.completed: pendaftaran berhasil. item_id cocok dengan yang ada di in_progress.
• voiceprint_audio_list.failed: pendaftaran gagal, dengan field reason yang menjelaskan kesalahan (misalnya, URL audio tidak dapat diakses). Kegagalan pendaftaran tidak menghentikan percakapan yang sedang berlangsung.

Peluncuran

Siapkan toleransi kesalahan

• Koneksi ulang klien: Terapkan koneksi ulang otomatis untuk menangani fluktuasi jaringan. Atur sinyal koneksi ulang di callback on_error dan gunakan exponential backoff (misalnya, tunggu 1 s, 2 s, 4 s) untuk percobaan ulang.
• Klasifikasi kesalahan: kesalahan klien (invalid_request_error) tidak memutus sesi; catat atau sesuaikan parameter. Kesalahan server (server_error) mengakhiri koneksi dan memerlukan rekoneksi.
• Penanganan interupsi: dalam mode server_vad / smart_turn, ucapan pengguna baru secara otomatis menginterupsi respons model yang sedang berlangsung (response.done mengembalikan status=cancelled). Saat menerima input_audio_buffer.speech_started, segera bersihkan buffer pemutaran lokal untuk menghindari tumpang tindih audio.

Siklus hidup koneksi

Sesi WebSocket tipikal mengikuti siklus hidup ini:

1. Koneksi: klien memulai koneksi WebSocket dan server mengembalikan event session.created.
2. Konfigurasi: klien mengirim session.update untuk mengatur mode interaksi, suara, tool, dan parameter lainnya. Selesaikan langkah ini sebelum mengirim audio apa pun.
3. Berinteraksi: klien terus-menerus mengalirkan audio (input_audio_buffer.append). Server melakukan inferensi berdasarkan deteksi VAD atau pemicu manual dan mengembalikan ucapan dan teks secara streaming.
4. Tutup: klien menutup koneksi WebSocket. Server juga dapat memutus jika koneksi tidak aktif terlalu lama.

Optimasi latensi

• Ukuran chunk audio: kirim sekitar 100 md data audio per chunk (16 kHz x 16 bit x mono = 3.200 byte per chunk). Ini menyeimbangkan kinerja real-time dengan efisiensi jaringan.
• Pemutaran streaming: mulai memutar audio segera setelah response.audio.delta tiba. Jangan tunggu response.done untuk memutar respons penuh.
• Bersihkan buffer pada interupsi: saat menerima input_audio_buffer.speech_started, segera bersihkan buffer pemutaran lokal untuk mencegah audio lama terus diputar.

Model dan wilayah yang didukung

Referensi API

• API WebSocket
• Event klien
• Event server
• SDK Klien AOQ
• Ikhtisar API Realtime (deskripsi protokol WebRTC)