AI_TRANSCRIBE - Hologres - Alibaba Cloud Documentation Center

AI_TRANSCRIBE converts audio files into structured text data. Built on speech recognition models from Alibaba Cloud Model Studio, this function supports multiple timestamp granularity modes to fit scenarios such as meeting transcription, subtitle generation, and speaker diarization.

Scenarios

Meeting transcription: Convert meeting recordings to text for archival and search.
Subtitle generation: Segment audio at the sentence level and generate timestamped subtitles.
Speaker diarization: Identify different speakers in multi-speaker audio, suitable for meeting minutes.
Audio search: Provide word-level timestamps for precise audio and video content search.
Language learning: Enable word-by-word playback and read-along practice for language learning applications.

Limits

Hologres V4.2 or later.
OSS access must be configured to access audio files stored in OSS.
A speech recognition model is deployed in the Hologres console using the managed model deployment process. No API key configuration is required. For more information, see Managed models.

Syntax

AI_TRANSCRIBE(model, audio_file [, options])

Parameters

Parameter	Data type	Required	Description
model	TEXT	Yes	The model name. Deploy the model in the Hologres console as a managed model before use. For more information, see Managed models.
audio_file	FILE	Yes	The audio file, obtained by using the `to_file()` function or an Object Table.
options	JSONB	No	Configuration options. Must be explicitly cast by using `::jsonb`.

Options

Parameter	Data type	Default	Description
timestamp_granularity	TEXT	'none'	The timestamp granularity mode. Valid values: 'none', 'sentence', 'speaker', and 'word'.

Timestamp granularity modes

The timestamp_granularity parameter supports the following values:

none (default): Converts the entire audio file into a single text block. Suitable for RAG scenarios. The output contains the text and duration fields.
sentence: Segments the transcription at the sentence level. Suitable for subtitle generation. The output contains a segments array, where each item includes the start, end, and text fields.
speaker: Identifies different speakers in multi-speaker audio. Suitable for meeting minutes. The output contains a segments array, where each item additionally includes the speaker_id field.
word: Provides word-level timestamps. Suitable for audio and video search, and language learning. The output contains a segments array with word-level start, end, and text fields.

Note

The options parameter must be explicitly cast by using ::jsonb. Otherwise, an ambiguous function error occurs.

Correct usage:

SELECT AI_TRANSCRIBE('fun_asr', file, '{"timestamp_granularity": "sentence"}'::jsonb);

Incorrect usage (missing ::jsonb):

SELECT AI_TRANSCRIBE('fun_asr', file, '{"timestamp_granularity": "sentence"}');
-- Error: function ai_transcribe(unknown, file, unknown) is not unique

Examples

Note

The audio files in the following examples are from the AliMeeting open-source meeting speech dataset. Replace the OSS path, endpoint, and RAM role ARN with your actual values.

The examples use oss-cn-beijing-internal.aliyuncs.com as the OSS endpoint because the FunASR model service is deployed in the China (Beijing) region. When you use an internal endpoint, the OSS bucket must be in the same region as the model service.

Example 1: Basic transcription (none mode)

SELECT AI_TRANSCRIBE(
  'fun_asr',
  TO_FILE(
    'oss://xxxx/audio/alimeeting_audio/R8003_M8001_N_SPK8002.wav',
    'oss-cn-beijing-internal.aliyuncs.com',
    'acs:ram::xxx:role/xxxx'
  )
);

Sample output:

{
  "text": "So what did you need to see me about? Well, the thing is, I called you in today to discuss the student exchange event planning. I'd like to hear your thoughts on how to organize it, so we can exchange ideas.",
  "duration": 15.2
}

Example 2: Sentence-level segmentation (sentence mode)

SELECT AI_TRANSCRIBE(
  'fun_asr',
  TO_FILE(
    'oss://xxxx/audio/alimeeting_audio/R8003_M8001_N_SPK8002.wav',
    'oss-cn-beijing-internal.aliyuncs.com',
    'acs:ram::xxx:role/xxxx'
  ),
  '{"timestamp_granularity": "sentence"}'::jsonb
);

Sample output:

{
  "duration": 15.2,
  "segments": [
    {"start": 0.16, "end": 3.48, "text": "So what did you need to see me about?"},
    {"start": 3.72, "end": 15.04, "text": "Well, the thing is, I called you in today to discuss the student exchange event planning. I'd like to hear your thoughts on how to organize it, so we can exchange ideas."}
  ]
}

Example 3: Speaker diarization (speaker mode)

SELECT AI_TRANSCRIBE(
  'fun_asr',
  TO_FILE(
    'oss://xxxx/audio/alimeeting_audio/R8007_M8010_N_SPK8054.wav',
    'oss-cn-beijing-internal.aliyuncs.com',
    'acs:ram::xxx:role/xxxx'
  ),
  '{"timestamp_granularity": "speaker"}'::jsonb
);

Sample output:

{
  "duration": 45.6,
  "segments": [
    {"start": 0.5, "end": 8.2, "speaker_id": 0, "text": "Hi everyone, let's discuss the project progress today."},
    {"start": 8.5, "end": 20.3, "speaker_id": 1, "text": "Sure. The frontend development is 80% complete on my end. I expect to start integration testing next week."},
    {"start": 20.8, "end": 35.1, "speaker_id": 0, "text": "How about the backend? Can you keep up with the timeline?"},
    {"start": 35.5, "end": 45.2, "speaker_id": 2, "text": "The backend APIs are ready. We are waiting for the frontend integration."}
  ]
}

Example 4: Word-level timestamps (word mode)

SELECT AI_TRANSCRIBE(
  'fun_asr',
  TO_FILE(
    'oss://xxxx/audio/alimeeting_audio/R8003_M8001_N_SPK8002.wav',
    'oss-cn-beijing-internal.aliyuncs.com',
    'acs:ram::xxx:role/xxxx'
  ),
  '{"timestamp_granularity": "word"}'::jsonb
);

Sample output (partial):

{
  "duration": 15.2,
  "segments": [
    {"start": 0.16, "end": 0.52, "text": "So"},
    {"start": 0.52, "end": 0.88, "text": "what"},
    {"start": 0.88, "end": 1.24, "text": "did"},
    {"start": 1.24, "end": 1.60, "text": "you"},
    {"start": 1.60, "end": 1.96, "text": "need"},
    {"start": 1.96, "end": 2.32, "text": "to"},
    {"start": 2.32, "end": 2.68, "text": "see"},
    {"start": 2.68, "end": 3.04, "text": "me"},
    {"start": 3.04, "end": 3.48, "text": "about"}
  ]
}

Example 5: Batch transcription with Object Table

-- Create an Object Table
CREATE OBJECT TABLE audio_files WITH (
  path = 'oss://xxxx/audio/alimeeting_audio/',
  oss_endpoint = 'oss-cn-beijing-internal.aliyuncs.com',
  role_arn = 'acs:ram::xxx:role/xxxx'
);

-- Refresh the Object Table
REFRESH OBJECT TABLE audio_files;

-- Transcribe all audio files
SELECT
  object_uri,
  AI_TRANSCRIBE('fun_asr', file, '{"timestamp_granularity": "sentence"}'::jsonb) AS transcription
FROM audio_files
LIMIT 2;

Sample output:

         object_uri          |                          transcription
-----------------------------+--------------------------------------------------------------
 R8003_M8001_N_SPK8002.wav  | {"duration": 15.2, "segments": [{"start": 0.16, "end": 3.48, "text": "So what did you need to see me about?"}, {"start": 3.72, "end": 15.04, "text": "Well, the thing is, I called you in today to discuss the student exchange event planning. I'd like to hear your thoughts on how to organize it, so we can exchange ideas."}]}
 R8007_M8010_N_SPK8054.wav  | {"duration": 45.6, "segments": [{"start": 0.5, "end": 8.2, "text": "Hi everyone, let's discuss the project progress today."}, {"start": 8.5, "end": 20.3, "text": "Sure. The frontend development is 80% complete on my end. I expect to start integration testing next week."}]}
(2 rows)

Example 6: Incremental transcription with Dynamic Table

Use a Dynamic Table with incremental refresh to process audio files. This approach reduces resource consumption and suits large-scale audio processing scenarios such as customer service interactions and in-vehicle voice systems in autonomous driving.

-- Create an Object Table
CREATE OBJECT TABLE audio_files WITH (
  path = 'oss://xxxx/audio/alimeeting_audio/',
  oss_endpoint = 'oss-cn-beijing-internal.aliyuncs.com',
  role_arn = 'acs:ram::xxx:role/xxxx'
);

-- Refresh the Object Table
REFRESH OBJECT TABLE audio_files;

-- Create a Dynamic Table to automatically transcribe new files
CREATE DYNAMIC TABLE audio_transcriptions
WITH (freshness = '10 minutes')
AS SELECT
  object_uri,
  AI_TRANSCRIBE('fun_asr', file, '{"timestamp_granularity": "sentence"}'::jsonb) AS transcription
FROM audio_files;

-- Wait for automatic refresh or trigger manually
REFRESH DYNAMIC TABLE audio_transcriptions;

-- Query the transcription results
SELECT
  object_uri,
  transcription->>'duration' AS duration,
  (jsonb_array_elements(transcription->'segments'))->>'start' AS start_time,
  (jsonb_array_elements(transcription->'segments'))->>'end' AS end_time,
  (jsonb_array_elements(transcription->'segments'))->>'text' AS text
FROM audio_transcriptions
WHERE transcription->'segments' IS NOT NULL
LIMIT 10;

Sample query output:

         object_uri          | duration | start_time | end_time |                    text
-----------------------------+----------+------------+----------+--------------------------------------------
 R8003_M8001_N_SPK8002.wav  | 15.2     | 0.16       | 3.48     | So what did you need to see me about?
 R8003_M8001_N_SPK8002.wav  | 15.2     | 3.72       | 15.04    | Well, the thing is, I called you in today to discuss the student exchange event planning.
 R8007_M8010_N_SPK8054.wav  | 45.6     | 0.5        | 8.2      | Hi everyone, let's discuss the project progress today.
 R8007_M8010_N_SPK8054.wav  | 45.6     | 8.5        | 20.3     | Sure. The frontend development is 80% complete on my end. I expect to start integration testing next week.
(4 rows)