This guide explains how to integrate the Native RTS SDK with an FFmpeg-based player on Android to implement RTS, using ijkplayer (tag k0.8.8) as an example.
Prerequisites
You must have compiled the ijkplayer source code. For instructions, see the README.md file in the ijkplayer repository.
Procedure
-
Download and decompress the ijkplayer source code. For the download link, see ijkplayer.
-
Download and decompress the Native RTS SDK. For the download link, see Release notes.
-
Modify the ijkplayer build scripts.
-
Modify ijkplayer-android/init-android.sh so that the call to
pull_forkretains onlyarmv7aandarm64.@@ -39,11 +39,11 @@ function pull_fork() cd - } -pull_fork "armv5" +# pull_fork "armv5" pull_fork "armv7a" pull_fork "arm64" -pull_fork "x86" -pull_fork "x86_64" +# pull_fork "x86" +# pull_fork "x86_64" ./init-config.sh ./init-android-libyuv.sh -
Modify ijkplayer-android/config/module-lite.sh to enable PCM decoding because the Native RTS SDK outputs PCM data.
# aliyun rts export COMMON_FF_CFG_FLAGS="$COMMON_FF_CFG_FLAGS --enable-decoder=pcm_s16be_planar" export COMMON_FF_CFG_FLAGS="$COMMON_FF_CFG_FLAGS --enable-decoder=pcm_s16le" export COMMON_FF_CFG_FLAGS="$COMMON_FF_CFG_FLAGS --enable-decoder=pcm_s16le_planar" -
Modify ijkplayer-android/android/contrib/compile-ffmpeg.sh to retain only armv7a and arm64 in FF_ACT_ARCHS_64.
FF_ACT_ARCHS_64="armv7a arm64" -
Modify ijkplayer-android/android/compile-ijk.sh so that ACT_ABI_64 includes only armv7a and arm64.
ACT_ABI_64="armv7a arm64"
-
-
Integrate the Native RTS SDK into ijkplayer as a plug-in. You can choose one of the following two methods.
Integration method
Description
Advantage
Disadvantage
Extend FFmpeg.
Extend the FFmpeg demuxer in ijkplayer.
Easier to use. No special logic is required to handle ARTC URLs.
Requires recompiling the FFmpeg library.
Extend ijkplayer.
Add an AVInputFormat to ijkplayer.
Does not require compiling FFmpeg.
Requires adding custom logic to
ff_ffplay.c. -
Integrate ijkplayer into your project.
-
Copy the RtsNetSDK.jar file from the Native RTS SDK to your project.
-
In the Activity that uses RTS playback, load the Native RTS SDK dynamic library.
static { System.loadLibrary("RtsSDK"); } -
Import the
ijkplayer-arm64,ijkplayer-armv7a, andijkplayer-javamodules.
-
-
Call ijkplayer APIs to implement RTS.
-
Create an ijkplayer instance
mIjkPlayer = new IjkMediaPlayer(); -
Set callbacks
mIjkPlayer.setOnPreparedListener(new IMediaPlayer.OnPreparedListener() { @Override public void onPrepared(IMediaPlayer iMediaPlayer) { } }); mIjkPlayer.setOnInfoListener(new IMediaPlayer.OnInfoListener() { @Override public boolean onInfo(IMediaPlayer iMediaPlayer, int arg1, int arg2) { }); mIjkPlayer.setOnVideoSizeChangedListener(new IMediaPlayer.OnVideoSizeChangedListener() { @Override public void onVideoSizeChanged(IMediaPlayer iMediaPlayer, int width, int height, int sarNum, int sarDen) { } }); mIjkPlayer.setOnErrorListener(new IMediaPlayer.OnErrorListener() { @Override public boolean onError(IMediaPlayer iMediaPlayer, int framework_err, int impl_err) { return false; } }); -
Set the display surface
mIjkPlayer.setSurface(surface); -
Set the playback source
mIjkPlayer.setDataSource("artc://<your_stream_url>"); -
Control the playback state
public void prepare() { if(mIjkPlayer != null){ mIjkPlayer.prepareAsync(); } } @Override public void start() { if(mIjkPlayer != null){ mIjkPlayer.start(); } } public void stop() { if(mIjkPlayer != null){ mIjkPlayer.stop(); } } public void release() { if(mIjkPlayer != null){ mIjkPlayer.release(); } }
-
Listen for RTS events
-
Message callbacks in ijkplayer
Open the
ijkplayer/android/ijkplayerproject.-
Modify the ff_ffplay.c file, and insert the following code block after the
static int audio_open(FFPlayer *opaque, int64_t wanted_channel_layout, int wanted_nb_channels, int wanted_sample_rate, struct AudioParams *audio_hw_params){}method.extern int artcDemuxerMessage(struct AVFormatContext *s, int type, void *data, size_t data_size); // Alibaba Cloud RTS: Receive RTS message int onArtcDemuxerMessage(struct AVFormatContext *s, int type, void *data, size_t data_size) { return artcDemuxerMessage(s, type, data, data_size); } int artcDemuxerMessage(struct AVFormatContext *s, int type, void *data, size_t data_size) { // Alibaba Cloud RTS: Send message to the app FFPlayer *ffp = (FFPlayer *)s->opaque; const char *data_msg = (const char *)data; ffp_notify_msg4(ffp,FFP_MSG_ARTC_DIRECTCOMPONENTMSG,type,0,data_msg,data_size); return 0; } -
Modify the ff_ffplay.c file, and insert the following RTS code block into the
static int is_realtime(AVFormatContext *s){}function.static int is_realtime(AVFormatContext *s) { if( !strcmp(s->iformat->name, "rtp") || !strcmp(s->iformat->name, "rtsp") || !strcmp(s->iformat->name, "sdp") // --- Start of RTS code block --- || !strcmp(s->iformat->name, "artc") // --- End of RTS code block --- ) return 1; if(s->pb && ( !strncmp(s->filename, "rtp:", 4) || !strncmp(s->filename, "udp:", 4) ) ) return 1; return 0; } -
Modify the ff_ffplay.c file. In the
static int read_thread(void *arg){}method, insert the following RTS code block.static int read_thread(void *arg) { ...... ic = avformat_alloc_context(); if (!ic) { av_log(NULL, AV_LOG_FATAL, "Could not allocate context.\n"); ret = AVERROR(ENOMEM); goto fail; } // --- Start of RTS code block --- ic->opaque = ffp; ic->control_message_cb = onArtcDemuxerMessage; // --- End of RTS code block --- ...... if (ffp->skip_calc_frame_rate) { av_dict_set_int(&ic->metadata, "skip-calc-frame-rate", ffp->skip_calc_frame_rate, 0); av_dict_set_int(&ffp->format_opts, "skip-calc-frame-rate", ffp->skip_calc_frame_rate, 0); } // --- Start of RTS code block --- if(strncmp(is->filename, "artc://", 7) == 0) { extern AVInputFormat ff_rtc_demuxer; is->iformat = &ff_rtc_demuxer; } else { if(ffp->iformat_name) is->iformat = av_find_input_format(ffp->iformat_name); } // --- End of RTS code block --- ...... pkt->flags = 0; // --- Start of RTS code block --- if(strncmp(is->filename, "artc://", 7) == 0) { bool videoExist = is->video_stream >= 0; bool audioExist = is->audio_stream >= 0; // av_log(NULL, AV_LOG_INFO, "videoDuration %lld audioDuration %lld rate %f videoframeQue %d audioFrameque %d\n", // is->videoq.duration, is->audioq.duration, ffp->pf_playback_rate, // frame_queue_nb_remaining(&is->pictq), frame_queue_nb_remaining(&is->sampq)); if(!videoExist) { if(is->audioq.duration > 300 ) { // accelerate if(ffp->pf_playback_rate <= 1.0) { ffp->pf_playback_rate = 1.3; ffp->pf_playback_rate_changed = 1; av_log(NULL, AV_LOG_INFO, "Alibaba Cloud RTS: set rate to %f\n", ffp->pf_playback_rate); } } else if(is->audioq.duration < 200) { // restore speed if(ffp->pf_playback_rate > 1.0) { ffp->pf_playback_rate = 1.0; ffp->pf_playback_rate_changed = 1; av_log(NULL, AV_LOG_INFO, "Alibaba Cloud RTS: restore rate 1.0\n"); } } } else if((!videoExist || (videoExist && is->videoq.duration > 300)) && (!audioExist || (audioExist && is->audioq.duration > 300))) { if(ffp->pf_playback_rate <= 1) { ffp->pf_playback_rate = 1.3; ffp->pf_playback_rate_changed = 1; av_log(NULL, AV_LOG_INFO, "Alibaba Cloud RTS: set rate 1.1\n"); } } else if((videoExist && is->videoq.duration <= 100) || (audioExist && is->audioq.duration <= 100)){ if(ffp->pf_playback_rate > 1) { ffp->pf_playback_rate = 1; ffp->pf_playback_rate_changed = 1; av_log(NULL, AV_LOG_INFO, "Alibaba Cloud RTS: set rate 1\n"); } } } // --- End of RTS code block --- ...... } -
Modify the
ff_ffmsg.hfile to add the interface declaration for RTS messages.#define FFP_MSG_ARTC_DIRECTCOMPONENTMSG 3000 -
Send RTS messages to the upper layer.
-
Modify the
ijkplayer_android_def.hfile to add the enumeration type for RTS messages.enum media_event_type { // Other code is omitted. MEDIA_ARTC_MESSAGE = 3000, // Alibaba Cloud RTS: msg info key }; -
Modify the
ijkplayer_jni.cfile to send RTS messages to the upper layer.static void message_loop_n(JNIEnv *env, IjkMediaPlayer *mp) { //... Other code is omitted. while (1) { switch (msg.what) { // Alibaba Cloud RTS: Post RTS event case FFP_MSG_ARTC_DIRECTCOMPONENTMSG: if (msg.obj) { const char * result = (const char *)msg.obj; ALOGE("Alibaba Cloud RTS: FFP_MSG_ARTC_DIRECTCOMPONENTMSG = %d , %s\n", msg.arg1,result); jstring data_msg = (*env)->NewStringUTF(env, result); post_event2(env, weak_thiz, MEDIA_ARTC_MESSAGE, msg.arg1, 0, data_msg); J4A_DeleteLocalRef__p(env, &data_msg); } else { post_event2(env, weak_thiz, MEDIA_ARTC_MESSAGE, 0, 0, NULL); } break; } } } -
Modify the
IjkMediaPlayer.javafile to receive RTS messages.private static class EventHandler extends Handler { @Override public void handleMessage(Message msg) { IjkMediaPlayer player = mWeakPlayer.get(); if (player == null || player.mNativeMediaPlayer == 0) { DebugLog.w(TAG, "IjkMediaPlayer went away with unhandled events"); return; } switch (msg.what) { // Alibaba Cloud RTS: Post event case MEDIA_ARTC_MESSAGE: if(msg.arg1 == FFP_ARTC_CONNECT_LOST){ player.notifyOnARTCMessage("NetWorkDisconnect",0,""); }else if(msg.arg1 == FFP_ARTC_RECOVERED){ player.notifyOnARTCMessage("NetWorkRetrySuccess",0,""); }else{ if(msg.obj == null){ player.notifyOnARTCMessage("DirectComponentMSG",0, ""); }else{ String result = (String) msg.obj; result = result.replaceAll("\"",""); player.mAliyunRTSMsgBuilder = new StringBuilder("{\"content\":"); player.mAliyunRTSMsgBuilder.append("\"").append(result).append("\"}"); player.notifyOnARTCMessage("DirectComponentMSG",0, (String) result); } } break; } } }Add the RTS message callback to the
IMediaPlayer.javafile.// Alibaba Cloud RTS: RTS message callback interface OnARTCMessageListener{ void onMessage(String name,int externValue,String extraMsg); }Add the listener settings to the
AbstractMediaPlayer.javafile.// Alibaba Cloud RTS: Set RTS message listener public final void setOnARTCMessageListener(OnARTCMessageListener listener){ this.mOnARTCMessageListener = listener; } // Alibaba Cloud RTS: Callback protected final void notifyOnARTCMessage(String name,int externValue,String extraMsg){ if(mOnARTCMessageListener != null){ mOnARTCMessageListener.onMessage(name,externValue,extraMsg); } }Run ./compile-ijk.sh to recompile.
-
-
-
Retry method in ijkplayer
-
Modify the
ff_ffplay.hfile by adding the definition of therts_reload_flagvariable.int rts_reload_flag; -
Modify the ff_ffplay.c file, and insert the following code block after the
static int audio_open(FFPlayer *opaque, int64_t wanted_channel_layout, int wanted_nb_channels, int wanted_sample_rate, struct AudioParams *audio_hw_params){}method.extern int artc_reload(AVFormatContext *ctx); void ffp_rts_reload(FFPlayer *ffp){ if(rts_reload_flag == 0) { rts_reload_flag = 1; } } -
Modify the ff_ffplay.c file, and insert the following RTS code block in the
static int read_thread(void *arg){}function.static int read_thread(void *arg) { ...... #ifdef FFP_MERGE if (is->paused != is->last_paused) { is->last_paused = is->paused; if (is->paused) is->read_pause_return = av_read_pause(ic); else av_read_play(ic); } #endif // --- Start of RTS code block --- if(rts_reload_flag){ rts_reload_flag = 0; av_log(ffp, AV_LOG_ERROR, "param == ffp_rts_reload\n"); VideoState *is = ffp->is; AVFormatContext *ic = is->ic; artc_reload(ic); } // --- End of RTS code block --- ...... } -
Modify the
ijkplayer.hfile by adding the definition of theijkmp_rts_reloadfunction.void ijkmp_rts_reload(IjkMediaPlayer *mp); -
Modify the
ijkplayer.cfile to implement theijkmp_rts_reloadfunction.void ijkmp_rts_reload(IjkMediaPlayer *mp) { ffp_rts_reload(mp->ffplayer); } -
Add the
rtsReloadinterface to the upper layer.-
Modify the
ijkplayer_jni.cfile by adding thereloadinterface.// Alibaba Cloud RTS: Reload native static void IjkMediaPlayer_reload(JNIEnv *env,jobject thiz) { IjkMediaPlayer *mp = jni_get_media_player(env,thiz); ijkmp_rts_reload(mp); LABEL_RETURN: ijkmp_dec_ref_p(&mp); } static JNINativeMethod g_methods[] = { // Alibaba Cloud RTS: Reload { "_reload", "()V", (void *) IjkMediaPlayer_reload }, } -
Modify the
IMediaPlayer.javafile by adding thereloadinterface.// Alibaba Cloud RTS: Reload void reload(); -
Modify the
IjkMediaPlayer.javafile to call the nativereloadmethod.// Alibaba Cloud RTS: Native method private native void _reload(); @Override public void reload() { _reload(); }
-
-
-
Implement stream degradation
-
Listen for the RTS message callback.
mIjkPlayer.setOnARTCMessageListener(new IMediaPlayer.OnARTCMessageListener() { @Override public void onMessage(String name, int externValue, String extraMsg) {} }); -
Implement the stream degradation logic in the RTS message callback method.
Stream degradation is the strategy of replacing the
artc://prefix in the playback URL withrtmp://or changing the URL to thehttp://xxxx.flvformat. After changing the URL, update the player's data source and restart playback.if (SOURCE_URL.startsWith("artc://")) { mIjkPlayer.stop(); // Stop playback. mIjkPlayer.reset(); // Reset ijkplayer. If you do not reset ijkplayer when reusing it, a crash occurs. mIjkPlayer.setDataSource("http://xxx.flv"); mIjkPlayer.prepareAsync(); }When the player's event callback receives a message from the playback component, parse the JSON string of the event description to obtain the event code, which corresponds to a message from the RTS SDK. The required action depends on the code. You must implement stream degradation if you receive
E_DNS_FAIL,E_AUTH_FAIL,E_CONN_TIMEOUT,E_SUB_TIMEOUT, orE_SUB_NO_STREAM. If you receiveE_STREAM_BROKEN, first retry playback. If the error persists, implement stream degradation. If you receiveE_RECV_STOP_SIGNAL, stop playback directly. Stream degradation is not required.// Custom RtsError enumeration class public enum RtsError { /** * DNS resolution failed. */ E_DNS_FAIL(20001), /** * Authentication failed. */ E_AUTH_FAIL(20002), /** * Connection signaling timed out. */ E_CONN_TIMEOUT(20011), /** * Subscription signaling timed out or returned an error. */ E_SUB_TIMEOUT(20012), /** * The subscribed stream does not exist. */ E_SUB_NO_STREAM(20013), /** * Media timeout. No audio or video packets were received. */ E_STREAM_BROKEN(20052), /** * Received a stop signal from the CDN. */ E_RECV_STOP_SIGNAL(20061); private int code; RtsError(int code) { this.code = code; } public int getCode() { return code; } public void setCode(int code) { this.code = code; } } mIjkPlayer.setOnARTCMessageListener(new IMediaPlayer.OnARTCMessageListener() { @Override public void onMessage(String name, int externValue, String msg) { if("DirectComponentMSG".equals(name)){ if (msg.contains("code=" + RtsError.E_DNS_FAIL.getCode()) || msg.contains("code=" + RtsError.E_AUTH_FAIL.getCode()) || msg.contains("code=" + RtsError.E_CONN_TIMEOUT.getCode()) || msg.contains("code=" + RtsError.E_SUB_TIMEOUT.getCode()) || msg.contains("code=" + RtsError.E_SUB_NO_STREAM.getCode()) || msg.contains("code=" + RtsError.E_STREAM_BROKEN.getCode()) || msg.contains("code=" + RtsError.E_RECV_STOP_SIGNAL.getCode())) { // TODO: Implement the stream degradation logic. } } } });
-