This topic describes how to integrate the Native RTS SDK with an FFmpeg-based player on Android to implement RTS. This topic uses the k0.8.8 tag of ijkplayer as an example.
Prerequisites
The ijkplayer source code has been compiled. For more information, see the README.md file in the ijkplayer repository.
Procedure
Download and decompress the ijkplayer source code. For the download link, see ijkplayer.
Download and decompress the Native RTS SDK. For the download link, see Release notes.
Modify the ijkplayer compilation script.
Modify ijkplayer-android/init-android.sh. In the `pull_fork` call, retain only the armv7a and arm64 architectures.
Modify ijkplayer-android/config/module-lite.sh to support Pulse-Code Modulation (PCM) decoding. The Native RTS SDK outputs PCM data.
# aliyun rts export COMMON_FF_CFG_FLAGS="$COMMON_FF_CFG_FLAGS --enable-decoder=pcm_s16be_planar" export COMMON_FF_CFG_FLAGS="$COMMON_FF_CFG_FLAGS --enable-decoder=pcm_s16le" export COMMON_FF_CFG_FLAGS="$COMMON_FF_CFG_FLAGS --enable-decoder=pcm_s16le_planar"Modify ijkplayer-android/android/contrib/compile-ffmpeg.sh. For `FF_ACT_ARCHS_64`, retain only the armv7a and arm64 architectures.
FF_ACT_ARCHS_64="armv7a arm64"Modify ijkplayer-android/android/compile-ijk.sh. For `ACT_ABI_64`, retain only the armv7a and arm64 architectures.
ACT_ABI_64="armv7a arm64"
Integrate the Native RTS SDK into ijkplayer as a plugin. You can use one of the following two methods:
Integration method
Description
Advantages
Disadvantages
Extend FFmpeg
Extend the demuxer of FFmpeg in ijkplayer.
This method is simple. You do not need to add separate logic for Alibaba Real-Time Communication (ARTC) URLs.
You must recompile the FFmpeg library.
Extend ijkplayer
Add AVInputFormat to ijkplayer.
You do not need to compile FFmpeg.
You must add logic code to the ff_ffplay.c file.
Copy the rtsdec.c, ali_net_api.h, rts_api.h, and rts_messages.h files from the Native RTS SDK to the ijkplayer/android/contrib/ffmpeg-arm64/libavformat and ijkplayer/android/contrib/ffmpeg-armv7a/libavformat directories.
Modify the Makefile files and compile the rtsdec.c file.
Modify the ijkplayer/android/contrib/ffmpeg-arm64/libavformat/Makefile and ijkplayer/android/contrib/ffmpeg-armv7a/libavformat/Makefile files to compile the rtsdec.c file.
Modify the allformats.c files.
Modify the ijkplayer/android/contrib/ffmpeg-arm64/libavformat/allformats.c and ijkplayer/android/contrib/ffmpeg-armv7a/libavformat/allformats.c files to support the ARTC protocol by default.
extern AVInputFormat ff_rtc_demuxer; av_register_input_format(&ff_rtc_demuxer);Compile.
Add the Android NDK environment variable. Then, navigate to the ijkplayer/android/contrib directory and run the ./compile-ffmpeg.sh all script. After the compilation is complete, verify that the FFmpeg compilation output files exist in the ijkplayer/android/contrib/build directory.
Add the dynamic libraries of the Native RTS SDK.
Copy the dynamic libraries of the Native RTS SDK to the ijkplayer/android/contrib/build/ffmpeg-arm64/output and ijkplayer/android/contrib/build/ffmpeg-armv7a/output directories.
Copy the rts_api.h and rts_messages.h header files from the Native RTS SDK to the ijkplayer/android/contrib/build/ffmpeg-arm64/output/include and ijkplayer/android/contrib/build/ffmpeg-armv7a/output/include directories, respectively.
Import the dynamic libraries of the Native RTS SDK.
Modify the ijkplayer/android/ijkplayer/ijkplayer-armv7a/src/main/jni/ffmpeg/Android.mk and ijkplayer/android/ijkplayer/ijkplayer-arm64/src/main/jni/ffmpeg/Android.mk files.
include $(CLEAR_VARS) LOCAL_MODULE := rtssdk LOCAL_SRC_FILES := $(MY_APP_FFMPEG_OUTPUT_PATH)/libRtsSDK.so include $(PREBUILT_SHARED_LIBRARY)Modify the ijkplayer/ijkmedia/ijkplayer/Android.mk file to add the Native RTS SDK dynamic library as a dependency.
Add the RTS logic to the ff_ffplay.c file.
Modify the ijkplayer/ijkmedia/ijkplayer/ff_ffplay.c file to set the AVInputFormat function pointer for ARTC.
extern AVInputFormat ff_rtc_demuxer; extern int artc_reload(AVFormatContext *ctx); extern void av_set_rts_demuxer_funcs(const struct rts_glue_funcs *funcs); extern void artc_set_rts_param(char* key, char* value); extern long long artc_get_state(AVFormatContext *ctx, int key); int version = 2; const struct rts_glue_funcs* rts_funcs = get_rts_funcs(version); // set to ffmpeg plugin av_set_rts_demuxer_funcs(rts_funcs); artc_set_rts_param((char*)"AutoReconnect", (char*)"false");Compile.
Navigate to the ijkplayer/android directory and run the ./compile-ijk.sh all script.
Add the dynamic libraries of the Native RTS SDK.
Copy the dynamic libraries of the Native RTS SDK to the ijkplayer/android/contrib/build/ffmpeg-arm64/output and ijkplayer/android/contrib/build/ffmpeg-armv7a/output directories.
Copy the rts_api.h and rts_messages.h header files from the Native RTS SDK to the ijkplayer/android/contrib/build/ffmpeg-arm64/output/include and ijkplayer/android/contrib/build/ffmpeg-armv7a/output/include directories, respectively.
Import the dynamic libraries of the Native RTS SDK.
Modify the ijkplayer/android/ijkplayer/ijkplayer-armv7a/src/main/jni/ffmpeg/Android.mk and ijkplayer/android/ijkplayer/ijkplayer-arm64/src/main/jni/ffmpeg/Android.mk files.
include $(CLEAR_VARS) LOCAL_MODULE := rtssdk LOCAL_SRC_FILES := $(MY_APP_FFMPEG_OUTPUT_PATH)/libRtsSDK.so include $(PREBUILT_SHARED_LIBRARY)Modify the ijkplayer/ijkmedia/ijkplayer/Android.mk file to add the Native RTS SDK dynamic library as a dependency.
Add the RTS logic to the ff_ffplay.c file.
Modify the ijkplayer/ijkmedia/ijkplayer/ff_ffplay.c file to set the AVInputFormat function pointer for ARTC.
if(strncmp(is->filename, "artc://", 7) == 0) { extern AVInputFormat ff_rtc_demuxer; extern int artc_reload(AVFormatContext *ctx); extern void av_set_rts_demuxer_funcs(const struct rts_glue_funcs *funcs); extern void artc_set_rts_param(char* key, char* value); extern long long artc_get_state(AVFormatContext *ctx, int key); int version = 2; const struct rts_glue_funcs* rts_funcs = get_rts_funcs(version); // set to ffmpeg plugin av_set_rts_demuxer_funcs(rts_funcs); artc_set_rts_param((char*)"AutoReconnect", (char*)"false"); is->iformat = &ff_rtc_demuxer; } else { if(ffp->iformat_name) is->iformat = av_find_input_format(ffp->iformat_name); }Copy the rtsdec.c file from the Native RTS SDK to the ijkplayer/ijkmedia/ijkplayer directory. Then, modify the android.mk file in the same directory to add the compilation description for the rtsdec.c file.
LOCAL_SRC_FILES += rtsdec.c
Integrate ijkplayer into your project.
Copy the RtsNetSDK.jar file from the Native RTS SDK to your project.
Import the Native RTS SDK dynamic library into the Activity that is used for RTS playback.
static { System.loadLibrary("RtsSDK"); }Import the ijkplayer-arm64, ijkplayer-armv7a, and ijkplayer-java modules.
Call ijkplayer APIs to implement the RTS feature.
Create an ijkplayer instance
mIjkPlayer = new IjkMediaPlayer();Set the callbacks
mIjkPlayer.setOnPreparedListener(new IMediaPlayer.OnPreparedListener() { @Override public void onPrepared(IMediaPlayer iMediaPlayer) { } }); mIjkPlayer.setOnInfoListener(new IMediaPlayer.OnInfoListener() { @Override public boolean onInfo(IMediaPlayer iMediaPlayer, int arg1, int arg2) { }); mIjkPlayer.setOnVideoSizeChangedListener(new IMediaPlayer.OnVideoSizeChangedListener() { @Override public void onVideoSizeChanged(IMediaPlayer iMediaPlayer, int width, int height, int sarNum, int sarDen) { } }); mIjkPlayer.setOnErrorListener(new IMediaPlayer.OnErrorListener() { @Override public boolean onError(IMediaPlayer iMediaPlayer, int framework_err, int impl_err) { return false; } });Set the display window
mIjkPlayer.setSurface(surface);Set the playback source
mIjkPlayer.setDataSource("artc://<streaming URL>");Control the status
public void prepare() { if(mIjkPlayer != null){ mIjkPlayer.prepareAsync(); } } @Override public void start() { if(mIjkPlayer != null){ mIjkPlayer.start(); } } public void stop() { if(mIjkPlayer != null){ mIjkPlayer.stop(); } } public void release() { if(mIjkPlayer != null){ mIjkPlayer.release(); } }
Listen for RTS events
Listen for message callbacks from the Native RTS SDK in ijkplayer
Open the ijkplayer/android/ijkplayer project.
Modify the ff_ffplay.c file. Insert the following code block after the
static int audio_open(FFPlayer *opaque, int64_t wanted_channel_layout, int wanted_nb_channels, int wanted_sample_rate, struct AudioParams *audio_hw_params){}method.extern int artcDemuxerMessage(struct AVFormatContext *s, int type, void *data, size_t data_size); //aliyun rts:receive artc message int onArtcDemuxerMessage(struct AVFormatContext *s, int type, void *data, size_t data_size) { return artcDemuxerMessage(s, type, data, data_size); } int artcDemuxerMessage(struct AVFormatContext *s, int type, void *data, size_t data_size) { //aliyun rts:send message to app FFPlayer *ffp = (FFPlayer *)s->opaque; const char *data_msg = (const char *)data; ffp_notify_msg4(ffp,FFP_MSG_ARTC_DIRECTCOMPONENTMSG,type,0,data_msg,data_size); return 0; }Modify the ff_ffplay.c file. Insert the following RTS code block into the
static int is_realtime(AVFormatContext *s){}method.static int is_realtime(AVFormatContext *s) { if( !strcmp(s->iformat->name, "rtp") || !strcmp(s->iformat->name, "rtsp") || !strcmp(s->iformat->name, "sdp") // ***rts code block begin*** || !strcmp(s->iformat->name, "artc") // ***rts code block end*** ) return 1; if(s->pb && ( !strncmp(s->filename, "rtp:", 4) || !strncmp(s->filename, "udp:", 4) ) ) return 1; return 0; }Modify the ff_ffplay.c file. Insert the following RTS code block into the
static int read_thread(void *arg){}method.static int read_thread(void *arg) { ...... ic = avformat_alloc_context(); if (!ic) { av_log(NULL, AV_LOG_FATAL, "Could not allocate context.\n"); ret = AVERROR(ENOMEM); goto fail; } // ***rts code block begin*** ic->opaque = ffp; ic->control_message_cb = onArtcDemuxerMessage; // ***rts code block end*** ...... if (ffp->skip_calc_frame_rate) { av_dict_set_int(&ic->metadata, "skip-calc-frame-rate", ffp->skip_calc_frame_rate, 0); av_dict_set_int(&ffp->format_opts, "skip-calc-frame-rate", ffp->skip_calc_frame_rate, 0); } // ***rts code block begin*** if(strncmp(is->filename, "artc://", 7) == 0) { extern AVInputFormat ff_rtc_demuxer; is->iformat = &ff_rtc_demuxer; } else { if(ffp->iformat_name) is->iformat = av_find_input_format(ffp->iformat_name); } // ***rts code block end*** ...... pkt->flags = 0; // ***rts code block begin*** if(strncmp(is->filename, "artc://", 7) == 0) { bool videoExist = is->video_stream >= 0; bool audioExist = is->audio_stream >= 0; // av_log(NULL, AV_LOG_INFO, "videoDuration %lld audioDuration %lld rate %f videoframeQue %d audioFrameque %d\n", // is->videoq.duration, is->audioq.duration, ffp->pf_playback_rate, // frame_queue_nb_remaining(&is->pictq), frame_queue_nb_remaining(&is->sampq)); if(!videoExist) { if(is->audioq.duration > 300 ) { // accelerate if(ffp->pf_playback_rate <= 1.0) { ffp->pf_playback_rate = 1.3; ffp->pf_playback_rate_changed = 1; av_log(NULL, AV_LOG_INFO, "aliyun rts set rate to %f\n", ffp->pf_playback_rate); } } else if(is->audioq.duration < 200) { // restore speed if(ffp->pf_playback_rate > 1.0) { ffp->pf_playback_rate = 1.0; ffp->pf_playback_rate_changed = 1; av_log(NULL, AV_LOG_INFO, "aliyun rts restore rate 1.0\n"); } } } else if((!videoExist || (videoExist && is->videoq.duration > 300)) && (!audioExist || (audioExist && is->audioq.duration > 300))) { if(ffp->pf_playback_rate <= 1) { ffp->pf_playback_rate = 1.3; ffp->pf_playback_rate_changed = 1; av_log(NULL, AV_LOG_INFO, "aliyun rts set rate 1.1\n"); } } else if((videoExist && is->videoq.duration <= 100) || (audioExist && is->audioq.duration <= 100)){ if(ffp->pf_playback_rate > 1) { ffp->pf_playback_rate = 1; ffp->pf_playback_rate_changed = 1; av_log(NULL, AV_LOG_INFO, "aliyun rts set rate 1\n"); } } } // ***rts code block end*** ...... }Modify the ff_ffmsg.h file to add the interface declaration for RTS messages.
#define FFP_MSG_ARTC_DIRECTCOMPONENTMSG 3000Send RTS messages to the upper layer.
Modify the ijkplayer_android_def.h file to add the RTS message enumeration type.
enum media_event_type { // Other code is omitted. MEDIA_ARTC_MESSAGE = 3000, // aliyun rts : msg info key };Modify the ijkplayer_jni.c file to send RTS messages to the upper layer.
static void message_loop_n(JNIEnv *env, IjkMediaPlayer *mp) { //... Other code is omitted. while (1) { switch (msg.what) { //aliyun rts: post artc event case FFP_MSG_ARTC_DIRECTCOMPONENTMSG: if (msg.obj) { const char * result = (const char *)msg.obj; ALOGE("aliyun rts : FFP_MSG_ARTC_DIRECTCOMPONENTMSG = %d , %s\n", msg.arg1,result); jstring data_msg = (*env)->NewStringUTF(env, result); post_event2(env, weak_thiz, MEDIA_ARTC_MESSAGE, msg.arg1, 0, data_msg); J4A_DeleteLocalRef__p(env, &data_msg); } else { post_event2(env, weak_thiz, MEDIA_ARTC_MESSAGE, 0, 0, NULL); } break; } } }Modify the IjkMediaPlayer.java file to receive RTS messages.
private static class EventHandler extends Handler { @Override public void handleMessage(Message msg) { IjkMediaPlayer player = mWeakPlayer.get(); if (player == null || player.mNativeMediaPlayer == 0) { DebugLog.w(TAG, "IjkMediaPlayer went away with unhandled events"); return; } switch (msg.what) { //aliyun artc:post event case MEDIA_ARTC_MESSAGE: if(msg.arg1 == FFP_ARTC_CONNECT_LOST){ player.notifyOnARTCMessage("NetWorkDisconnect",0,""); }else if(msg.arg1 == FFP_ARTC_RECOVERED){ player.notifyOnARTCMessage("NetWorkRetrySuccess",0,""); }else{ if(msg.obj == null){ player.notifyOnARTCMessage("DirectComponentMSG",0, ""); }else{ String result = (String) msg.obj; result = result.replaceAll("\"",""); player.mAliyunRTSMsgBuilder = new StringBuilder("{\"content\":"); player.mAliyunRTSMsgBuilder.append("\"").append(result).append("\"}"); player.notifyOnARTCMessage("DirectComponentMSG",0, (String) result); } } break; } } }Add the RTS message callback to the IMediaPlayer.java file.
//aliyun rts:artc message callback interface OnARTCMessageListener{ void onMessage(String name,int externValue,String extraMsg); }Add the listener settings to the AbstractMediaPlayer.java file.
//aliyun rts : set artc message listener public final void setOnARTCMessageListener(OnARTCMessageListener listener){ this.mOnARTCMessageListener = listener; } //aliyun rts : call back protected final void notifyOnARTCMessage(String name,int externValue,String extraMsg){ if(mOnARTCMessageListener != null){ mOnARTCMessageListener.onMessage(name,externValue,extraMsg); } }Run the ./compile-ijk.sh script to recompile.
Add a retry method for the Native RTS SDK in ijkplayer
Modify the ff_ffplay.h file to add the definition of the `rts_reload_flag` variable.
int rts_reload_flag;Modify the ff_ffplay.c file. Insert the following method block after the
static int audio_open(FFPlayer *opaque, int64_t wanted_channel_layout, int wanted_nb_channels, int wanted_sample_rate, struct AudioParams *audio_hw_params){}method.extern int artc_reload(AVFormatContext *ctx); void ffp_rts_reload(FFPlayer *ffp){ if(rts_reload_flag == 0) { rts_reload_flag = 1; } }Modify the ff_ffplay.c file. Insert the following RTS code block into the
static int read_thread(void *arg){}method.static int read_thread(void *arg) { ...... #ifdef FFP_MERGE if (is->paused != is->last_paused) { is->last_paused = is->paused; if (is->paused) is->read_pause_return = av_read_pause(ic); else av_read_play(ic); } #endif // ***rts code block begin*** if(rts_reload_flag){ rts_reload_flag = 0; av_log(ffp, AV_LOG_ERROR, "param == ffp_rts_reload\n"); VideoState *is = ffp->is; AVFormatContext *ic = is->ic; artc_reload(ic); } // ***rts code block end*** ...... }Modify the ijkplayer.h file to add the definition of the `ijkmp_rts_reload` method.
void ijkmp_rts_reload(IjkMediaPlayer *mp);Modify the ijkplayer.c file to implement the `ijkmp_rts_reload` method.
void ijkmp_rts_reload(IjkMediaPlayer *mp) { ffp_rts_reload(mp->ffplayer); }Add the `rtsReload` interface to the upper layer.
Modify the ijkplayer_jni.c file to add the `reload` interface.
//aliyun rts : reload native static void IjkMediaPlayer_reload(JNIEnv *env,jobject thiz) { IjkMediaPlayer *mp = jni_get_media_player(env,thiz); ijkmp_rts_reload(mp); LABEL_RETURN: ijkmp_dec_ref_p(&mp); } static JNINativeMethod g_methods[] = { //aliyun rts: reload { "_reload", "()V", (void *) IjkMediaPlayer_reload }, }Modify the IMediaPlayer.java file to add the `reload` interface.
//aliyun rts: reload void reload();Modify the IjkMediaPlayer.java file to call the native `reload` method.
//aliyun rts : native method private native void _reload(); @Override public void reload() { _reload(); }
Use the RTS message callback to implement stream degradation
Listen for the RTS message callback.
mIjkPlayer.setOnARTCMessageListener(new IMediaPlayer.OnARTCMessageListener() { @Override public void onMessage(String name, int externValue, String extraMsg) {} });Implement the stream degradation logic in the RTS message callback method.
Stream degradation is a strategy in which you change the prefix of the player's source URL string from `artc://` to `rtmp://` or `http://xxxx.flv`, update the player's URL source, and then restart playback.
if (SOURCE_URL.startsWith("artc://")) { mIjkPlayer.stop();// Stop playback. mIjkPlayer.reset();// Reset the player. If you reuse ijkplayer without resetting it, the application will crash. mIjkPlayer.setDataSource("http://xxx.flv"); mIjkPlayer.prepareAsync(); }During playback startup or live streaming, the player event callback may receive a pass-through message from the playback component. Parse the JSON string of the event description. The `code` field contains a message from the RTS SDK. If the message is `E_DNS_FAIL`, `E_AUTH_FAIL`, `E_CONN_TIMEOUT`, `E_SUB_TIMEOUT`, or `E_SUB_NO_STREAM`, degrade the stream. If the message is `E_STREAM_BROKEN`, retry playback once. If the message is received again, degrade the stream. If the message is `E_RECV_STOP_SIGNAL`, stop playback directly. Stream degradation is not required.
// Custom Rts information enumeration class public enum RtsError { /** * DNS resolution failed. */ E_DNS_FAIL(20001), /** * Authentication failed. */ E_AUTH_FAIL(20002), /** * Connection signaling timed out. */ E_CONN_TIMEOUT(20011), /** * Subscription signaling returned an error or timed out. */ E_SUB_TIMEOUT(20012), /** * The subscribed stream does not exist. */ E_SUB_NO_STREAM(20013), /** * Media timeout. No audio or video packets were received. */ E_STREAM_BROKEN(20052), /** * Received a stop signal from the CDN. */ E_RECV_STOP_SIGNAL(20061); private int code; RtsError(int code) { this.code = code; } public int getCode() { return code; } public void setCode(int code) { this.code = code; } } mIjkPlayer.setOnARTCMessageListener(new IMediaPlayer.OnARTCMessageListener() { @Override public void onMessage(String name, int externValue, String msg) { if("DirectComponentMSG".equals(name)){ if (msg.contains("code=" + RtsError.E_DNS_FAIL.getCode()) || msg.contains("code=" + RtsError.E_AUTH_FAIL.getCode()) || msg.contains("code=" + RtsError.E_CONN_TIMEOUT.getCode()) || msg.contains("code=" + RtsError.E_SUB_TIMEOUT.getCode()) || msg.contains("code=" + RtsError.E_SUB_NO_STREAM.getCode()) || msg.contains("code=" + RtsError.E_STREAM_BROKEN.getCode()) || msg.contains("code=" + RtsError.E_RECV_STOP_SIGNAL.getCode())) { //TODO: Implement the degradation logic. } } } });