This topic describes how to integrate Native Real-Time Streaming (RTS) SDK for Android with a third-party player that is based on FFmpeg to use the Real-Time Streaming (RTS) feature. In this topic, the third-party player ijkplayer k0.8.8 is used.
Prerequisites
The source code of ijkplayer is compiled. For more information, see the README.md file of ijkplayer.
Procedure
Download and decompress the source code of ijkplayer. To obtain the download link, visit the ijkplayer page.
Download and decompress the Native RTS SDK package. For the download link, see Release notes.
Modify the compilation script of ijkplayer.
Modify ijkplayer-android/init-android.sh to retain only the architectures of ARMv7-A and ARM64 for pull_fork.
Modify ijkplayer-android/config/module-lite.sh to support pulse-code modulation (PCM) decoding. This operation is required because Native RTS SDK exports PCM data.
# aliyun rts export COMMON_FF_CFG_FLAGS="$COMMON_FF_CFG_FLAGS --enable-decoder=pcm_s16be_planar" export COMMON_FF_CFG_FLAGS="$COMMON_FF_CFG_FLAGS --enable-decoder=pcm_s16le" export COMMON_FF_CFG_FLAGS="$COMMON_FF_CFG_FLAGS --enable-decoder=pcm_s16le_planar"Modify ijkplayer-android/android/contrib/compile-ffmpeg.sh to retain only the architectures of ARMv7-A and ARM64 for FF_ACT_ARCHS_64.
FF_ACT_ARCHS_64="armv7a arm64"Modify ijkplayer-android/android/compile-ijk.sh to retain only the architectures of ARMv7-A and ARM64 for ACT_ABI_64.
ACT_ABI_64="armv7a arm64"
Integrate Native RTS SDK as a plug-in into ijkplayer. The following table describes the two integration methods.
Integration method
Description
Advantage
Disadvantage
Extend FFmpeg.
Extend FFmpeg by adding a demuxer plug-in.
This method simplifies development. You do not need to develop extra logic for Alibaba Real-Time Communication (ARTC)-based URLs.
You must re-compile FFmpeg.
Extend ijkplayer.
Extend ijkplayer by adding an AVInputFormat struct.
You do not need to compile FFmpeg.
You must add logic code to the ff_ffplay.c file.
Copy the rtsdec.c, ali_net_api.h, rts_api.h, and rts_messages.h files of Native RTS SDK to the ijkplayer/android/contrib/ffmpeg-arm64/libavformat and ijkplayer/android/contrib/ffmpeg-armv7a/libavformat directories.
Modify the Makefile files and compile the rtsdec.c file.
Modify the ijkplayer/android/contrib/ffmpeg-arm64/libavformat/Makefile and ijkplayer/android/contrib/ffmpeg-armv7a/libavformat/Makefile files, as shown in the following figure. Then, compile the rtsdec.c file.
Modify the allformats.c files.
Modify the ijkplayer/android/contrib/ffmpeg-arm64/libavformat/allformats.c and ijkplayer/android/contrib/ffmpeg-armv7a/libavformat/allformats.c files, as shown in the following figure. By default, the ARTC protocol is supported.
extern AVInputFormat ff_rtc_demuxer; av_register_input_format(&ff_rtc_demuxer);Compile FFmpeg.
Add the Android NDK environment variable and run the ./compile-ffmpeg.sh all command in the ijkplayer/android/contrib directory. After you compile FFmpeg, make sure that an output file of FFmpeg compilation exists in the ijkplayer/android/contrib/build directory.
Add the dynamic frameworks of Native RTS SDK.
Copy the dynamic frameworks of Native RTS SDK to the ijkplayer/android/contrib/build/ffmpeg-arm64/output and ijkplayer/android/contrib/build/ffmpeg-armv7a/output directories.
Copy the header files rts_api.h and rts_messages.h of Native RTS SDK separately to the ijkplayer/android/contrib/build/ffmpeg-arm64/output/include and ijkplayer/android/contrib/build/ffmpeg-armv7a/output/include directories.
Import the dynamic frameworks of Native RTS SDK.
Modify the ijkplayer/android/ijkplayer/ijkplayer-armv7a/src/main/jni/ffmpeg/Android.mk and ijkplayer/android/ijkplayer/ijkplayer-arm64/src/main/jni/ffmpeg/Android.mk files.
include $(CLEAR_VARS) LOCAL_MODULE := rtssdk LOCAL_SRC_FILES := $(MY_APP_FFMPEG_OUTPUT_PATH)/libRtsSDK.so include $(PREBUILT_SHARED_LIBRARY)Modify the ijkplayer/ijkmedia/ijkplayer/Android.mk file by adding the dynamic frameworks of Native RTS SDK as dependencies.
Add the logic code of RTS to the ff_ffplay.c file.
Modify the ijkplayer/ijkmedia/ijkplayer/ff_ffplay.c file by configuring the AVInputFormat function pointer for ARTC.
extern AVInputFormat ff_rtc_demuxer; extern int artc_reload(AVFormatContext *ctx); extern void av_set_rts_demuxer_funcs(const struct rts_glue_funcs *funcs); extern void artc_set_rts_param(char* key, char* value); extern long long artc_get_state(AVFormatContext *ctx, int key); int version = 2; const struct rts_glue_funcs* rts_funcs = get_rts_funcs(version); // set to ffmpeg plugin av_set_rts_demuxer_funcs(rts_funcs); artc_set_rts_param((char*)"AutoReconnect", (char*)"false");Compile ijkplayer.
Run the ./compile-ijk.sh all command in the ijkplayer/android directory.
Add the dynamic frameworks of Native RTS SDK.
Copy the dynamic frameworks of Native RTS SDK to the ijkplayer/android/contrib/build/ffmpeg-arm64/output and ijkplayer/android/contrib/build/ffmpeg-armv7a/output directories.
Copy the header files rts_api.h and rts_messages.h of Native RTS SDK separately to the ijkplayer/android/contrib/build/ffmpeg-arm64/output/include and ijkplayer/android/contrib/build/ffmpeg-armv7a/output/include directories.
Import the dynamic frameworks of Native RTS SDK.
Modify the ijkplayer/android/ijkplayer/ijkplayer-armv7a/src/main/jni/ffmpeg/Android.mk and ijkplayer/android/ijkplayer/ijkplayer-arm64/src/main/jni/ffmpeg/Android.mk files.
include $(CLEAR_VARS) LOCAL_MODULE := rtssdk LOCAL_SRC_FILES := $(MY_APP_FFMPEG_OUTPUT_PATH)/libRtsSDK.so include $(PREBUILT_SHARED_LIBRARY)Modify the ijkplayer/ijkmedia/ijkplayer/Android.mk file by adding the dynamic frameworks of Native RTS SDK as dependencies.
Add the logic code of RTS to the ff_ffplay.c file.
Modify the ijkplayer/ijkmedia/ijkplayer/ff_ffplay.c file by configuring the AVInputFormat function pointer for ARTC.
if(strncmp(is->filename, "artc://", 7) == 0) { extern AVInputFormat ff_rtc_demuxer; extern int artc_reload(AVFormatContext *ctx); extern void av_set_rts_demuxer_funcs(const struct rts_glue_funcs *funcs); extern void artc_set_rts_param(char* key, char* value); extern long long artc_get_state(AVFormatContext *ctx, int key); int version = 2; const struct rts_glue_funcs* rts_funcs = get_rts_funcs(version); // set to ffmpeg plugin av_set_rts_demuxer_funcs(rts_funcs); artc_set_rts_param((char*)"AutoReconnect", (char*)"false"); is->iformat = &ff_rtc_demuxer; } else { if(ffp->iformat_name) is->iformat = av_find_input_format(ffp->iformat_name); }Copy the rtsdec.c file of Native RTS SDK to the ijkplayer/ijkmedia/ijkplayer directory. Then, modify the android.mk file in the directory by adding the compilation description of the rtsdec.c file.
LOCAL_SRC_FILES += rtsdec.c
Integrate ijkplayer into your project.
Copy the RtsNetSDK.jar file of Native RTS SDK to the project.
Import the dynamic frameworks of Native RTS SDK into the activity that uses RTS playback.
static { System.loadLibrary("RtsSDK"); }Import the ijkplayer-arm64, ijkplayer-armv7a, and ijkplayer-java modules.
Call the methods of ijkplayer to use the RTS feature.
Create ijkplayer
mIjkPlayer = new IjkMediaPlayer();Configure callbacks
mIjkPlayer.setOnPreparedListener(new IMediaPlayer.OnPreparedListener() { @Override public void onPrepared(IMediaPlayer iMediaPlayer) { } }); mIjkPlayer.setOnInfoListener(new IMediaPlayer.OnInfoListener() { @Override public boolean onInfo(IMediaPlayer iMediaPlayer, int arg1, int arg2) { }); mIjkPlayer.setOnVideoSizeChangedListener(new IMediaPlayer.OnVideoSizeChangedListener() { @Override public void onVideoSizeChanged(IMediaPlayer iMediaPlayer, int width, int height, int sarNum, int sarDen) { } }); mIjkPlayer.setOnErrorListener(new IMediaPlayer.OnErrorListener() { @Override public boolean onError(IMediaPlayer iMediaPlayer, int framework_err, int impl_err) { return false; } });Set the user interface (UI) view
mIjkPlayer.setSurface(surface);Set the playback source
mIjkPlayer.setDataSource("artc://<Streaming URL>");Set playback control
public void prepare() { if(mIjkPlayer != null){ mIjkPlayer.prepareAsync(); } } @Override public void start() { if(mIjkPlayer != null){ mIjkPlayer.start(); } } public void stop() { if(mIjkPlayer != null){ mIjkPlayer.stop(); } } public void release() { if(mIjkPlayer != null){ mIjkPlayer.release(); } }
Listen to RTS events
Listen to the message callback of Native RTS SDK in ijkplayer
Open the ijkplayer/android/ijkplayer project.
Modify the ff_ffplay.c file by inserting the following code block after the
static int audio_open(FFPlayer *opaque, int64_t wanted_channel_layout, int wanted_nb_channels, int wanted_sample_rate, struct AudioParams *audio_hw_params){}method.extern int artcDemuxerMessage(struct AVFormatContext *s, int type, void *data, size_t data_size); //aliyun rts:receive artc message int onArtcDemuxerMessage(struct AVFormatContext *s, int type, void *data, size_t data_size) { return artcDemuxerMessage(s, type, data, data_size); } int artcDemuxerMessage(struct AVFormatContext *s, int type, void *data, size_t data_size) { //aliyun rts:send message to app FFPlayer *ffp = (FFPlayer *)s->opaque; const char *data_msg = (const char *)data; ffp_notify_msg4(ffp,FFP_MSG_ARTC_DIRECTCOMPONENTMSG,type,0,data_msg,data_size); return 0; }Modify the ff_ffplay.c file by inserting the following RTS code block into the
static int is_realtime(AVFormatContext *s){}method.static int is_realtime(AVFormatContext *s) { if( !strcmp(s->iformat->name, "rtp") || !strcmp(s->iformat->name, "rtsp") || !strcmp(s->iformat->name, "sdp") // *** Beginning of the RTS code block *** || !strcmp(s->iformat->name, "artc") // *** End of the RTS code block *** ) return 1; if(s->pb && ( !strncmp(s->filename, "rtp:", 4) || !strncmp(s->filename, "udp:", 4) ) ) return 1; return 0; }Modify the ff_ffplay.c file by inserting the following RTS code block into the
static int read_thread(void *arg){}method.static int read_thread(void *arg) { ...... ic = avformat_alloc_context(); if (!ic) { av_log(NULL, AV_LOG_FATAL, "Could not allocate context.\n"); ret = AVERROR(ENOMEM); goto fail; } // *** Beginning of the RTS code block *** ic->opaque = ffp; ic->control_message_cb = onArtcDemuxerMessage; // *** End of the RTS code block *** ...... if (ffp->skip_calc_frame_rate) { av_dict_set_int(&ic->metadata, "skip-calc-frame-rate", ffp->skip_calc_frame_rate, 0); av_dict_set_int(&ffp->format_opts, "skip-calc-frame-rate", ffp->skip_calc_frame_rate, 0); } // *** Beginning of the RTS code block *** if(strncmp(is->filename, "artc://", 7) == 0) { extern AVInputFormat ff_rtc_demuxer; is->iformat = &ff_rtc_demuxer; } else { if(ffp->iformat_name) is->iformat = av_find_input_format(ffp->iformat_name); } // *** End of the RTS code block *** ...... pkt->flags = 0; // *** Beginning of the RTS code block *** if(strncmp(is->filename, "artc://", 7) == 0) { bool videoExist = is->video_stream >= 0; bool audioExist = is->audio_stream >= 0; // av_log(NULL, AV_LOG_INFO, "videoDuration %lld audioDuration %lld rate %f videoframeQue %d audioFrameque %d\n", // is->videoq.duration, is->audioq.duration, ffp->pf_playback_rate, // frame_queue_nb_remaining(&is->pictq), frame_queue_nb_remaining(&is->sampq)); if(!videoExist) { if(is->audioq.duration > 300 ) { // accelerate if(ffp->pf_playback_rate <= 1.0) { ffp->pf_playback_rate = 1.3; ffp->pf_playback_rate_changed = 1; av_log(NULL, AV_LOG_INFO, "aliyun rts set rate to %f\n", ffp->pf_playback_rate); } } else if(is->audioq.duration < 200) { // restore speed if(ffp->pf_playback_rate > 1.0) { ffp->pf_playback_rate = 1.0; ffp->pf_playback_rate_changed = 1; av_log(NULL, AV_LOG_INFO, "aliyun rts restore rate 1.0\n"); } } } else if((!videoExist || (videoExist && is->videoq.duration > 300)) && (!audioExist || (audioExist && is->audioq.duration > 300))) { if(ffp->pf_playback_rate <= 1) { ffp->pf_playback_rate = 1.3; ffp->pf_playback_rate_changed = 1; av_log(NULL, AV_LOG_INFO, "aliyun rts set rate 1.1\n"); } } else if((videoExist && is->videoq.duration <= 100) || (audioExist && is->audioq.duration <= 100)){ if(ffp->pf_playback_rate > 1) { ffp->pf_playback_rate = 1; ffp->pf_playback_rate_changed = 1; av_log(NULL, AV_LOG_INFO, "aliyun rts set rate 1\n"); } } } // *** End of the RTS code block *** ...... }Modify the ff_ffmsg.h file by adding the method declaration for RTS messages.
#define FFP_MSG_ARTC_DIRECTCOMPONENTMSG 3000Send RTS messages to the upper layer.
Modify the ijkplayer_android_def.h file by adding the RTS message enumeration type.
enum media_event_type { // Other code is omitted. MEDIA_ARTC_MESSAGE = 3000, // aliyun rts : msg info key };Modify the ijkplayer_jni.c file to send RTS messages to the upper layer.
static void message_loop_n(JNIEnv *env, IjkMediaPlayer *mp) { //... Other code is omitted. while (1) { switch (msg.what) { //aliyun rts: post artc event case FFP_MSG_ARTC_DIRECTCOMPONENTMSG: if (msg.obj) { const char * result = (const char *)msg.obj; ALOGE("aliyun rts : FFP_MSG_ARTC_DIRECTCOMPONENTMSG = %d , %s\n", msg.arg1,result); jstring data_msg = (*env)->NewStringUTF(env, result); post_event2(env, weak_thiz, MEDIA_ARTC_MESSAGE, msg.arg1, 0, data_msg); J4A_DeleteLocalRef__p(env, &data_msg); } else { post_event2(env, weak_thiz, MEDIA_ARTC_MESSAGE, 0, 0, NULL); } break; } } }Modify the IjkMediaPlayer.java file to receive RTS messages.
private static class EventHandler extends Handler { @Override public void handleMessage(Message msg) { IjkMediaPlayer player = mWeakPlayer.get(); if (player == null || player.mNativeMediaPlayer == 0) { DebugLog.w(TAG, "IjkMediaPlayer went away with unhandled events"); return; } switch (msg.what) { //aliyun artc:post event case MEDIA_ARTC_MESSAGE: if(msg.arg1 == FFP_ARTC_CONNECT_LOST){ player.notifyOnARTCMessage("NetWorkDisconnect",0,""); }else if(msg.arg1 == FFP_ARTC_RECOVERED){ player.notifyOnARTCMessage("NetWorkRetrySuccess",0,""); }else{ if(msg.obj == null){ player.notifyOnARTCMessage("DirectComponentMSG",0, ""); }else{ String result = (String) msg.obj; result = result.replaceAll("\"",""); player.mAliyunRTSMsgBuilder = new StringBuilder("{\"content\":"); player.mAliyunRTSMsgBuilder.append("\"").append(result).append("\"}"); player.notifyOnARTCMessage("DirectComponentMSG",0, (String) result); } } break; } } }Add the RTS message callback to the IMediaPlayer.java file.
//aliyun rts:artc message callback interface OnARTCMessageListener{ void onMessage(String name,int externValue,String extraMsg); }Add the listener settings to the AbstractMediaPlayer.java file.
//aliyun rts : set artc message listener public final void setOnARTCMessageListener(OnARTCMessageListener listener){ this.mOnARTCMessageListener = listener; } //aliyun rts : call back protected final void notifyOnARTCMessage(String name,int externValue,String extraMsg){ if(mOnARTCMessageListener != null){ mOnARTCMessageListener.onMessage(name,externValue,extraMsg); } }Run the ./compile-ijk.sh command for recompilation.
Add a method used to perform retries for Native RTS SDK in ijkplayer
Modify the ff_ffplay.h file by adding the definition of the rts_reload_flag variable.
int rts_reload_flag;Modify the ff_ffplay.c file by inserting the following code block after the
static int audio_open(FFPlayer *opaque, int64_t wanted_channel_layout, int wanted_nb_channels, int wanted_sample_rate, struct AudioParams *audio_hw_params){}method.extern int artc_reload(AVFormatContext *ctx); void ffp_rts_reload(FFPlayer *ffp){ if(rts_reload_flag == 0) { rts_reload_flag = 1; } }Modify the ff_ffplay.c file by inserting the following RTS code block into the
static int read_thread(void *arg){}method.static int read_thread(void *arg) { ...... #ifdef FFP_MERGE if (is->paused != is->last_paused) { is->last_paused = is->paused; if (is->paused) is->read_pause_return = av_read_pause(ic); else av_read_play(ic); } #endif // *** Beginning of the RTS code block *** if(rts_reload_flag){ rts_reload_flag = 0; av_log(ffp, AV_LOG_ERROR, "param == ffp_rts_reload\n"); VideoState *is = ffp->is; AVFormatContext *ic = is->ic; artc_reload(ic); } // *** End of the RTS code block *** ...... }Modify the ijkplayer.h file by adding the definition of the ijkmp_rts_reload method.
void ijkmp_rts_reload(IjkMediaPlayer *mp);Modify the ijkplayer.c file to implement the ijkmp_rts_reload method.
void ijkmp_rts_reload(IjkMediaPlayer *mp) { ffp_rts_reload(mp->ffplayer); }Add the rtsReload method to the upper layer.
Modify the ijkplayer_jni.c file by adding the reload method.
//aliyun rts : reload native static void IjkMediaPlayer_reload(JNIEnv *env,jobject thiz) { IjkMediaPlayer *mp = jni_get_media_player(env,thiz); ijkmp_rts_reload(mp); LABEL_RETURN: ijkmp_dec_ref_p(&mp); } static JNINativeMethod g_methods[] = { //aliyun rts: reload { "_reload", "()V", (void *) IjkMediaPlayer_reload }, }Modify the IMediaPlayer.java file by adding the reload method.
//aliyun rts: reload void reload();Modify the IjkMediaPlayer.java file to allow you to call the native reload method.
//aliyun rts : native method private native void _reload(); @Override public void reload() { _reload(); }
Call the method for listening to the RTS message callback to live stream over a degraded protocol
Listen to the RTS message callback.
mIjkPlayer.setOnARTCMessageListener(new IMediaPlayer.OnARTCMessageListener() { @Override public void onMessage(String name, int externValue, String extraMsg) {} });Implement the degradation logic in the method that is used to listen to the RTS message callback.
You can change the prefix of a source URL from artc:// to rtmp:// or http://xxxx.flv, and then update the UrlSource in ApsaraVideo Player. This way, you can use a degraded protocol to play live streams.
if (SOURCE_URL.startsWith("artc://")) { mIjkPlayer.stop();// Stop playback. mIjkPlayer.reset();// Reset ijkplayer. If you do not reset ijkplayer, a crash occurs when you reuse the player. mIjkPlayer.setDataSource("http://xxx.flv"); mIjkPlayer.prepareAsync(); }During start-up or live streaming, when a message that is passed through by the playback component is returned in a player event callback, parse the JSON string that describes the player event. The code that you obtain is the message in RTS SDK. When you receive E_DNS_FAIL, E_AUTH_FAIL, E_CONN_TIMEOUT, E_SUB_TIMEOUT, or E_SUB_NO_STREAM, you need to use a degraded protocol for live streaming. When you receive E_STREAM_BROKEN, you need to retry playback once. If you still receive E_STREAM_BROKEN, use a degraded protocol for live streaming. When you receive E_RECV_STOP_SIGNAL, you need to stop playback.
// Configure the class that enumerates RTS information. public enum RtsError { /** * DNS resolution failed. */ E_DNS_FAIL(20001), /** * Authentication failed. */ E_AUTH_FAIL(20002), /** * Connection signaling timed out. */ E_CONN_TIMEOUT(20011), /** * An error occurred during subscription signaling, or subscription signaling timed out. */ E_SUB_TIMEOUT(20012), /** * The stream to which you subscribe does not exist. */ E_SUB_NO_STREAM(20013), /** * The media timed out and no audio and video packets were received. */ E_STREAM_BROKEN(20052), /** * The stop signaling message was received. */ E_RECV_STOP_SIGNAL(20061); private int code; RtsError(int code) { this.code = code; } public int getCode() { return code; } public void setCode(int code) { this.code = code; } } mIjkPlayer.setOnARTCMessageListener(new IMediaPlayer.OnARTCMessageListener() { @Override public void onMessage(String name, int externValue, String msg) { if("DirectComponentMSG".equals(name)){ if (msg.contains("code=" + RtsError.E_DNS_FAIL.getCode()) || msg.contains("code=" + RtsError.E_AUTH_FAIL.getCode()) || msg.contains("code=" + RtsError.E_CONN_TIMEOUT.getCode()) || msg.contains("code=" + RtsError.E_SUB_TIMEOUT.getCode()) || msg.contains("code=" + RtsError.E_SUB_NO_STREAM.getCode()) || msg.contains("code=" + RtsError.E_STREAM_BROKEN.getCode()) || msg.contains("code=" + RtsError.E_RECV_STOP_SIGNAL.getCode())) { // TODO Execute the degradation logic. } } } });