Android dynamically loads so!

background


For an ordinary android application, the proportion of the so library is usually very high, because we inevitably encounter various needs to use native in the development, so the so library is dynamic It can reduce a lot of package size. It has been two years since Tencent's bugly team published an article about dynamic so in 2020. After two years of testing, the dynamic loading of so is actually a very mature one. It is a technology, but unfortunately, many companies have not yet covered this aspect or do not know where to start, because so dynamic actually involves downloading, so version management, dynamic loading implementation and many other aspects, we might as well throw Open these extra things and start from the most essential so dynamic loading! Here is this example, I named it sillyboy, welcome pr and follow-up likes!
so dynamic loading introduction
Dynamic loading is actually the process of removing our so library when it is packaged into apk, and downloading it through the network package at the appropriate time. This involves the downloader, as well as the version management after downloading, etc. to ensure that a so library is loaded correctly. Here, we will not discuss these auxiliary processes. Let's take a look at how to implement the simplest loading process.

Start with an example
We build a native project, and then program the following content in it, the following is cmake
# For more information about using CMake with Android Studio, read the
# documentation: https://d.android.com/studio/projects/add-native-code.html

# Sets the minimum version of CMake required to build the native library.

cmake_minimum_required(VERSION 3.18.1)

# Declares and names the project.

project("nativecpp")

# Creates and names a library, sets it as either STATIC
# or SHARED, and provides the relative paths to its source code.
# You can define multiple libraries, and CMake builds them for you.
# Gradle automatically packages shared libraries with your APK.

add_library( # Sets the name of the library.
nativecpp

# Sets the library as a shared library.
SHARED

# Provides a relative path to your source file(s).
native-lib.cpp)

add_library(
nativecpptwo
SHARED
test.cpp

)

# Searches for a specified prebuilt library and stores the path as a
# variable. Because CMake includes system libraries in the search path by
# default, you only need to specify the name of the public NDK library
# you want to add. CMake verifies that the library exists before
# completing its build.

find_library( # Sets the name of the path variable.
log-lib

# Specifies the name of the NDK library that
# you want CMake to locate.
log)

# Specifies libraries CMake should link to your target library. You
# can link multiple libraries, such as libraries you define in this
# build script, prebuilt third-party libraries, or system libraries.

target_link_libraries( # Specifies the target library.
nativecpp

# Links the target library to the log library
# included in the NDK.
${log-lib})


target_link_libraries( # Specifies the target library.
nativecpptwo

# Links the target library to the log library
# included in the NDK.
nativecpp
${log-lib})

As you can see, we have generated two so libraries, one is nativecpp, and the other is nativecpptwo (why two? We can continue to read below)
The most critical test.cpp code is also given here


#include
#include
#include


extern "C"
JNIEXPORT void JNICALL
Java_com_example_nativecpp_MainActivity_clickTest(JNIEnv *env, jobject thiz) {
// print a sentence here
__android_log_print(ANDROID_LOG_INFO,"hello","native layer method");

}

Very simple, just a native method, just print a log, we can call the method in the java/kotin layer, that is
public native void clickTest();

so library retrieval and deletion
To realize the dynamic loading of so, at least you need to know which so is involved in the process of this project! Don't worry, when we build gradle, we have already provided the corresponding build process, that is, the built task [
mergeDebugNativeLibs], in this process, all the native libraries in a project will be collected in a process, followed by task [stripDebugDebugSymbols] is a symbol table clearing process, if you understand native development, it is easy to know that this is a A process of reducing the volume of so, we will not detail here. So we can easily think that we only need to insert a custom task in these two tasks to traverse and delete so that we can delete so, so it is easy to write such code

ext {
deleteSoName = ["libnativecpptwo.so","libnativecpp.so"]
}
// This is one of the tasks executed in the configuration stage in the initialization-configuration-execution stage. AfterEvaluate is completed, all tasks can be obtained, so that our customized data can be inserted into it.
task(dynamicSo) {
}.doLast {
println("dynamicSo insert!!!! ")
//projectDir is under which project, projectDir is the path
print(getRootProject().findAll())

def file = new File("${projectDir}/build/intermediates/merged_native_libs/debug/out/lib")
// delete all so libraries by default
if (file.exists()) {
file.listFiles().each {
if (it.isDirectory()) {
it.listFiles().each {
target ->
print("file ${target.name}")
def compareName = target.name
deleteSoName.each {
if (compareName.contains(it)) {
target.delete()
}
}
}
}
}
} else {
print("nil")
}
}
afterEvaluate {
print("dynamicSo task start")
def customer = tasks.findByName("dynamicSo")
def merge = tasks.findByName("mergeDebugNativeLibs")
def strip = tasks.findByName("stripDebugDebugSymbols")
if (merge != null || strip != null) {
customer.mustRunAfter(merge)
strip.dependsOn(customer)
}

}

As you can see, we have defined a custom task dynamicSo, whose execution is defined in afterEvaluate and depends on mergeDebugNativeLibs, while stripDebugDebugSymbols depends on our generated dynamicSo to achieve an insertion operation. So why execute it in afterEvaluate? That is because the android plugin generates tasks such as mergeDebugNativeLibs during the configuration phase. The original gradle build does not have such a task, so we only need to insert it after configuring all tasks. We can take a look at gradle life cycle

Through the conditional retrieval, we delete the so we want, namely ibnativecpptwo.so and libnativecpp.so.
dynamically load so
According to the two so retrieved above, we can upload it to our own backend in the project, and then download it to the user's mobile phone through the network. Here we will demonstrate it, and we will put it directly under the data directory. Bar

In the real project process, there should be verification operations, such as md5 verification or decompression, etc. This is not the point, we will just skip it!
So, how to load a so library into our original apk? Here is the original loading process of so, you can see that the system is checked by the classloaderWhether there is a so library in the native directory for loading, then let's reflect and add our custom path to it, can we? The same idea as tinker is adopted here, and the retrieval path of so can be added to our classloader, for example
private static final class V25 {
private static void install(ClassLoader classLoader, File folder) throws Throwable {
final Field pathListField = ShareReflectUtil.findField(classLoader, "pathList");
final Object dexPathList = pathListField.get(classLoader);

final Field nativeLibraryDirectories = ShareReflectUtil.findField(dexPathList, "nativeLibraryDirectories");

List origLibDirs = (List) nativeLibraryDirectories.get(dexPathList);
if (origLibDirs == null) {
origLibDirs = new ArrayList<>(2);
}
final Iterator libDirIt = origLibDirs.iterator();
while (libDirIt.hasNext()) {
final File libDir = libDirIt.next();
if (folder.equals(libDir)) {
libDirIt.remove();
break;
}
}
origLibDirs.add(0, folder);

final Field systemNativeLibraryDirectories = ShareReflectUtil.findField(dexPathList, "systemNativeLibraryDirectories");
List origSystemLibDirs = (List) systemNativeLibraryDirectories.get(dexPathList);
if (origSystemLibDirs == null) {
origSystemLibDirs = new ArrayList<>(2);
}

final List newLibDirs = new ArrayList<>(origLibDirs.size() + origSystemLibDirs.size() + 1);
newLibDirs.addAll(origLibDirs);
newLibDirs.addAll(origSystemLibDirs);

final Method makeElements = ShareReflectUtil.findMethod(dexPathList, "makePathElements", List.class);

final Object[] elements = (Object[]) makeElements.invoke(dexPathList, newLibDirs);

final Field nativeLibraryPathElements = ShareReflectUtil.findField(dexPathList, "nativeLibraryPathElements");
nativeLibraryPathElements.set(dexPathList, elements);
}
}

In the original retrieval path, we added our retrieval path at the front, that is, the position where the array is 0, so that the classloader can find it when looking for the so library that we have made dynamic!
finished?
The general so library, such as when it does not depend on other so, can be loaded directly in this way, but if there is a dependent so library, it will not work! I believe you can see it when you read other blogs because of the Namespace problem. Specifically, in the process of loading our dynamic library, if we need to rely on other dynamic libraries, then we need a linking process, right! The implementation here is Linker. The path retrieved in Linker is bound by the system through the Namespace mechanism after the ClassLoader instance is created. When we inject a new path, although the path in the ClassLoader has increased, the Namespace in the Linker has been bound. The path set is not updated synchronously, so the libxxx.so file (the current so) can be found, but the dependent so cannot be found. bugly article
Many implementations use Tinker's implementation. Since our system's classloader is like this, we can replace this when appropriate! Of course, this is what the bugly team did, but the author believes that replacing a classloader is obviously too expensive for a common application, and the compatibility risk is quite high. Of course, there are many ways, such as using the Relinker library. Customize the logic we load.
In order not to fry cold rice, hehe, although I also like to eat fried rice (manual dog head), here we do not use the method of replacing the classloader, but use the idea of ​​​​relinker to load! Specifically, you can see the implementation of silentboy. In fact, it does not depend on relinker and tinker, because I copied the key, hahaha, okay, let's see how to implement it! But before that, we need to know some pre-knowledge
ELF file
Our so library is essentially an elf file, so the so library also conforms to the format of the elf file. The ELF file consists of 4 parts, namely the ELF header (ELF header), the program header table (Program header table), and the section (Section) and Section header table. In fact, a file does not necessarily contain all the content, and their positions are not necessarily arranged as shown. Only the position of the ELF header is fixed, and the position and size of the other parts are determined by the values ​​of the ELF header. to decide.

Then in our so, if it depends on other so, where does this information exist! ? Yes, it actually exists in the elf file, otherwise how can the linker find it, it actually exists in the .dynamic segment, so we just need to find the offset of the dynamic segment, and then we can get to the dynamic, and the dependent so The information is actually there.
We can use readelf (after there is a toolchains directory in ndk) to check, readelf -d nativecpptwo.so The -d here means to check the dynamic segment

This involves the knowledge of dynamic loading of so. I can recommend a book called Programmer's Self-Cultivation - Link Loading and Libraries. Here is a sketch

Let's look at the essence again. The dynamic structure is as follows, defined in elf.h
typedef struct{
Elf32_Sword d_tag;
union{
Elf32_Addr d_ptr;
....
}
}

When the value of d_tag ​​is DT_NEEDED, it represents the dependent shared object file, and d_ptr represents the file name of the dependent shared object. Seeing this, readers already know that if we know the file name, we can use System.loadLibrary to load the so determined by the file name! Without replacing the classloader, you can ensure that the dependent library is loaded first! We can summarize the principle of this scheme again, as shown in Fig.

For example, if we want to load so3, we need to load so2 first. If so2 has dependencies, then we call System.loadLibrary to load so1 first. At this time, so1 has no dependencies, so there is no need to call Linker to find other so libraries. . Our final solution is that as long as we can parse the corresponding elf file, then find the offset, and find the value corresponding to the desired target item (DT_NEED) (ie the dependent so file name)
public List parseNeededDependencies() throws IOException {
channel.position(0);
final List dependencies = new ArrayList();
final Header header = parseHeader();
final ByteBuffer buffer = ByteBuffer.allocate(8);
buffer.order(header.bigEndian ? ByteOrder.BIG_ENDIAN : ByteOrder.LITTLE_ENDIAN);

long numProgramHeaderEntries = header.phnum;
if (numProgramHeaderEntries == 0xFFFF) {
/**
* Extended Numbering
*
* If the real number of program header table entries is larger than
* or equal to PN_XNUM(0xffff), it is set to sh_info field of the
* section header at index 0, and PN_XNUM is set to e_phnum
* field. Otherwise, the section header at index 0 is zero
* initialized, if it exists.
**/
final SectionHeader sectionHeader = header.getSectionHeader(0);
numProgramHeaderEntries = sectionHeader.info;
}

long dynamicSectionOff = 0;
for (long i = 0; i < numProgramHeaderEntries; ++i) {
final ProgramHeader programHeader = header.getProgramHeader(i);
if (programHeader.type == ProgramHeader.PT_DYNAMIC) {
dynamicSectionOff = programHeader.offset;
break;
}
}

if (dynamicSectionOff == 0) {
// No dynamic linking info, nothing to load
return Collections.unmodifiableList(dependencies);
}int i = 0;
final List neededOffsets = new ArrayList();
long vStringTableOff = 0;
DynamicStructure dynStructure;
do {
dynStructure = header.getDynamicStructure(dynamicSectionOff, i);
if (dynStructure.tag == DynamicStructure.DT_NEEDED) {
neededOffsets.add(dynStructure.val);
} else if (dynStructure.tag == DynamicStructure.DT_STRTAB) {
vStringTableOff = dynStructure.val; // d_ptr union
}
++i;
} while (dynStructure.tag != DynamicStructure.DT_NULL);

if (vStringTableOff == 0) {
throw new IllegalStateException("String table offset not found!");
}

// Map to file offset
final long stringTableOff = offsetFromVma(header, numProgramHeaderEntries, vStringTableOff);
for (final Long strOff : neededOffsets) {
dependencies.add(readString(buffer, stringTableOff + strOff));
}

return dependencies;
}

expand
When we get here, we can solve the problems related to the dynamic loading of the so library. Then some people may ask, there will be multiple System.load methods in the project. What if the loaded so does not exist yet? For example, it is still in the process of downloading. It is actually very simple. At this time, our bytecode instrumentation will come in handy. As long as we replace System.load with our custom loading so logic, we can perform certain logic processing, hehe , because the author has written an introduction to a bytecode instrumentation library before, so I won't repeat it this time. You can see Sipder, and you can also use other bytecode instrumentation frameworks to implement it. I believe this is not a problem. .
Summarize
Readers here, I believe they can understand the steps of dynamically loading so, and finally the source code can be found in SillyBoy, of course, I hope you all like it! Of course, there are better implementations and comments are welcome! !

Related Articles

Explore More Special Offers

  1. Short Message Service(SMS) & Mail Service

    50,000 email package starts as low as USD 1.99, 120 short messages start at only USD 1.00