Xianyu Flutter picture framework architecture evolution

1. Those years
Pictures are a commonplace topic for an end-to-end R&D. Xianyu is one of the first teams in the industry to invest in Flutter technology. From the very beginning of using Flutter, images have been our core focus and key optimization feature. Pictures are one of the most important forms of content expression in the Xianyu business scenario. The quality of the picture display experience will have a huge impact on the experience of Xianyu users. Have you also encountered:

Image loading takes up too much memory?
After using flutter, the local resources are repeated and the utilization rate is not high?
Flutter's native image loading efficiency is not high under the hybrid scheme?
In response to the above problems, since the launch of the first version of the Flutter business, Xianyu has never stopped optimizing the picture frame. From the original optimization at the beginning, to the external texture of the black technology behind; from the memory usage to the package size; the text will be introduced one by one. I hope that the optimization ideas and methods among them can bring you some inspiration.

2. Native mode
Looking at image loading from a technical perspective, in fact, in simple terms, the pursuit is nothing more than maximizing the efficiency of loading—loading as many images as possible as quickly as possible with as little resource cost as possible.

The first version of Xianyu Pictures is basically a pure native solution. If you don’t want to magically change a lot of underlying logic, the native solution is definitely the easiest and cheapest solution. The functional modules of the native solution are as follows:

Architecture diagram of native image scheme

If you go straight to it without doing anything, then you may find that the effect is not as good as you expected. So if we start with the native solution, what specific optimization methods do we have?

2.1. Set image cache
Yes, you guessed it, the cache. For image loading, the most conceivable solution is to use cache. First, the native Image component supports custom image caching, and the specific implementation class is ImageCache. The setting dimension of ImageCache is two directions: 1. The number of cached pictures. Set by maximumSize. The default is 1000 sheets. 2. The size of the cache space. Set by maximumSizeBytes. The default value is 100M. Compared with the limit of the number of sheets, the size setting method is more in line with our final expectations.

By setting the size of ImageCache reasonably, the caching mechanism can be fully utilized to speed up image loading. Not only that, Xianyu also made two additional important optimizations at this point:

Low-end mobile phone adaptation
After the launch, we received feedback from online public opinion one after another, and found that it is not optimal to set the same cache size for all models. Especially if the large cache is set on low-end machines, not only will the experience become worse, but it will even affect the stability. Based on the actual situation, we implemented a Flutter plugin that can obtain basic machine information from the Native side. Through the obtained information, we set different caching strategies according to the configuration of different mobile phones. Reduce the size of the image cache appropriately on low-end machines, and enlarge it appropriately on high-end mobile phones. This can get the best cache performance on phones with different configurations.

disk cache
Students who are familiar with APP development know that mature image loading frameworks generally have multi-level caches. In addition to the common memory cache, a file cache is generally configured. In terms of loading efficiency, the loading speed is improved by changing space for time. In terms of stability, this will not take up too much precious memory resources and cause OOM. But unfortunately, the image loading framework that comes with Flutter does not have an independent disk cache. So we expanded the disk caching capability based on the native solution.

In terms of specific architecture implementation, we have not completely built a disk cache by ourselves. Our strategy is to reuse existing capabilities. First, we expose the disk cache function of the Native image loading framework through the interface. Then, by bridging, the native disk caching capability is grafted to the Flutter layer. When the image is loaded on the Flutter side, if the memory is not hit, it will go to the disk cache for a second search. If there is no hit, the network request will be taken.

By increasing the disk cache, Flutter's image loading efficiency is further improved.

multilevel cache

2.2. Set up CDN optimization
CDN optimization is another very important image optimization method. The efficiency improvement of CDN optimization is mainly: minimizing the size of the transmitted image. Common strategies include:

Crop to display size
Simply put, the actual size of the image you want to load may be larger than the size of your actual display window. Then you don't need to load the full large image, you just need to load an image that covers the size of the window. In this way, the size of the transmitted image can be minimized by cropping out the unneeded parts. From an end-to-end perspective, it can improve the loading speed and reduce memory usage.

Appropriately compress the image size
Here is mainly to increase the ratio of image compression according to the actual situation. The size of the picture is further reduced by compression without affecting the display effect.

Image Format
It is recommended to use the webp format first, and the image resources are relatively small. Flutter natively supports webp (including animation). It is especially emphasized here that the webp animation is not only much smaller in size than gif, but also has better support for transparent effects. The webp animation is an ideal alternative to the gif scheme.

Demonstrate with pictures

Based on the above reasons, the Xianyu picture framework implements a set of CDN size matching algorithms on the Flutter side. Through this algorithm, the requested image will be automatically matched to the most appropriate size and properly compressed according to the actual display size. If the image format allows, the image should be converted into webp format as much as possible. In this way, the transmission of cdn pictures can be as efficient as possible.

2.3. Other optimizations
In addition to the above strategies, Flutter has some other means to optimize the performance of images.

Image preload
If you want to display images as fast as possible, the official also provides a preloading mechanism: precacheImage. precacheImage can pre-load the image into memory, and it can be released in seconds when it is actually used.

Element reuse optimization
In fact, this is a general optimization scheme for Flutter. Rewrite the didWidgetUpdate scheme, and decide whether to re-render the Element by comparing whether the descriptions of the images in the two widgets before and after are consistent. This avoids unnecessary re-rendering of the same image.

Long list optimization
In general, Listview is the most common scrolling container for flutter. The performance in Listview directly affects the final user experience.

Flutter's Listview is different from that of Native. Its biggest feature is the concept of a viewPort. The part beyond the viewPort will be forcibly recycled.

Based on the above principles, we have two suggestions:

cell split
Try to avoid the appearance of large cells, which can greatly reduce the performance loss during the frequent creation of cells. In fact, it is not only the image loading process that is affected here. Text, video and other components should also avoid performance problems caused by overly complex cells.

Use buffers wisely
ListView can set the preloaded content size by setting cacheExtent. View rendering speed can be improved by preloading. But this value needs to be set reasonably, not the bigger the better. Because the larger the preload cache, the greater the pressure on the overall memory of the page.

2.4. Weaknesses of the program
It needs to be pointed out objectively here: if it is a pure Flutter APP, the native solution is perfect and sufficient. However, from the perspective of hybrid APP, there are the following two defects:

1. Unable to reuse the native image loading capability

There is no doubt that the native image scheme is a completely independent image loading scheme. For a hybrid APP, the native solution and the native image framework are independent of each other, and the capabilities cannot be reused. Capabilities such as CDN cropping & compression require repeated construction. Especially some of the unique picture decoding capabilities of Native, Flutter is difficult to use. This will result in inconsistent support for image formats within the scope of the APP.

2. Insufficient memory performance

From the perspective of the entire APP, in the case of using the native image solution, we actually maintain two large cache pools: one is the native image cache, and the other is the image cache on the Flutter side. The two caches cannot communicate with each other, which is undoubtedly a huge waste. In particular, the peak memory performance of the memory is very stressful.

3. Get through Native
After several rounds of optimization, the native-based solution has achieved a very large performance improvement. However, the memory water level of the entire APP is still relatively high (especially on the Ios side). The pressure of reality forces us to continue to optimize the picture frame more deeply. Based on the analysis of the shortcomings of the above native solutions, we have a bold idea: Can we fully reuse the native image loading capabilities?

3.1. External textures
How to get through the picture capabilities of Flutter and Native? We thought of external textures. External texture is not Flutter's own technology, it is a performance optimization method commonly used in the audio and video field.

At this stage, we implemented the texture connection of Flutter and Native based on the shared-Context solution. Through this solution, Flutter can get the loaded images from the Native image library and display them by sharing textures. In order to implement this texture sharing channel, we have deeply customized the engine layer. The detailed process is as follows:



This solution not only opens up the image architecture of Native and Flutter, but also optimizes the performance of image loading throughout the process. Students who want to know the details can continue to read this article: Unexpectedly - Flutter External Texture.

The external texture is a big leap in Xianyu's picture scheme. Through this technology, we not only realize the local capability multiplexing of the picture scheme, but also realize the texture externalization of the video capability. This avoids a lot of repetitive construction and improves the performance of the entire app.

3.2. Multi-page memory optimization
This optimization strategy is really forced out. After analyzing the online data, we found that the Flutter page stack has a very interesting feature:

In the case of a multi-page stack, the underlying page will not be released. Even when memory is very tight, no collection will be performed. This leads to a problem: as the number of pages increases, the memory consumption grows linearly. The highest proportion here is the proportion of image resources.

Is it possible to directly recycle the pictures in the page when the page is at the bottom of the page stack?

Driven by this idea, we carried out a new round of optimization of the image architecture. The pictures in the entire picture frame will listen for changes in the page stack. When the party finds that it is already at the top of the stack, it automatically recycles the corresponding image texture to release resources. This solution prevents the memory size occupied by images from increasing linearly as the number of pages increases. The principle is as follows:

TB1mF0NbZKfxu4jSZPfXXb3dXXa-780-558.png

It should be noted that the page judgment position at this stage actually requires the page stack (specifically, the hybrid stack) to provide additional interfaces to achieve. The coupling between systems is relatively high.

3.3. Windfall packet size
After getting through the native and Flutter side image frameworks, we found an unexpected gain: Native and Flutter can share local image resources. That is to say, we no longer need to keep the same image resources on the Flutter and Native sides. This can greatly improve the reuse rate of local resources, thereby reducing the overall packet size. Based on this solution, we have implemented a set of resource management functions, and the script can automatically synchronize local image resources on different sides. By doing so, the local resource utilization is improved and the packet size is reduced.

3.4. Other optimizations
PlaceHolderEnhanced
The native Image has no PlaceHolder function. If you want to use the native solution, you need to use FadeInImage. We have a lot of customization for the scene of Xianyu, so we implemented a mechanism of PlaceHolder ourselves.

In terms of core functions, we introduce the concept of loading state into: 1. Uninitialized 2. Loading 3. Loading complete and so on. For different states, the display logic of PlaceHolder can be controlled in fine-grained manner.

3.5. Overall Architecture


3.6. Weaknesses of the scheme
After all, the engine has been changed
With the continuous advancement of Xianyu's business, the cost of engine upgrades is something we must consider. Whether we can achieve the same function without changing the engine is our core requirement. (PS: I admit we are greedy)

There is still room for optimization in channel performance
The external texture scheme needs to communicate with the native capabilities through a bridge. This includes the delivery of image requests and the synchronization of various states of image loading. Especially when the listview slides quickly, the amount of data sent through the bridge is still considerable. In the current scheme, the bridge is called separately when each image is loaded. In the case of a large number of pictures, this will obviously be a bottleneck.

Too much coupling
When implementing the image recycling scheme, the current scheme requires the stack to provide an interface at the bottom of the stack. Here comes the scheme coupling, and it is difficult to abstract an independent and clean image loading scheme.

4. Clean&Efficient
The time has come to 2020. With the gradual deepening of the understanding of the basic capabilities of Flutter, we have implemented a picture framework with a better overall solution.

4.1. Non-intrusive external textures
Can the external texture be modified without modifying the engine? The answer is yes.

In fact, Flutter provides an official external texture scheme.

Moreover, the texture of the Native operation and the texture displayed on the Flutter side are the same object at the bottom, and no additional data copy is generated. This ensures that texture sharing is efficient enough. So why did Xianyu implement a set based on shared-Context alone before? Before version 1.12, the external texture scheme of the official Ios had performance problems. During each rendering process (regardless of whether the texture is updated or not), the CVPixelBuffer will be frequently acquired, causing unnecessary performance loss (the process has locking loss). This problem has been fixed in version 1.12 (official commit address), so the official solution is enough to meet the needs.

In this context, we re-enable the official solution to implement the external texture function.

4.2. Independent memory optimization
As mentioned before, the old version of the page stack-based image resource recycling requires an interface that strongly relies on the stack function. On the one hand, unnecessary dependencies are generated, and more importantly, the overall solution cannot be independent into a general solution. To solve this problem, we conducted in-depth research on the bottom layer of Flutter. We found that Flutter's layer layer can stably perceive changes in the page stack.


Then each page uses the router object obtained from the context as an identifier to reorganize all the picture objects in a page. All the identifiers that get the same router object are the same page. In this way, all images can be managed on a page-by-page basis. The virtual page stack structure is simulated by the LRU algorithm as a whole. In this way, the image resources of the bottom page of the stack can be recycled.

4.3. Other optimizations
1. High multiplexing of channels

First, we aggregate the image requests in this frame by one frame, and then pass it to the Native image loading framework in one channel request. This avoids frequent bridge calls. The optimization effect is especially obvious in scenarios such as fast scrolling.

2. Efficient texture reuse

After using external textures for image loading, we found that multiplexing textures can further improve performance. Take a simple scenario. We know that in the e-commerce scene, there are often labels and pictures such as base maps in the product display. Such images tend to be heavily duplicated across different products. At this time, the rendered texture can be directly reused to different display components. This can further optimize GPU memory usage and avoid duplicate creation. In order to manage textures accurately, we introduce a reference counting algorithm to manage the reuse of textures. Through these schemes, we achieve efficient reuse of textures across pages.

Additionally, we moved the mapping between textures and requests to the Flutter side. In this way, the texture multiplexing can be completed on the shortest path, which further reduces the communication pressure of the bridge.

4.5. Overall Architecture


5. Optimize the effect
Since the latest version is still in grayscale, the specific data will be written and introduced to you in detail later. Subordinate data is mainly based on plan two.

memory optimization

By opening up Native, compared to the first online version, the abort rate of Ios is reduced by 25% while the display effect remains unchanged, and the user experience is significantly improved.
Multi-page stack memory optimization
The memory optimization of multi-page stack has obvious effect on memory optimization in multi-page scenarios. We did a limit test and the effect is as follows: (test environment, non-free fish APP)

​ It can be seen that the optimization of the multi-page stack can better control the memory usage of multiple Flutter pages.

Packet size reduction
By accessing external textures, local resources are better reused, and the package size is reduced by 1M. Early Xianyu access to Flutter will take the transformation of existing pages as the entry point. The duplication of resources is serious, but with the new business of Xianyu Flutter more and more. There are fewer and fewer duplicate resources for Flutter and Native. The effect of external textures on package size has gradually become weaker.

6. Summary
This article introduces the continuous optimization made by Xianyu in the direction of the Flutter picture frame. The details of typical picture technical solutions in different periods of Xianyu are introduced. Hope it can give readers some inspiration. This is an endless journey, and our optimization of Xianyu pictures will continue. In particular, our latest solution is limited in space, and this article is only a preliminary introduction. For more technical details, including test data, we will write a special article to introduce it to you later. After the program is perfected, we will gradually open source it.