Imgcook 3.0 Series: How Does Design-based Code Generation Recognize Icons?

By Jinzhong from F(x) Team of Taobao Technology Department, Alibaba Group

Background

The design and display of the front pages cannot be separated from pictures. In imgcook, icons of many pictures are a very important part. An icon is often composed of simple lines and has a distinct meaning, so it is widely used in the expression of common action points, such as home page, search, settings, return, and users. In addition, icons can convey the meaning in the text vividly by its highly standardized design.

In the comprehensive procedure of imgcook, about 6,156 icons are detected every month on average and about 2.67 icons are used for every new module. Therefore, to some extent, knowing which icons are used in the design helps to make clear the semantics within the scope of the icon. Thus, this helps imgcook achieve more intelligent judgments. In this case, icon semantization is very important.

Analysis

In the previous imgcook procedure, the F(x) Team used resnet to classify and identify more than 80 icons and extract their semantic information. Moreover, the team also clustered similar pictures as a group to predict the semantics of icons based on the similarity. However, the result of online verification is not particularly satisfactory.

First of all, there are many types and large numbers of icons. The effect of direct model training is not particularly ideal. Therefore, scientific classification and sample management are needed to improve the quality.

Secondly, although most icons follow certain design principles such as simplicity and intuition, icon design is still a very creative process. The style of icons is constantly upgrading. Therefore, data and model with wider coverage and stronger generalization capability are needed.

To sum up, the identification and semantization of icons are indispensable parts of the entire imgcook procedure. The optimization of icon identification not only requires to optimizing the model, but also focusing on data sources and iteration. To achieve the auto iteration, the icon identification model has been upgraded from level 3 (L3) to level 4 (L4).

(Levels of Icon Identification Capabilities)

This article will describe how imgcook optimizes data and model by building a complete closed-loop procedure for the icon model.

Technical Solutions

Data Procedure

Data is the most important and consistent part of the closed-loop procedure for the icon identification model. If the data cannot update and iterate by itself, the procedure is rather an experiment about model change than an engineered procedure. In the closed-loop procedure of imgcook, two data collection schemes are provided to ensure that the data is up-to-date and can supplement by itself.

Manual addition of data.
Automated collection of icons via imgcook procedure.

If your data is already categorized, you can directly import the data via the panel:

Manually Importing Data in Batches

In addition, the icon automatically extracted by imgcook is directly collected and automatically labeled:

Automatically Labeled Data

By doing so, the background data set keeps collecting newly designed and categorized icons, thus expanding the coverage of the icon data set.

For automatically collected icons, the system will perform a prediction. If the prediction succeeds, the system automatically adds labels to the icons. If the prediction fails, the system notifies maintainers to add labels manually. Maintainers can label these icons with a simple click.

After the labeling, the system sends the data to OSS for model training then.

Model Procedure

As the training data is updated in real time, the training layer of the model only needs to synchronize the data in OSS and labeled data to generate training samples. After data synchronization and image data enhancement, the archive files can be directly imported to pipcook for training. After the training, the current model will be accurately compared with the online model for grey release. After the grey release, the new model will replace the old one and enter into the main procedure of imgcook.

Closed Loop of the Icon Model

Implementation Results

The overall quality of icon identification has been improved after the new model procedure is deployed.

Data Results

Firstly, as the icons in the procedure have been strictly cleaned, deduplicated, and normalized, the bad sample rate has been greatly reduced. Since the new procedure was launched, the total number of new training sets has reached 2,729, and the average number of data retrieved monthly has reached 909. The number of categories has also increased from 97 to 105.

Model Results

The following table shows the model comparison after two times of iteration. The accuracy of model test sets has reached 83%.

	Icon quantity	Model accuracy
The first iteration in August	14,912	80.152%
The fifth iteration in November	17,540	83.035%

The online identification results, including the accuracy of bound class names have also been improved to some extent.

Outlook

The core idea of this system is to continuously optimize the model by constantly supplementing data and ensuring data reliability. The data procedure is the core of the entire closed-loop procedure for the icon identification model. However, optimizations of the entire process of icon identification is not only restricted in the data layer. In the future, the F(x) team will make further studies on a more intelligent way to filter and clean the data and on more suitable models for training predictions.

Community

Imgcook 3.0 Series: How Does Design-based Code Generation Recognize Icons?

Background

Analysis

Technical Solutions

Data Procedure

Model Procedure

Implementation Results

Data Results

Model Results

Outlook

Read previous post:

Read next post:

Alibaba F(x) Team

You may also like

Comments

Alibaba F(x) Team

Related Products

AIRec

Artificial Intelligence Service for Conversational Chatbots Solution

Alibaba Cloud Model Studio

Log Management for AIOps Solution