Nanjing New Generation Artificial Intelligence Research Institute Sun Mingjun: AI chip evaluation is not a one-day effort, requiring AI enterprises to participate more-Alibaba Cloud Developer Community

Currently, DNN benchmark V0.5 evaluation tool is open-source on Github.

How popular is the domestic chip industry?

One of the most obvious features is that more and more emerging technology companies have launched dedicated chips suitable for special scenarios. However, behind the chip "blowout", there are also many problems hidden.

Recently, make.com interviewed Sun Mingjun, deputy director of artificial intelligence department of cloud Institute of China Information and Communication Research Institute, head of general group of China Artificial Intelligence Industry Development Alliance (AIIA), president of Nanjing New Generation Artificial Intelligence Research Institute, she was asked to interpret the benchmark evaluation problem behind AI chips from the perspective of a third party.

Figure | Sun Mingjun, president of Nanjing New Generation AI research institute

nanjing Xinxin New Generation AI research institute, aiming at AI chip evaluation

how much does Nanjing attach importance to artificial intelligence?

As can be seen from the wisdom Newport vigorously promoted by Nanjing Economic and Technological Development Zone, in recent years, in addition to introducing enterprise research institutes such as Jingdong, Cobos and Innovation Works, it also cooperates with some government agencies to build new research institutes, such as Nanjing New Generation Artificial Intelligence Research Institute.

Last year, on the basis of early cooperation, China Information and Communication Research Institute (hereinafter referred to as "Information and Communication Institute") and Nanjing Economic and Technological Development Zone, relying on the advantages of China Information and Communication Research Institute in the field of artificial intelligence technology reserves and industrial resources, we cooperate to build a new generation of artificial intelligence research institute in Nanjing.

It is reported that in recent years, the Institute of Information and Communications has carried out a series of research work on policies, standards, evaluation and testing in cloud computing, big data, artificial intelligence, broadband mobile communication, mobile Internet and other fields.

In this context, the new generation of artificial intelligence research institute in Nanjing has also shouldered the standard evaluation work on artificial intelligence. As Sun Mingjun said, "standards or evaluations are all done by professional institutions of platform type, so the Institute of Information Technology wants to do this."

on the other hand, a variety of AI acceleration chips have emerged in the entire AI industry. However, these chips with complex functions do not have a unified measurement standard to evaluate computing performance, the specific situation of the computing power per unit energy consumption. At the same time, for those who want to buy chips, it is difficult to judge whether the chip can meet the needs of real scenarios from the official information of the manufacturer.

More importantly, the existing benchmark test is not suitable for evaluating AI chips. In the case of unequal supply and demand information, a neutral third party is more urgently needed to provide appropriate evaluation solutions.

The new generation of artificial intelligence research institute in Nanjing is responsible for this evaluation. Last year, the China Artificial Intelligence Industry Development Alliance (guided by the National Development and Reform Commission, the Ministry of Science and Technology, the Ministry of Industry and Information Technology, the Internet Information Office, and initiated by the China Information and Communication Research Institute) joined forces with Alibaba Group, Baidu, cambrian technology and other enterprises have launched AIIA DNN benchmark-artificial intelligence end-side chip benchmark test and evaluation scheme.

At the 2018 AIIA AI developer conference, Sun Mingjun released V0.5 of the evaluation scheme on behalf of the China AI Industry Development Alliance.

AI chip evaluation tool to guide and standardize the specialized chip market

"It is very important to set up a benchmark test for a dedicated chip" Sun Mingjun repeatedly stressed the importance and key of the test platform in the development of the entire AI industry in an interview with make.com. "This is an indispensable link in the industrial chain. Especially in the early stage of industrial development, when the technical route is not yet clear, what a good benchmark test should do is to set up a clear index technology competition system, which can help enterprises to make rapid progress, at the same time, it objectively reflects the current situation of products. If an industry is full of too many mixed products, it is difficult to go on."

therefore, there must be a third-party neutral institution to prevent the phenomenon of bad currency expelling good currency in this field.

According to the introduction of Sun Mingjun, AIIA DNN benchmark V0.5 test is mainly oriented to the end side, aiming to objectively reflect the performance of processors with deep neural network acceleration capability when completing inference tasks. The evaluation tool of V0.5 is based on Android and Linux. The machine learning training software frameworks that support evaluation include TensorFlow and Caffe, the mobile app has been adapted to HiAI, MACE, SNPE, TensorFlow Lite, and Tengine.

At the "AI in 5G -- leading the new era Forum" held on March this year, the institute released the first round of evaluation results, including four typical scenarios and two major types of evaluation indicators. Evaluation scenarios include image classification, target detection, super resolution, and segmentation network. Evaluation indicators include speed (fps) and algorithm performance. Algorithm performance indicators include top1, top5, mAP, mIoU, PSNR, etc.

When it comes to the current AI benchmark test, in addition to AIIA, Ali, Cambrian and Baidu all have actions: Ali launched AI Matrix at the Yunqi Conference last year; Institute of Computing, Cambrian, xunfei, Jingdong, six companies, including ruidike and AMD, jointly launched BenchIP. MLPerf abroad was also led by Google in last May to carry out research on relevant benchmark measurement tools in conjunction with major technology companies and universities around the world.

When asked about the advantages of AIIA's evaluation tools over other benchmark testing tools, Sun Mingjun stressed, "We have no product tendency and are very neutral and independent third-party tests without product color."

it is worth mentioning that Sun Mingjun also emphasizes that AIIA DNN benchmark is the first Benchmark to distinguish integer and floating point comparisons in the deep learning processor field.

AI chip evaluation is not a one-day effort, open-source tools encourage enterprises to participate more

although the AI chip evaluation tool has been launched, considering the complexity of dedicated chips, it also brings some difficulties to the evaluation work.

"A big problem we are facing is adaptation. For example, TensorFlow and Caffe have to do a lot of work to adapt to Qualcomm and Haisi, but this problem did not exist in the previous general CPU evaluation."

sun Mingjun indicates the difficulty of adaptation, there are many scenarios to be tested due to the large number of scenarios on the chip. In addition, latency, bandwidth, and energy consumption should also be taken into consideration. In addition, various neural network models have different parameters and different output curves under different parameters, therefore, the benchmark tool must be continuously iterated.

Although many enterprises are also doing benchmark tests on AI chips, the entire industry seems to lack unified standards. In response, Sun Mingjun explained, "the reason why AI benchmark testing has not yet been recognized as a standard industry testing system by any enterprise lies in the diversity of artificial intelligence processors, the size, function, different architectures, processes, application fields, scopes, and specific scenarios lead to the complexity of artificial intelligence processors."

if the evaluation system is to be established well, all possible scenarios and problems of the processor must be taken into account. This is also the difficulty of the industry.

Therefore, in order to attract more enterprises to participate in the testing of dedicated chips, DNN benchmark V0.5 evaluation tool has been open source on Github.

Selected, One-Stop Store for Enterprise Applications
Support various scenarios to meet companies' needs at different stages of development

Start Building Today with a Free Trial to 50+ Products

Learn and experience the power of Alibaba Cloud.

Sign Up Now