6 years of Double 11, DataV's way of data visualization

Since 2012, Alibaba's annual Double 11 promotion will launch a large screen to display transactions in real time in a variety of vivid display methods. The real-time data big screen is characterized by various sizes: large screen, large amount of data, and large amount of displayed information. It can be said to be the holy grail of data visualization.

The Alibaba Cloud DataV team has participated in the construction of the Double 11 media digital large screen for 6 consecutive years. This year's "China Regional Economic Lightning Map" media touch screen has become one of the core focuses of the Double 11 Gala, with hundreds of millions of buyers across the country on the big screen. of orders flocked to sellers along the real road, and the effect was shocking. The large touch screen supports users to drill down and query data by themselves, which greatly enhances the readability of data. On that day, more than 500 Chinese and foreign media used it as a live broadcast data explanation platform, realizing the perfect integration of old and new media.

This year's Double Eleven data big screen project, for the first time integrated application of "DataV data large screen computing and rendering load balancing", "global logistics trunk visualization model", "DataV MScreen multi-view fusion", "WebGL post-production visual effect enhancement" and many more This new data visualization technology developed by DataV can still ensure silky smoothness and stunning visual effects in the scenario of super 12K resolution single-screen 3D rendering and concurrent access to 30+ real-time data sources.

In the 2016 Double 11 media screen, the main visual is a three-dimensional city scene of Shenzhen. The color of the building represents the size of the order amount in the building. The redder the color, the higher the order amount. The panel on the right displays information such as the user's crowd portrait.


As the hero behind the double 11 data big screen, DataV has tried almost all the paths of technology precipitation after several large-scale projects: open source code, productization, packaging solutions, platformization... The application scenarios are also from the double 11 e-commerce battle , expanded to smart cities, smart transportation and many other fields. Their journey and thinking along the way are worthy of reference for later generations.

The birth of DataV is accompanied by Alibaba Group's exploration of data-driven operations.

In 2011, Taobao hatched its first data product, Data Cube, which aims to reduce the cost of understanding data for small and medium-sized sellers on the platform through vivid data charts, and then combined with functions such as chart interaction to allow sellers to leverage on the actual operation. store data. The emergence of Data Cube has popularized the value of data operation both internally and externally. At that time, the current DataV team was responsible for the visualization and development of Data Cube, and they quickly undertook the technical upgrading and transformation of Alibaba's internal data screen. In the process , and jointly promote the construction of Ali's real-time data indicator system.

In 2012, Double 11 had its own media center for the first time—a temporary command center transformed from a training classroom. The hardware of the large media screen was supported by two projectors and two laptops. Such equipment looks like today. Very simple and crude. At that time, because the estimated transaction amount was too low, there was an oolong. Not long after the 2012 Double 11, the actual transaction amount exceeded the preset maximum value of the chart, and the GMV real-time line chart broke the table as soon as it came up. Although there was no special sorting out of indicators and lack of experience in developing visual charts, the Double 11 screen that debuted for the first time brought a huge visual impact and a real sense of data.

Until this year's Double 11, the DataV team has been responsible for the visual development of the double 11 media data large screen for 6 years. The resolution of the large screen has increased from 1024*768 to 12K, and the screen area is almost 400 square meters, showing the complexity of data and real-time data links The road has undergone earth-shaking changes.

On the one hand, the technical capabilities are upgraded, and on the other hand, the DataV team has also switched from the role of internal services to the provider of visualization products and solutions, and is exporting its real-time data visualization capabilities to the outside world. At the same time, DataV is also transforming from pure data visualization to data visual analysis.

Currently, DataV is fully cooperating with Alibaba Cloud's ET City Brain project to implement a visual analysis platform for city-level multi-dimensional data. The first-phase data command screen has become a part of the Hangzhou traffic police's daily work, and it has also combined machine intelligence for image recognition. Wait for the AI ​​scene to start a new round of technical solution upgrades.

In addition, DataV is also working with many government enterprises that are embarking on technological transformation to upgrade the data command screen to help them quickly find the sense of data in the initial stage of data operation and management.

From open source to productization
In the early days, DataV precipitated the datav.js component library and made it open source, but later, DataV abandoned this open source project and provided it as a product. Why?

The original intention of open source is to give visualization technology to more data presentation scenarios, but in the process of promotion, it encountered a large number of customized issues, and developers encountered a lot of problems in processing data and chart docking (such as D3 rich The demo library is still a bit high threshold for data developers), so open source did not achieve the original vision. In addition, the datav.js component library architecture also took into account the support for low-version IE browsers at that time, but later this demand became weaker, so this library was gradually deprecated.

During this process, the DataV team observed that providing a visual component library does not mean opening up the full link of data Kanban production. For those who directly face data or want to interact with data, the component library lowers some of the thresholds. But in addition, high-performance data visualization also involves multiple skills such as style design, data connection, component architecture design, etc., which will be more complete based on a product. So the DataV team began to figure out the path to productization.

First of all, although data visualization is a front-end post under the system of large companies, it is actually a comprehensive reflection of business understanding, data understanding, and visual expression, which requires business, data, front-end, and design to be completed together. From a purely technical point of view, the front-end components are only a part. How data is processed and how to connect is actually a more critical step in the implementation of visualization in business. Compared with the well-known development path of ECharts, DataV hopes to connect data, business, front-end, and design in a productized way to truly lower the threshold.

In this way, data development can easily realize the front-end work of professional charts, and designers and businesses can also easily create "moving" data boards, which greatly reduces the engineering and communication costs between visualization stacks.
Secondly, for a commercial product, developers can more accurately find out the real needs of users through feedback with customers. Although open source front-end components are widely used, developers themselves do not know what kind of scenarios others use, so it is difficult to touch the pain points and demands. From the front-end alone, after the basic chart development is completed, there will also be bottlenecks in the direction.

DataV's current positioning is still a tool for visual display and analysis on the big screen.

In terms of function, the most important thing is to provide a large-screen interface configuration tool that can be dragged and a variety of templates. It is easy to achieve the double 11 large-screen effect by making a PPT template. It also includes a variety of data connection capabilities, including common types of databases, API interfaces, etc.


In terms of features, in addition to the regular cool-looking basic chart components, DataV has made great efforts in the field of WebGIS, integrating various WebGIS capabilities. It is very perfect for d3.js, leaflet.js, mapbox.js, native canvas, SVG, traditional back-end drawing map tiles, mapping and projection of various geographic data, and event response of map multi-layers. integration. General BI or visualization tools are relatively easy to implement on basic charts, but the GIS part is relatively weak. DataV can easily realize the integration of multi-layer multi-drawing technology and geographic-related data integration through interface configuration.

In 2016, DataV front-end development engineer Mir shared the "Double 11 Data Large Screen Technology Evolution", taking the Double 11 over the past years as an example, to introduce the process of DataV's development from an initial exe program to a large-scale WebGL application.

DataV.gl: WebGL-based Smart City Visual Analysis Solution
After the research and practice of WebGL in the past few years, DataV has launched the DataV.gl component, which greatly improves the rendering performance and realizes the customization of 3D digital cities. From this point of view, it has achieved a world-leading level.

DataV packs 3D visualization capabilities into an easy-to-use component. Users can import 3D model files to configure data and adjust styles through graphical menus just like editing other charts. In terms of effect, DataV.gl supports 100,000 The building rendering of the order of magnitude supports the presentation of pipeline routes and natural landscapes in the city. Users can also customize the movement trajectory of the camera and realize the free exploration of the city picture.

From a technical point of view, DataV.gl greatly improves the computational and engineering efficiency of 3D urban geographic visualization. Using the classic browser-based GPGPU technology and the computing power of the graphics card, DataV.gl supports smooth rendering of massive data for computationally intensive visualization schemes such as flow fields and force maps. Compared with the traditional CPU-based Canvas force map rendering solution, DataV.gl's GPU-accelerated force map rendering solution has a performance improvement of more than 30 times.

At the same time, DataV.gl supports urban data planning files (CAD and other formats) and standard vector geographic data as input data. Through geographic data visualization mapping services, 3D urban building models can be directly modeled, which is easier than traditional designers use 3D software to model. Compared with the time-consuming weeks, the efficiency has been greatly improved.


In the past two years, Uber has made great achievements in the field of 3D visualization of geographic information, and the released deck.gl front-end library has become an industry benchmark. However, a single source visualization technology framework cannot provide big data visualization solutions across industries and multiple scenarios. At the same time, deck.gl also has many limitations in meeting individual needs and selecting front-end technology stacks.

Uber deck.gl is a visual front-end library that can meet the needs of data visualization development, while other links require users to find solutions by themselves, while datav.gl is a complete set of high-performance visualization solutions for massive data, users can "one-stop" Complete data visualization business.

If you use a car as an analogy, Uber deck.gl is like a car engine, and datav.gl is a complete vehicle; only the engine can't go on the road, and other parts are needed for assembly.

In addition, the differences between datav.gl and deck.gl are:

(1) In terms of functional positioning, datav.gl has a wider coverage than deck.gl. It not only provides the function of a visualization library, but also provides a complete set of big data visualization solutions.

(2) Distinguish from the target user group, deck.gl is biased towards developer groups, and datav.gl is biased towards enterprise and government users.

(3) From the perspective of user demand adaptation, deck.gl is deeply bound to React, and the basic official visual components are more convenient to use in front-end development, but some personalized requirements require in-depth analysis of deck and React frameworks to develop, and the relative cost higher. datav.gl provides a developer version. Users can use official components or develop custom components according to their own technology stack and development habits.

(4) In terms of rendering efficiency, both datav.gl and deck.gl are based on WebGL, with similar basic performance. Datav.gl has advantages in some specific massive data visualization scenarios, as mentioned above.

(5) From the perspective of visual material resource integration, datav.gl accesses some resources such as live streaming media, 3D modeling, etc., and provides fool-like components; deck.gl is relatively complex.

The future of DataV
Regarding the future direction, DataV's focus is mainly on the visualization of ecology and technology exploration.

Although the big data screen can serve all walks of life and has huge market potential, this market needs more ecological partners to join in, whether it is from the various aspects of the landing of the big screen or from the field knowledge required to finally enter the industry. different sparks. For example, in the ET City Brain project, many ecological forces and DataV have joined forces to form a complete set of city-level data visual analysis large-screen solutions.

Of course, as a technical team, DataV also attaches great importance to the visual developers on the platform. For this reason, they have launched the developer version, which enables developers to carry out secondary custom development. Following certain development specifications, they can integrate self-made components into In the product, it can be configured graphically like using native components. And, support developers to release their own components to the Alibaba Cloud market to realize visualization capabilities - this is also the continuation of the DataV team's original intention to promote the development of data visualization and find a commercial outlet for visualization.

At the same time, DataV also subtly shares its own technical ideas with external developers through technical documents, and relies on a large amount of usage data accumulated by the platform to provide real and accurate improvement directions for developers. Not only hope that end users can feel the value of data visualization, but also hope that this value can be passed on to thousands of developers who are struggling for visualization.

Another major direction of DataV is that it will continue to increase investment in the research and exploration direction of visual analysis, technically optimize the product framework to support more powerful data interaction functions, and strengthen cooperation with scientific research institutions such as universities on the basis of visualization theory. Ensure that you have stronger and lasting power support in the direction of data visualization.