×
Community Blog How Taobao Applies AI to Personalized Recommendations

How Taobao Applies AI to Personalized Recommendations

This article introduces Taobao Mobile's recommendation system and Alibaba's Artificial Intelligence Online Serving (AI OS).

Recommendation services, also known as recommender systems, have rapidly become one of the most significant traffic hubs for modern e-commerce sites, and any site with a wide amount of content and users. In a nutshell, a recommendation system is a complex filtering system that can anticipate consumer tastes in a digital environment.

These AI-driven programs are based on the premise that only those who know their customers better than they know themselves will take the lead in the age of modern retail. Taobao Mobile's recommendation system is one of the most advanced solutions of its kind and one of Taobao's main traffic portals and transaction networks. With a quick scroll through Taobao, it's not hard to see that this system involves some of the most complex business formats and scenario-oriented technologies ever seen in e-commerce.

The Power of AI: Why Taobao Knows Online Shoppers Better Than They Know Themselves

In this post, we will be taking a deep dive into how Taobao Mobile's recommendation system was developed from the ground up.

Taobao Mobile's Recommendation System

The rapid development of Taobao Mobile Recommendation System originated from Alibaba's "All in Wireless" strategy proposed back in 2014. In the wireless age, smart phones have come to more or less conquer the Internet; with this this revolution in hardware came many changes. For one, smart phone screens are much smaller than their desktop and laptop counterparts, and users cannot open multiple browser windows at the same time on a smart phone, changing how users could interact with web applications. To adapt to these changes, Taobao's mobile team developed a system of personalized recommendations, through its now highly sophisticated recommendation system, to improve each user's browsing efficiency on their mobile devices. After years of research and development, this system of recommendations has become the largest traffic portal on Taobao's mobile app. The app now serving hundreds of millions of users every day, and recommendations are a huge part of the app experience, being only second to search as one of the most commonly used ways to navigate the mobile app.

Taobao Mobile's Recommendation System

Today, in the Taobao app, recommendations are not only for items to shop for, but also for platform live broadcasts, stores, brands, user-generated content (UGC) and professionally generated content (PGC), among other things. Taobao Mobile provides a wide range of recommendations for hundreds of specific scenarios. Different to search, where users proactively express what they are searching for, the recommendation system rarely actively interacts with users. Rather, it interacts with users only through backend algorithm models. Therefore, since its inception, recommendation has been a project that has been built using big data and AI.

Major Features

Compared with other recommendation products, Taobao Mobile boasts the following advantages:

  1. Involved in a Shopper's Decision-Making Cycle: The main value of Taobao Mobile's Recommendation System is its uncanny ability to identify the potential wants and needs of users and help them make purchase decisions even before they may know what they want themselves. A user's decision-making cycle during an online shopping experience can be a relatively long and complex process, which involves the discovery of new needs, the acquisition of new and related information, as well as things like item comparison and order placement decisions. Therefore, e-commerce recommendation systems must work with this process, making recommendation decisions based on the user's behavior.
  2. Integrates a Feature of Timeliness: Timeliness is also considered into the underlying system; a system is relies heavily on Flink. Things that are purchased on Taobao are often purchased to satisfy needs of relatively low frequency and are for purchases that may only be valid for a short window of time. For example, people generally only buy a smart phone once every two to three years, and the decision-making cycle may just spread across a few hours, or just several days' time at most. Therefore, a recommendation system must be extremely time-sensitive when it comes to providing for these needs. It needs to quickly understand and capture the real-time interests of users and explore their unknown needs as well.
  3. Developed around a Complicated User Structure: Users on Taobao Mobile are not only users with registered accounts. App users also include users that aren't logged in, as well as new users, users that rarely active on the platform, and users that are no longer on the platform. Given all of these situations, the Taobao team had formulate differentiated recommendation policies and optimize the recommendation model accordingly.
  4. Covers Multiple Scenarios: Taobao Mobile Recommendation System covers hundreds of specific scenarios on the platform. Independent optimize for each scenario is far from realistic, as each scenario has different conditions that require different hyperparameters. Moreover, manual optimization is also unrealistic. Therefore, on Taobao, transfer learning and automatic hyperparameter learning has been paramount.
  5. Involves Multiple Targets and Types:

Features of Taobao Mobile's recommendation system

Technical Framework

The following slide shows the technical framework of Taobao Mobile's Recommendation System. During the 2019 Double 11 Shopping Festival, Alibaba migrated all of its business operations to the cloud. Therefore, the technical architecture of Taobao Mobile Recommendation System also took root on the cloud. The basic components of recommendation include recommendation algorithms and models, raw logs and features obtained by processing log data, and offline computing and service capabilities, such as vector search, machine learning platforms, and online sorting. Last year, in addition to our models and systems on the cloud we also deployed end-computing deep learning models to achieve a collaborative system of computing between the cloud and end devices.

Technical Framework for recommendations

The Secret Behind Taobao's AI-Powered Personalized Recommendations

This article introduces Alibaba's Artificial Intelligence Online Serving (AI OS) and the evolution of its technical architecture and practices.

1) Introduction to AI OS

AI OS is an online service platform developed by the Alibaba search engineering team that integrates personalized search, recommendation, and advertising. The AI OS engine system supports various business scenarios, including all Taobao Mobile search pages, Taobao Mobile information flows (You May Like), venues for major promotion activities, product recommendations on the Taobao homepage, and personalized recommendation and product selection by category and industry. It covers more than 80% of Taobao Mobile users. AI OS uses a set of technologies to support search, recommendation, and advertising, which is rare in large Internet companies. Alibaba's platform technical strategy essentially involves e-commerce technology and big data AI technology. The e-commerce OS includes commodity management, category management, operation management, and transaction links. In the era of big data and deep learning, AI delivery, search recommendations, and ad serving have become technical scenarios independent of traditional e-commerce. In addition to Taobao Mobile scenarios, AI OS supports all scenarios of the e-commerce scenarios in the Alibaba Group, such as Southeast Asia Lazada, Juhuasuan, Fliggy, Youku, DingTalk, Cainiao, HeMa, Eleme, and Koubei and even cooperates with Alipay, a subsidiary within the internal economy.

Artificial Intelligence Online Serving

In the deep learning era, the AI OS engine system architecture has evolved significantly. However, Alibaba does not develop deep learning technologies independent of search and recommendation systems like other Internet companies. From the technical level to business scenarios, AI OS integrates search, recommendation, information flow, advertising, and deep learning to form a basic engine platform. The platform helps each part develop based on the other parts.

The following figure shows the technologies and concepts involved in AI OS based on AI OS business scenarios.

technologies and concepts involved in AI OS

The first layer displays the basic capabilities required by the distributed engine system. Recall and sorting are required for search, recommendation, and advertising. Distributed communication, high-performance index storage, and capabilities for efficient and flexible indexing and updating are required after the distributed engine system is expanded.

The intermediate layer represents the technical and scenario requirements of the deep learning era. For example, deep learning requires sample processing, training, and online prediction, and their application scenarios are personalized delivery. Personalized delivery is represented in search, recommendation, and advertising. Indexes need to support real-time updates, which is important in e-commerce systems.

The last layer contains resource management, high availability, computing engine support, O&M control, and plug-in support for AI OS.

The following figure shows the technologies of the AI OS engineering system.

technologies of the AI OS engineering system

The underlying layer is Hippo, which is an effective resource management system.

The top layer includes Taobao, cloud, and advertising businesses, which have been growing in recent years and gradually migrated to AI OS. Many Alibaba technologies and businesses are developed in a bottom-up model. With a strong sense of innovation, we build platform-based search and recommendation from the bottom up to 70% to 80% and promote them to a strategic level to accelerate mid-end building and form a full-coverage layout.

The right part of the figure shows the AI OS middleware, which contains basic components related to actual business features, including Service positioning: A system that runs tens of thousands of machines needs to have its own mechanism for service positioning. Service monitoring: This component provides service monitoring at a granularity of seconds. The metrics of internal applications are important for distributed system debugging. Index distribution: This is an important basic component for engine data updates. Message queue: This is a high-performance message component built using fragmented host resources. It features low CPU consumption and network throughput and is basically a free component. Layer-2 scheduling and auto-scaling: These are important methods for internal minute-level resource scheduling for search, recommendation, and advertising during major promotion activities.

The algorithm, offline, training, and computing platforms on the left were developed in the deep learning era. Sample and feature processing involves the algorithm platform Nebular, which needs to interwork with the training engine X-Deep Learning (XDL). The computing platform provides support for algorithm samples and sample training and is also a powerful technical fulcrum in the Alibaba Group. It grew up with search, and the two promote and support each other.

The middle part shows the most important technical developments in recent years and is closely related to businesses. With Mobile Intelligence, we do more than simple recommendation change and mixed result sorting. We train models and predict deep models on terminals. Taobao Mobile information flows are the largest application scenario for deep learning, training, and prediction in the world. This is our special area of exploration.

The HA3 search engine is Alibaba's classic engine, which supports full-text retrieval. The commercialization engine is a recall engine that supports advertising businesses, keyword matching with ads, and targeted scenario push. The graph engine iGraph is a large graph retrieval engine that supports online graph computing and retrieval. It has multiple online derivation capabilities, including personalized user relationships and knowledge graph. These engines support real-time data update because the AI OS framework supports data and update management. The AI OS framework depends on the technologies on the right and has extended deep learning capabilities.

Related Products

Machine Learning Platform for AI

Machine Learning Platform for AI provides end-to-end machine learning services, including data processing, feature engineering, model training, model prediction, and model evaluation. Machine Learning Platform for AI combines all of these services to make AI more accessible than ever.

Elasticsearch

Alibaba Cloud Elasticsearch is based on the open-source Elasticsearch engine and is designed for search and analytics scenarios. Elasticsearch features enterprise-level access control, security monitoring, and automatic updates.

Related Solutions

Alibaba Cloud Big Data Consulting Services for Retail

Alibaba Cloud experts provide retailers with a lightweight and customized big data consulting service to help you assess your big data maturity and plan your big data journey in two to six weeks, transforming your retail business through big data technologies.

Alibaba Cloud Big Data Consulting Services for Enterprise Data Technology

Alibaba Cloud experts provide enterprises with a lightweight and customized big data consulting service to help assess your big data maturity and plan your big data adoption with actionable insights, speeding up your digital transformation and mapping out your digital future.

Alibaba Cloud Intelligence Brain

Alibaba Cloud Intelligence Brain is an ultra-intelligent AI Platform for solving complex business and social problems.

0 0 0
Share on

Alibaba Clouder

2,631 posts | 633 followers

You may also like

Comments

Alibaba Clouder

2,631 posts | 633 followers

Related Products

  • Machine Learning Platform For AI

    An end-to-end platform that provides various machine learning algorithms to meet your data mining and analysis requirements.

    Learn More
  • Elasticsearch

    Alibaba Cloud Elasticsearch is based on the open-source Elasticsearch engine and provides commercial X-Pack plug-ins. Designed for scenarios such as search and analytics, Alibaba Cloud Elasticsearch features enterprise-level access control, security monitoring, and automatic updates.

    Learn More