Self cultivation of a real-time accurate touch system

Problem definition

In the Internet industry, the only constant is that it has been changing. As technical students, we often meet the following needs:

Inform users in a timely manner when the prices of their favorite commodities are reduced, so as to facilitate the transaction between the two parties; A new user or a user who has not made a deal within 90 days can guide the user to chat with the seller actively after browsing multiple commodities, or send a red envelope to the user to promote the user's first order;

These requirements are essentially such a logic: real-time collection and analysis of user behavior, and accurate access to qualified users through rule calculation. It is difficult for the common development mode to meet this kind of demand, so we have specially developed omega system to solve this kind of problem. The omega system is divided into three subsystems:

*Behavior Collection Center

*CEP Rule Center

*User touch center

We have described the first two parts in detail in the previous article. This time, we will focus on how the user touch system is designed and implemented with flexible configuration and accurate touch.

System design

2.1 Logic architecture

To facilitate readers' understanding, we briefly review the logic architecture of the omega system. The omega system is split based on the principle of high promotion and low coupling. Each part is an independent and complete system, and can also provide services after assembly.

*The first layer is the user behavior collection center, which cleans the data into regular user behavior data through the MTOP (Application Gateway) interface requested on the collection end and the user behavior buried point on the end;

*The second layer is the CEP rule computing center, which generates Blink (Flink) flow computing tasks by parsing DSLs, and outputs users who meet the rules;

*The third layer is the user touch center, which defines the touch strategy and channel, and delivers the strategy to the user in real time.

The three layers are interlinked, which can provide services independently or jointly undertake external business. At present, it has been undertaking user growth, gameplay and security related businesses.

Take the user growth business as an example. When users experience, the operation guides them to complete the transaction behavior through a reasonable strategy combination to reach the "aha" moment in the product form. These strategies may include rights and interests disclosure, POP and real-time push within the end, and push, SMS and outbound call outside the end. Omega system integrates active/passive access channels inside/outside the end, and takes users' real-time status as the core to realize a set of technical scheme system for strategy arrangement to meet long-term operation.

2.2 Touching process

The Reach process itself is relatively clear. We split the process into multiple small nodes. Each node is combined in a configurable way to ensure that each node is pluggable and replaceable. The overall user touch system processing flow is as follows:

*Receive CEP rule calculation results, including rule names and users who meet the rules;

*The Action routing layer queries the list of all actions subscribed to this rule according to the rule name;

*The Action filter layer filters the list of effective Actions according to certain strategies, including black/white list, grayscale, crowd and fatigue strategies;

*The Action distribution layer will execute according to the policy configuration, which can be universal touch, such as sending push and SMS; It can also be used to call other business systems, such as security system penalties; Action can also be distributed to the end for execution;

*After the Action is executed, the relevant information will be buried in accordance with the general protocol to facilitate subsequent data statistics;

User touch is the last link in the process of the omega system. It needs to encapsulate enough universal touch capabilities to ensure the real-time and effectiveness of touch, otherwise it will harm the user experience. Next, through detailed design, we will see how the user touch system can ensure that the touch strategy can be assembled, pluggable, flexible configuration, real-time touch and other features.

2.3 Detailed design

Note: Metaq is the MQ framework used internally by Alibaba; HSF is an RPC framework.

The goal of the user touch center is to provide services independently, support flexible pluggable configurations and precise access strategies, so the design focuses on reducing external dependence, and reducing direct dependence and coupling on external systems through MQ; Define the functional boundary of each sub module internally, and combine the sub modules through configuration.

The main function of the user access center is to maintain the access strategy and the access capability of packaging standards, which is divided into the following parts:

*Input data source: the user touch center can receive the calculation results of the upper rule center, or it can be triggered by the external business system;

*Touchtouch materials, including copy and pictures, are maintained in the cloud delivery system (free fish material management system). Later, offline data will be accessed to supplement more fine-grained basic information, including user portraits, commodity data, etc.

*The Action routing layer maintains the subscription relationship between Action and rules, including subscription validity time, priority and other elements;

*The Action filter layer is designed in the responsibility chain mode, and each filter is independent from each other, which can be dynamically plugged and flexibly configured;

*The Action implementation layer encapsulates the implementation of various universal access capabilities, mainly cloud and client. Later, the faas mode can provide the ability to flexibly and quickly launch Action. In order to ensure the real-time execution of Action on the client side, we have specially maintained a long connection channel with the client side. Through targeted optimization, we have improved the data transmission speed and arrival rate of the channel, and focused on ensuring the end-to-end access.

*After the action is reached, it will be recorded according to the unified burial point agreement. Later, the burial point reporting and data development process will be sorted out to reduce the cost of data development and facilitate the business side to view the effect and attribution of the action experiment.

Online effect

After the user touch center went online, it has undertaken multiple businesses through configuration, including many business scenarios such as free fish, golden scales, double 11 games, user growth, rental, leasing, etc. Through flexible operational configuration strategies and real-time accurate access to rights and interests, the following data results were obtained:

1:The accuracy rate of reaching the target population has been greatly improved;

2:The golden scale playing method is delayed within 1s;

3:It is better to teach people how to fish than to teach them how to fish, provide operation tools, and completely liberate and develop resources;

4:Among them, the "Double 11" project has high requirements for real-time and QPS, which fully verifies the performance and real-time access capability of Omega system, especially the user access center. In the end, the click rate of Push in browsing the commodity price reduction scenario has greatly increased compared with offline.

Summary and outlook

Omega system is a highly abstract solution for scenarios with high real-time requirements, operation oriented and fast experiments. Adhering to this concept, the user access center encapsulates a variety of universal access capabilities, supports flexible pluggable filter configurations and designs standard buried point protocols to support rapid business experiments and data attribution analysis. In the future, we will support the standard access of offline portrait data and the standardization of data return analysis, open up the upstream and downstream data of the business, and achieve process closed-loop in function. Readers are also welcome to exchange and discuss.