DMS + AnalyticDB Helps E-commerce Enterprises Make Decisions to Address Seven Major Challenges

This article focuses on the pain points of the e-commerce industry and discusses how to revitalize the e-commerce industry by using the Data + AI solution and the data analysis methodology.

This article is the first article of the Get Started with Data + AI series published by Alibaba Cloud ApsaraDB. This series of articles describes the application scenarios of the Data + AI solution in various industries based on real customer cases and best practices.

This article focuses on the pain points of the e-commerce industry and discusses how to revitalize the e-commerce industry by using the Data + AI solution and the data analysis methodology.

1. Industry Trends

In the digitalization era, the e-commerce industry has been significantly reformed and developed with data and AI as the core driving forces. To pursue great values and opportunities from massive data, e-commerce enterprises deeply analyze big data to collect accurate information about customer needs, optimize operations processes, and improve decision-making efficiency. In this case, AI provides more powerful intelligent service capabilities for e-commerce enterprises to explore unlimited data values.

In the past few years, e-commerce enterprises have realized digital transformation by building big data systems. However, the existing big data architecture mainly focuses on structured analysis instead of analyzing information in images and documents and fails to provide real-time and agile analysis applications. The use of multiple engines also increases the development and O&M costs. With the evolution of technologies, large language models (LLMs) and Retrieval-Augmented Generation (RAG) services implement multimodal analysis to retrieve data from more information dimensions, and the integrated offline and online data processing engines support streaming processing and batch processing in online data analysis scenarios.

How is the big data architecture quickly upgraded to a new Data + AI architecture that is able to support business innovations in the next three to five years? How are deep and accurate data insights implemented with more information dimensions? This article discusses how to revitalize the e-commerce industry by using the Data + AI solution and the data analysis methodology.

2. Technical Challenges

As online processing, real-time analysis, and intelligent decision-making become rigid demands of the e-commerce industry, the technical architecture of enterprises faces the following challenges in data analysis and AI construction:

2.1. Online Data Refreshing

After the configuration customization capability is provided, merchants want to view the updated data immediately after they modify the configuration. This requires the data warehouse engine with capabilities to support various functions and transactions, process complex logics, recalculate historical data online, and provide analysis services to customers based on the configuration.

2.2. Real-time Online Analysis

The e-commerce business requires not only real-time analysis but also comparative analysis based on historical data. Therefore, the engine must provide streaming processing and batch processing capabilities to meet the requirements for data processing and complex calculation of real-time metrics, offline metrics, cumulative metrics, comparison metrics, and trend analysis metrics.

2.3. Cost Optimization

Long-period data analysis is valuable to merchants. However, it requires the data warehouse engine that supports tiered storage of hot data and cold data to reduce long-period data storage costs. In this case, the tiered storage cannot affect the development and use of the data.

2.4. Stability Improvement

The e-commerce business must be stable at all times to provide paid services. Therefore, the data warehouse engine must ensure that the cluster performance does not degrade in exceptional circumstances.

2.5. Data Quality and Governance

The application of AI in the e-commerce industry depends on high-quality data and is affected by factors such as data accuracy, dispersion, and freshness.

2.6. Integration of Data Assets and AI

To realize the value of a large number of data assets, enterprises must apply an efficient integration mechanism to enable not only the mutual empowerment of data assets and AI but also the collaboration between the data asset team and AI team.

2.7. Costs, Talents and Organizations

AI implementation always requires high investments in the early stage, including investments in infrastructure, talent training, business processes, and organization reform. In this case, enterprises need to estimate the return on investment to reduce costs and improve efficiency.

3. Data + AI Solution from Alibaba Cloud

In September of this year, Alibaba Cloud ApsaraDB announced DMS + X: Unified, Open, and Multimodal Data Management and Data Serving Driven by Data + AI at Apsara Conference 2024. The platform simplifies data management and AI development by using OneMeta and OneOps and manages Data + AI development throughout its lifecycle. Based on Data Management (DMS) + X, Alibaba Cloud can help enterprises integrate data with AI, deploy business, and realize business values at the earliest opportunity.

To address the technical challenges in business development, e-commerce customers can use DMS with AnalyticDB to apply the new Data + AI architecture and develop AI-native warehouse intelligence capabilities to build new-generation online data warehouses.

Data Architecture Upgrade for Data + AI

Zero-ETL

The current e-commerce industry requires flexible business rule configuration and self-service report analysis. However, the traditional development model of extract, transform, load (ETL) and offline scheduling and processing fails to meet the analysis requirements of merchants. In this case, Alibaba Cloud AnalyticDB for PostgreSQL provides the Zero-ETL feature that allows you to synchronize incremental data from databases without the need to configure an ETL task. This feature can be used together with incremental real-time materialized views to process tasks and refresh data without scheduling. This makes business analysis timelier and more flexible.

Real-time Online Analysis

In the past years, it was costly to analyze the association between streaming data and historical data and learn the syntax of development and O&M engines. AnalyticDB for PostgreSQL provides incremental real-time materialized views that support various syntax, such as multi-table join queries, nested subqueries, and window functions. This feature also supports data refresh in rows and cascade refresh and does not require scheduling dependencies of data analysis tasks. With the direct write capability of AnalyticDB for PostgreSQL distributed compute nodes, the throughput of data write by using the UPSERT operation can be more than 100,000 requests per second (RPS).

Resource Isolation for HTAP

Hybrid Transaction/Analytical Processing (HTAP) is supported to meet the requirements of clusters on high-throughput data writes, historical data recalculation, real-time computing, and online analysis services. Based on the resource isolation capability of AnalyticDB for PostgreSQL, different resource groups are created to dynamically allocate resources based on the priority of business requirements in different periods of time. For example, to ensure the continuity of online analysis services and real-time computing in the morning, fewer resources are used to refresh data. On the contrary, in the early hours of the morning, most resources are allocated to perform batch data processing, and some resources are reserved to provide analysis services for key customers.

Long-period Data Archiving

In the past, historical data stored for more than two years could not be analyzed due to cost considerations, and it was inconvenient to manually dump tables without partitions to cold storage. AnalyticDB for PostgreSQL supports automatic archiving of long-period data at the partition or row level based on the directive field. Long-period data can be automatically routed to hot or cold storage data, or only to the hot storage data by specifying related parameters. This reduces storage costs and ensures that customers can use historical data with ease.

Meet Business Requirements of Key Customers

When isolated resources are provided to meet the business requirements of key customers, the overall computing and storage costs must also be taken into consideration. For batch processing, AnalyticDB for PostgreSQL allows data access across instances to avoid data redundant storage, provides real-time materialized views to calculate data in the central data warehouse and data warehouses of key customers, and retains the result data in the satellite data warehouse. Data that is frequently queried is incrementally synchronized to data warehouses of key customers by using Change Data Capture (CDC). This ensures the high performance of online services. Costs are also apportioned based on the resource usage of computing tasks.

Dynamic Resource Scaling

The continuity of dynamic resource scaling services must be ensured to provide paid services at all times. Although open source services of the Massively Parallel Processing (MPP) architecture are highly available, the performance of the entire cluster is degraded when the host on which a compute node runs fails. To ensure cluster performance, AnalyticDB for PostgreSQL provides the capability to dynamically scale resources.

Exploration of AI Scenarios

Alibaba Cloud ApsaraDB provides intelligent search and image-based search solutions that allow e-commerce customers to explore AI scenarios based on data warehouses.

Intelligent Search

With the rapid evolution of business environment, data has become a critical resource for enterprises to formulate business strategies. Accurate data analysis and efficient data management are essential to optimize decision-making processes and promote innovation.

DMS is a data management service published by Alibaba Cloud in 2013 to meet the all-in-one data management requirements of enterprises. DMS Data Copilot is an intelligent data assistant built by DMS based on the LLMs of Alibaba Cloud. It allows you to generate and optimize SQL statements by using natural language, which simplifies SQL statement writing and improves development efficiency.

Data teams of e-commerce enterprises develop data intelligence services for merchants and provide data analysis support to the internal operations teams. However, in addition to heavy development workloads, data teams also need to deal with the frequent data retrieval requirements of merchants, operations teams, and service teams, which cannot be fully met by using only a large number of data reports. This presents significant challenges in data R&D.

DMS Data Copilot can meet the flexible data retrieval requirements of all parties. It supports data acquisition by using natural language and quick chart generation. Users can ask questions to obtain the required results and view data change trends in the generated charts.

For internal teams, DMS Data Copilot improves the efficiency of data report development. For example, you want to count the top three business categories of merchants that visited the global overview page in the previous seven days based on sales areas and contract versions. In this case, you only need to enter your requirement. Then, DMS Data Copilot directly generates the corresponding SQL statements. For the personalized requirements of users, DMS Data Copilot also displays historical knowledge bases that are used as references to further improve the accuracy of answers.

Image-based Search

Based on the all-in-one RAG API of AnalyticDB for PostgreSQL, a complete link is built for image upload, embedding, and image-based search. The entire technology base of image-based search can be established and optimized within three days to provide customers with recommendations on similar goods.

4. Summary and Prospect

Alibaba Cloud ApsaraDB provides a complete and innovative Data + AI solution and best practices of the solution to address seven major technical challenges in the e-commerce industry. The combination of DMS and AnalyticDB meets the requirements of online data processing, real-time analysis, and AI practices. This greatly reduces the development and O&M costs of enterprises.

Data + AI provides a new way for enterprise growth. Enterprises must recognize the importance of Data + AI and use it as the key solution to promote intelligent transformation. This way, enterprises can remain competitive and leading in the market and better seize opportunities in the future. After e-commerce customers upgrade and transform their business based on the Data + AI solution and provide AI-native capabilities to the external environments, the development and application of AI can be more inclusive. Through gradual exploration and implementation, Alibaba Cloud looks forward to providing more intelligent e-commerce services in the future.

Community