Hologres:User profiling

Background

User profiling analyzes group characteristics and individual behavior based on user properties, actions, and preferences. A profiling system typically handles tag mapping, offline characteristic processing, and ad hoc data loading to enable real-time group segmentation and identification.

Scaling a profiling system to production brings two recurring challenges:

Hologres solutions

Hologres addresses these challenges through unified data links, a suite of profile computing plug-ins, and solution templates calibrated to your data scale and performance requirements.

• Data links

  Hologres consolidates real-time and offline data processing into a single pipeline, eliminating data inconsistency and data silo problems that arise from maintaining multiple links. The following figure illustrates the unified data link architecture.

  • Hologres integrates with DataWorks to resolve complex data dependencies through access configurations, providing stable offline data processing and loading pipelines.

  • For real-time writes, Hologres uses row-oriented storage based on the log-structured merge (LSM) structure, and integrates with Realtime Compute for Apache Flink for real-time tagging and characteristic processing.

  • Hologres supports federated queries, letting you access MaxCompute, Object Storage Service (OSS), and other Hologres instances through foreign tables without moving data.

• Profile computing

  Hologres is compatible with the PostgreSQL ecosystem and includes built-in functions for profile analysis. The following plug-ins extend these capabilities based on production experience at Alibaba Cloud.

  • Precise deduplication: roaringbitmap

    Hologres supports Roaring bitmaps for union and intersection operations on sets and bitwise aggregate operations using efficient compressed bitmaps. Roaring bitmaps work best on tables with unique data across multiple dimensions — typical use cases include UV (unique visitor) deduplication, tag-based filtering, and quasi-real-time user profile analysis.

  • Action data-based user identification: bit_construct and bit_match

    When action data is partitioned by day or hour across multiple rows, querying users who performed a sequence of actions requires repeated SELF JOIN operations. For example, to find users whose actions are [action='click' and page='Shopping cart'] and [action='view' and page='Favorites'] with the ds value ranging from 20210216 to 20210218, you need multiple joins across the action table.

    Hologres provides three functions — bit_construct, bit_or, and bit_match — to replace JOIN-heavy queries with bitmap operations. Each user's UID is stored as a bit array encoding which conditions they met, and bit_match applies AND logic across those arrays to identify qualifying users.

    WITH tbl as (
    SELECT uid, bit_or(bit_construct(
      a := (action='click' and page='Shopping cart'),
      b := (action='view' and page='Favorites'))) as uid_mask
      FROM ods_app_dwd
    WHERE ds < '20210218' AND ds > '20210216'
    GROUP BY uid )
    SELECT uid from tbl where bit_match('a&b', uid_mask);

    • bit_construct: evaluates each condition and stores the result as a bit value in an array. For conditions a and b in the example above, it returns values such as [1,0], [0,0], [0,1]....

    • bit_or: performs OR operations across bit arrays to aggregate actions per user.

    • bit_match: tests whether a bit array satisfies an expression. For the a&b expression, it returns True for [1,1] and False for [1,0].

  • Funnel analysis: Funnel functions

    Funnel analysis tracks user behavior across steps to calculate conversion rates. It applies to user behavior analysis, app traffic analysis, and product goal conversion — anywhere you need to measure how many users complete a sequence of actions.

    The window funnel function queries events within a sliding time window and returns the maximum number of steps matched. The retention function builds on the same approach to track whether users return after an initial action. Both functions replace complex JOIN chains with a single pass, reducing overhead and improving query performance.

  • Vector processing: Introduction

    Proxima is a high-performance vector search library developed by Alibaba DAMO Academy for nearest-neighbor search. It outperforms open source alternatives such as Facebook AI Similarity Search (Faiss) in stability and performance, and supports image, video, and face similarity search. Hologres integrates with Proxima to provide K-nearest neighbors (KNN) search, Radius nearest neighbors (RNN) search, and DOT_PRODUCT operations.

• Solutions

  The right solution depends on your data volume, tag count, and query performance requirements. Hologres offers three options:

  • Wide tables

    Best for scenarios with fewer than 1,000 tags and infrequent updates. Stable property tables are aggregated into a single wide table offline, converting multi-table JOIN operations into single-table queries. New tags are added as columns. For more information, see User properties wide tables.

  • Roaring bitmaps

    Best for large datasets with many tags where deduplication is required. Roaring bitmap storage provides natural deduplication, eliminates JOIN overhead, and accelerates data retrieval. For more information, see Analyze large user properties with roaring bitmaps.

  • Bit-sliced index (BSI)

    Best for association analysis between user attribute tags (such as gender and province) and behavior tags (such as page views and order amounts) on high-cardinality, high-volume datasets. BSI works alongside Roaring bitmaps to convert tag deduplication, UNION, and JOIN operations into binary BSI operations, simplifying computation and accelerating behavior tag analysis. For more information, see User profiling with bit-sliced index.

• Summary

  Hologres is used in production for tag computing and profile analysis across multiple Alibaba Group businesses — including Alimama, search, and AMap — and by a large base of public cloud users. Its scalability and stability under production workloads make it a practical foundation for a high-performance, low-maintenance profile analysis platform.