Log configuration supports flexible collection and analysis of online debug and feature logs through multiple output methods (console, DataHub, Kafka, file), suitable for isolated configurations and fine-grained data collection in different scenarios. - Platform For AI

PAI-Rec provides two logging mechanisms — debug logs and feature logs — to help you monitor and analyze your online recommendation service.

Log type	Purpose	Output targets
Debug log	Captures item data after the recall, filter, and coarse-ranking stages for pipeline analysis	Console, DataHub, Kafka, local file
Feature log	Captures online user-side and item-side features for feature analysis	DataHub

Debug log configuration

The debug log configuration corresponds to DebugConfs in the configuration overview. DebugConfs is a Map[string]object structure keyed by scenario name, allowing isolated configurations per scenario.

{
    "DebugConfs": {
        "${scene_name}": {
            "OutputType": "datahub",
            "Rate": 0,
            "DebugUsers": [
                "1001"
            ],
            "DatahubName": "dh_debug_log",
            "KafkaName": "pairec_debug_log",
            "FilePath": "/Users/username/pairec/debug_log/",
            "MaxFileNum": 20
        }
    }
}

Fields

Field	Type	Required	Description
`OutputType`	string	Yes	The output target for debug logs. Valid values: `console`, `datahub`, `kafka`, `file`.
`Rate`	int	Yes	The sampling ratio, from 0 to 100. Set to 0 to disable debug output. See Choosing a sampling ratio.
`DebugUsers`	[]string	No	Records logs only for the specified UIDs. For example, `"DebugUsers": ["1001"]` restricts logging to user 1001.
`DatahubName`	string	No	Required when `OutputType` is `datahub`. The custom name defined in `DatahubConfs` in your data source configuration.
`KafkaName`	string	No	Required when `OutputType` is `kafka`. The instance name defined in `KafkaConfs` in your data source configuration.
`FilePath`	string	No	Required when `OutputType` is `file`. The directory path for log output. The path is created automatically if it does not exist.
`MaxFileNum`	int	No	Applies when `OutputType` is `file`. The maximum number of log files in `FilePath`. Defaults to 20. Each file is capped at 1 G — when a file reaches the limit, file rotation occurs. When the file count exceeds `MaxFileNum`, the oldest file is deleted.

Choosing a sampling ratio

Online recommendation services typically handle high QPS. Because issues tend to repeat across requests, collecting every log entry is rarely necessary and adds overhead. Set Rate based on your service's QPS: lower values reduce volume for high-throughput services, while 100 captures all requests. Setting Rate to 0 disables debug output entirely.

Output log fields

Each debug log record contains the following fields.

Field	Type	Description
`request_id`	string	A unique ID for each recommendation request.
`module`	string	The pipeline stage that generated the log. Current values: `recall`, `filter`, `general_rank`.
`scene_id`	string	The scenario ID.
`exp_id`	string	The experiment ID.
`request_time`	integer	The request UNIX timestamp in seconds.
`uid`	string	The user ID.
`retrieveid`	string	The recall source ID.
`items`	string	The list of items for this recall source. See format details below.

`items` field format

item1:score1:{'dbmtl_prob_click':'0.03'},item2:score2:{'dbmtl_prob_click':'0.04'}

The meaning of score depends on the module value:

recall or filter: score is the recall score.
general_rank, rank, or sort: score is the model service score from the general rank or rank stage. The braces contain all model target scores — dbmtl_prob_click is one example; the actual names depend on your model service.

Within a module, records are grouped by retrieveid. For example, five recall sources produce five separate records when module is recall. The filter and general_rank modules follow the same pattern.

DataHub configuration

The engine automatically creates the DataHub topic on service startup. Once log output appears in the DataHub console, set up a subscription between MaxCompute and DataHub to persist data to a MaxCompute table.

{
    "DatahubConfs": {
        "dh_debug_log": {
            "Endpoint": "http://dh-cn-beijing-int-vpc.aliyuncs.com",
            "ProjectName": "project_test",
            "TopicName": "pairec_debug_log",
            "Schemas": [
                {"Field": "request_id", "Type": "string"},
                {"Field": "module", "Type": "string"},
                {"Field": "scene_id", "Type": "string"},
                {"Field": "request_time", "Type": "integer"},
                {"Field": "exp_id", "Type": "string"},
                {"Field": "items", "Type": "string"},
                {"Field": "retrieveid", "Type": "string"},
                {"Field": "uid", "Type": "string"}
            ]
        }
    }
}

Field	Type	Required	Description
`Endpoint`	string	Yes	The DataHub endpoint from the DataHub domain name list. Use a VPC endpoint if DataHub and PAI-Rec are in the same region; use a public endpoint otherwise.
`ProjectName`	string	Yes	The name of the DataHub project.
`TopicName`	string	Yes	The name of the DataHub topic.
`Schemas`	[]map	Yes	The DataHub topic schema definition.

Kafka configuration

Configure the instance name (matching KafkaName in DebugConfs), the BootstrapServers ingest endpoint, and the name of an existing Kafka topic.

{
    "KafkaConfs": {
        "pairec_debug_log": {
            "BootstrapServers": "alikafka-post-cn-xxxxx-1.alikafka.aliyuncs.com:9093,alikafka-post-cn-xxxxx-2.alikafka.aliyuncs.com:9093,alikafka-post-cn-xxxxx-3.alikafka.aliyuncs.com:9093",
            "Topic": "debug_log"
        }
    }
}

Important

When configuring through the PAI-Rec console and deploying to Elastic Algorithm Service (EAS), the virtual private cloud (VPC) and vSwitch in the network configuration must match those of your Kafka instance.

Feature log configuration

The feature log configuration is defined in FeatureLogConfs in the configuration overview. Like DebugConfs, it uses a Map[string]object structure keyed by scenario name for per-scenario isolation.

{
    "FeatureLogConfs": {
        "${scene_name}": {
            "OutputType": "datahub",
            "Rate": 10,
            "DatahubName": "",
            "UserFeatures": "",
            "ItemFeatures": ""
        }
    }
}

Fields

Field	Type	Required	Description
`OutputType`	string	Yes	The output target. Currently only `datahub` is supported.
`DatahubName`	string	Yes	The custom name defined in `DatahubConfs` in your data source configuration.
`Rate`	int	No	The sampling ratio, from 0 to 100. For legacy compatibility, omitting this field or setting it to 0 samples all data (equivalent to 100).
`UserFeatures`	string	No	A comma-separated list of user-side features to record. Use `*` to record all user features. If blank or omitted, no user features are recorded.
`ItemFeatures`	string	No	A comma-separated list of item-side features to record. Use `*` to record all item features. If blank, the recall ID and model scoring features are recorded.
`SplitUserItemLogs`	bool	No	When enabled, user features and item features are written as separate log records. By default, each record contains the full user feature set plus one item's features, which creates redundant user data at high QPS. Enable this field to eliminate that redundancy.

DataHub configuration

The engine automatically creates the DataHub topic on service startup.

{
    "DatahubConfs": {
        "dh_feature_log": {
            "Endpoint": "http://dh-cn-beijing-int-vpc.aliyuncs.com",
            "ProjectName": "",
            "TopicName": "pairec_feature_log",
            "Schemas": [
                {"Field": "request_id", "Type": "string"},
                {"Field": "scene_id", "Type": "string"},
                {"Field": "exp_id", "Type": "string"},
                {"Field": "request_time", "Type": "integer"},
                {"Field": "user_id", "Type": "string"},
                {"Field": "user_features", "Type": "string"},
                {"Field": "item_id", "Type": "string"},
                {"Field": "position", "Type": "string"},
                {"Field": "item_features", "Type": "string"}
            ]
        }
    }
}

Field	Type	Required	Description
`Endpoint`	string	Yes	The DataHub endpoint from the DataHub domain name list. Use a VPC endpoint if DataHub and PAI-Rec are in the same region; use a public endpoint otherwise.
`ProjectName`	string	Yes	The name of the DataHub project.
`TopicName`	string	Yes	The name of the DataHub topic.
`Schemas`	[]map	Yes	The DataHub topic schema definition.