EasyRec organizes feature engineering across three layers: data fields (raw input columns), feature fields (how models interpret those columns), and Feature Generator (FG) features (feature transformations that run identically offline and online, ensuring consistency between the two environments). Keeping these layers distinct lets you change how a raw column is interpreted — or how it is transformed — without touching the other layers.
Role of FG in offline training and online inference of EasyRec
Key concepts
FeatureStore: A feature management tool provided by Platform for AI (PAI) for storing and managing features in both offline and online systems. For more information, see FeatureStore overview.
Feature Generator (FG): A component that ensures consistency between offline and online feature processing. FG generates ID features, raw features, combo features, lookup features, match features, sequence features, and overlap features — with lookup features and sequence features being the most commonly used. For more information, see RTP FG.
EasyRec processor: Deployed on Elastic Algorithm Service (EAS) of PAI, this scoring service loads EasyRec deep learning models and performs performance optimization for recommendation, advertising, and search models. For more information, see EasyRec processor.
easyrec.conf: The EasyRec configuration file that describes data fields, feature types, and network structures used by the model.
fg.json: Describes the feature transformation process. Both offline and online systems run the same code from this file, which guarantees consistency between the two environments.
PAI-Rec: The recommendation engine that reads user features using a FeatureStore SDK.
How the pipelines work
Offline pipeline
FeatureStore uses UserViews (user-side feature views), ItemViews (item-side feature views), and a Label Table (MaxCompute tables with training labels) to build a training sample table.
FG — using the
fg_on_odps-1.3.59-jar-with-dependencies.jarpackage together withfg.json— transforms the training sample table into the result tablerank_sample_fg_encoded.PAI trains the model based on
easyrec.conf, then exports and stores it in Object Storage Service (OSS).
Online pipeline
The PAI-Rec engine fetches user features and the item IDs to score.
It requests the EasyRec processor to assemble features. The user features from requests and the item features in the cache are assembled for feature generation.
The EasyRec processor runs FG online to perform feature transformation using the same
fg.jsonlogic as the offline pipeline — this is what ensures offline/online consistency.The EasyRec processor scores the item IDs and returns the results.
Data fields and feature fields in easyrec.conf
EasyRec separates raw data description from model interpretation into two configuration blocks: data_config and feature_config.
data_config — describing raw input
data_config defines the names, types, and missing value defaults of raw input columns. Supported value types are int, double, and string. Data can come from CSV files, MaxCompute tables, or Apache Kafka data streams.
For full reference, see Data fields in EasyRec.
Example: key-value pair field with missing value imputation
input_fields: {
input_name: "prop_kv"
input_type: STRING
default_val: "-1024:0"
}The default_val of -1024:0 is a key-value pair used to impute missing values. During model training, missing values are replaced with this default when data is read. During online inference, missing values must be imputed before inference runs. If you use FG, configure missing value imputation in fg.json instead — it applies to both offline and online systems.
feature_config — interpreting fields for the model
feature_config specifies how each data field is parsed and used by the model. A single STRING field can be interpreted as an ID feature, a tag feature, or a sequence feature — the interpretation is entirely determined by the feature_type you set here.
For full reference, see feature_config.
ID feature
Use IdFeature for categorical identifiers such as user IDs or item IDs.
features {
input_names: "user_id"
feature_type: IdFeature
embedding_dim: 32
hash_bucket_size: 100000
}user_id values are hash-mapped to 100,000 buckets. Each bucket ID is then mapped to a 32-dimensional embedding vector during model training.
Raw feature
Use RawFeature for continuous numerical values such as click-through rates. Raw features are bucketed using boundaries before being mapped to embedding vectors.
features {
input_names: "ctr"
feature_type: RawFeature
boundaries: [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]
embedding_dim: 8
}The boundaries values partition the feature into discrete intervals. For ctr, this produces 11 intervals: (-inf, 0.1), [0.1, 0.2), ..., [0.9, 1.0). Each interval is assigned a bucket ID, which is then mapped to an 8-dimensional embedding vector.
Get boundaries from PAI's Binning component, which computes statistically meaningful split points from your training data. After configuring boundaries, set embedding_dim to convert the interval IDs into vectors.
Tag feature
Use TagFeature for multi-value categorical fields where a single record can carry multiple values separated by a delimiter.
features : {
input_names: "tags"
feature_type: TagFeature
separator: "|"
hash_bucket_size: 100000
embedding_dim: 24
}For example, an article with tags Entertainment|Funny|Popular is split on | into three values. Each value is hash-embedded into a 24-dimensional vector, and then average pooling produces a single average embedding vector for the field.
Lookup feature transformation in FG
FG supports several feature transformation types. Lookup feature transformation is one of the most commonly used: it queries a user-side map with an item-side key to generate a cross feature.
Example: user brand click count for the current item
{
"map": "user:map_brand_click_kv",
"key": "item:brand",
"feature_name": "map_brand_click_count",
"feature_type": "lookup_feature",
"needDiscrete": false
}This configuration uses item:brand as the lookup key to query user:map_brand_click_kv (the user's click counts across different brands) and produces map_brand_click_count — the number of times the user clicked on items of the current item's brand.
Offline: The
fg_on_odps-1.3.59-jar-with-dependencies.jarpackage executes this transformation as a MapReduce job.Online: The EasyRec processor runs FG to compute
map_brand_click_countand passes the value to the TensorFlow model for inference.
FAQ
Where do the boundaries values come from?
Run PAI's Binning component on your training data to compute split points. The component performs discretization and outputs a set of boundary values that reflect your data's distribution.
Once you have the boundaries, add them to feature_configs in your easyrec.conf:
feature_configs: {
input_names: "CTR"
feature_type: RawFeature
boundaries: [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]
embedding_dim: 8
}The boundaries create 11 intervals — (-inf, 0.1), [0.1, 0.2), through [0.9, 1.0) — and each interval is converted to an embedding vector using embedding_dim.
How does FG keep offline and online feature transformations consistent?
Both the offline MapReduce job and the online EasyRec processor read the same fg.json file and run the same transformation code. There is no separate offline and online implementation — consistency comes from using a single code path.
Where do I configure missing value imputation?
If you are not using FG, set the default_val parameter for each field in data_config of your easyrec.conf. Missing values are replaced with this default when data is read during training, and must be imputed before online inference runs.
If you are using FG, configure missing value imputation in fg.json. This single configuration applies to both offline and online systems.