:Feature generation

Feature generation focuses only on transformations required for both offline and online sample generation. If a transformation operation is required only in the offline stage, do not define it as an FG operation. The following diagram shows the position of the FG module in a recommendation system architecture.

The feature generation process consists of a series of feature operators (FG operators) that are executed in parallel according to the topological order of a directed acyclic graph (DAG) defined in the configuration file.

Configuration file example

Configure feature operators in the features list. Each feature operator must include the feature_name and feature_type parameters. For other configuration parameters, see Built-in feature operators.

The reserves parameter specifies the fields to be passed through from an offline task, which are output as-is without feature transformation.

{
  "features": [
    {
      "feature_name": "goods_id",
      "feature_type": "id_feature",
      "value_type": "string",
      "expression": "item:goods_id",
      "default_value": "-1024",
      "need_prefix": false
    },
    {
      "feature_name": "color_pair",
      "feature_type": "combo_feature",
      "value_type": "string",
      "expression": ["user:query_color", "item:color"],
      "default_value": "",
      "need_prefix": false
    },
    {
      "feature_name": "current_price",
      "feature_type": "raw_feature",
      "value_type": "double",
      "expression": "item:current_price",
      "default_value": "0",
      "need_prefix": false
    }, 
    {
      "feature_name": "usr_cate1_clk_cnt_1d",
      "feature_type": "lookup_feature",
      "map": "user:usr_cate1_clk_cnt_1d",
      "key": "item:cate1",
      "need_discrete": false,
      "need_key": false,
      "default_value": "0",
      "combiner": "max",
      "need_prefix": false,
      "value_type": "double"
    },
    {
      "feature_name": "recommend_match",
      "feature_type": "overlap_feature",
      "method": "is_contain",
      "query": "user:query_recommend",
      "title": "item:recommend",
      "default_value": "0"
    },
    {
      "feature_name": "norm_title",
      "feature_type": "text_normalizer",
      "expression": "item:title",
      "max_length": 512,
      "parameter": 0,
      "remove_space": false,
      "is_gbk_input": false,
      "is_gbk_output": false
    },
    {
      "feature_name": "title_terms",
      "feature_type": "tokenize_feature",
      "expression": "feature:norm_title",
      "default_value": "",
      "vocab_file": "tokenizer.json",
      "output_type": "word_id",
      "output_delim": ","
    },
    {
      "feature_name": "query_title_match_ratio",
      "feature_type": "overlap_feature",
      "method": "query_common_ratio",
      "query": "user:query_terms",
      "title": "feature:title_terms",
      "default_value": "0"
    },
    {
      "feature_name": "title_term_match_ratio",
      "feature_type": "overlap_feature",
      "method": "title_common_ratio",
      "query": "user:query_terms",
      "title": "feature:title_terms",
      "default_value": "0"
    },
    {
      "feature_name": "term_proximity_min_cover",
      "feature_type": "overlap_feature",
      "method": "proximity_min_cover",
      "query": "user:query_terms",
      "title": "feature:title_terms",
      "default_value": "0"
    }
  ],
  "input_alias": {
    "non_exist_field1": "exist_field1",
    "non_exist_field2": "exist_field2"
  },
  "reserves": [
    "request_id",
    "user_id",
    "is_click",
    "is_pay",
    "sample_weight",
    "event_unix_time"
  ]
}

Special configuration item input_alias: A dictionary that maps a potentially non-existent input field name to an actual field name. (The input_alias configuration is supported in version 1.0.0 and later. You can usually skip this configuration.)

• Use case 1: Set a shorter alias for a long field name.

• Use case 2: Set an alias for the second parameter when a custom feature operator uses the same input for two different parameters.

The same input field can be reused across different features, but cannot be reused within a single feature transformation. You can configure input_alias to bypass this restriction.

• For example, if a custom feature operator has two input parameters that both require the same field A, you can configure two inputs, A and B, and an input_alias to map "B": "A". At runtime, the parameters for the custom feature operator are changed from (A, B) to (A, A).

Input domains

An input domain indicates the entity from which an input originates. The following four types are supported:

• user: User-side features, including user profiles and user-level statistical features.

• context: Contextual features that change with each request, such as time, location, and weather.

• item: Item-side features, including static content features and item-level statistical features.

• feature: The output of another feature operator.

The feature input domain is special; it configures dependencies between feature operators. Collectively, all feature operators form a directed acyclic graph (DAG). The framework executes these feature transformation operations in parallel according to the topological order. The topology is shown in the following figure.

By default, the output of intermediate nodes in a DAG is not used as the output of the FG. You can use the stub_type feature configuration parameter to change this behavior.

Multi-value types and separators

FG supports complex input types, such as Array and Map, which are consistent with the complex types in MaxCompute.

String-type multi-valued features can use chr(29) as a delimiter.

For example, in v1^]v2^]v3, ^] is the multi-value separator. It is a single character with the ASCII code "\x1D", not two characters. To enter this character, press C-q C-5 in emacs or C-v C-5 in vi.

Feature binning (discretization)

The framework supports the following six types of binning operations:

• hash_bucket_size: Hashes the feature transformation result and applies a modulo operation.

• vocab_list: Maps the feature transformation result to an index in a list.

• vocab_dict: Maps the feature transformation result to a value in a dictionary. The value must be convertible to the int64 type.

• vocab_file: Reads a vocab_list or vocab_dict from a file.

• boundaries: Converts the feature transformation result to a corresponding bucket ID based on specified boundaries.

• num_buckets: Uses the feature transformation result directly as the bucket ID.

hash_bucket_size

Hashes the transformation result and applies a modulo operation. This method is applicable to any feature value type.

• Result range: [0,hash_bucket_size)

• The feature binning result for an empty value is hash(default_value)%hash_bucket_size.

{
  "hash_bucket_size": 128000,
  "default_value": "default_value"
}

vocab_list

Bins the input by mapping a feature value to its corresponding index in the vocab_list array.

• The element type of the vocab_list array must be the same as the value_type configuration.

• num_oov_bucket: Non-negative integer, the number of out-of-vocabulary buckets.

  • All out-of-vocabulary inputs will be assigned IDs in the range [vocabulary_size, vocabulary_size+num_oov_buckets) based on a hash of the input value.

  • A positive num_oov_buckets cannot be specified withdefault_bucketize_value.

• default_bucketize_value: The integer ID value to return for out-of-vocabulary feature values.

  • You cannot specify this when num_oov_buckets is positive.

  • The default value is vocab_list.size().

{
  "vocab_list": [
    "",
    "<OOV>",
    "token1",
    "token2",
    "token3",
    "token4"
  ],
  "num_oov_bucket": 0,
  "default_bucketize_value": 1
}

vocab_dict

The binning result is the value in the vocab_dict dictionary that corresponds to the feature value. This supports mapping different feature values to the same binning result.

• The data type of the keys in the vocab_dict dictionary must be the same as the value_type configuration.

• The value of vocab_dict must be convertible to the int64 type.

• num_oov_bucket: Non-negative integer, the number of out-of-vocabulary buckets.

  • All out-of-vocabulary inputs will be assigned IDs in the range [vocabulary_size, vocabulary_size+num_oov_buckets) based on a hash of the input value.

  • A positive num_oov_buckets cannot be specified withdefault_bucketize_value.

• default_bucketize_value: The integer ID value to return for out-of-vocabulary feature values.

  • This cannot be specified with a positivenum_oov_buckets.

  • The default value is vocab_dict.size().

{
  "vocab_dict": {
    "token1": 1,
    "token2": 2,
    "token3": 3,
    "token4": 1
  },
  "num_oov_bucket": 0,
  "default_bucketize_value": 4
}

vocab_file

Loads a vocab_list or vocab_dict from a file.

{
  "vocab_file": "vocab.txt",
  "num_oov_bucket": 0,
  "default_bucketize_value": 4
}

• vocab_file: The path to the vocabulary file. The file contains a vocabulary, with one term per line. You can optionally specify a mapping value.

  • Relative paths are supported. When you deploy the online service, it must be placed in the same directory as fg.json.

  • If only a token is present, it is mapped to the line number (starting from 0). If a value is present, the token and the value are separated by a whitespace character (a space or a tab). The value must be of the int64 type.

• num_oov_bucket and default_bucketize_value have the same meaning as described above.

boundaries

Buckets numerical features based on specified bin boundaries.

• The element type of the boundaries array must be the same as the value_type configuration.

• Buckets include the left boundary and exclude the right boundary.

• For example, boundaries=[0., 1., 2.] generates buckets (-inf, 0.), [0., 1.), [1., 2.), and [2., +inf).

{
  "boundaries": [0.0, 1.0, 2.0],
  "default_value": -1
}

num_buckets

Uses the feature transformation result directly as the bucket ID. This method is suitable for feature values that can be converted to an integer.

• Result range: [0,num_buckets)

• If the feature value is outside the configured range, it is assigned default_bucketize_value.

{
  "num_buckets": 128000,
  "default_bucketize_value": 127999
}

Built-in feature operators

Configuration methods vary among feature operators. All feature operators that can be leaf nodes in the DAG support feature binning.

For more information, see Built-in feature operators.

Operator combinations

By configuring a DAG, you can combine various built-in operators to perform powerful feature transformations.

Example 1: Average the first 4 sequence elements

{
  "features": [
    {
      "feature_name": "top_n_prices",
      "feature_type": "sequence_raw_feature",
      "expression": "user:clk_prices",
      "separator": ",",
      "sequence_length": 4,
      "stub_type": true
    },
    {
      "feature_name": "top_n_avg_price",
      "feature_type": "expr_feature",
      "expression":"reduce_mean(top_n_prices)",
      "default_value": "-1",
      "variables":["feature:top_n_prices"]
    }
  ]
}

Example 2: Average sequence elements with a condition

{
  "features": [
    {
      "feature_name": "valid_list",
      "feature_type": "expr_feature",
      "expression":"clk_times < 10",
      "variables":["user:clk_times"],
      "value_dimension": 5
    },
    {
      "feature_name": "top_n_prices",
      "feature_type": "bool_mask_feature",
      "expression": ["user:clk_prices", "feature:valid_list"],
      "value_type": "float",
      "separator": ","
    },
    {
      "feature_name": "top_n_avg_price",
      "feature_type": "expr_feature",
      "expression":"reduce_mean(top_n_prices)",
      "default_value": "-1",
      "variables":["feature:top_n_prices"]
    }
  ]
}

Note: In the preceding example, clk_prices and clk_times are two parallel sequences.

Custom feature operators

Custom feature operators can be dynamically loaded and executed by the framework as plugins.

For more information, see Custom feature operators.

Performance optimization

The performance of the FG module is highly dependent on its configuration. The general principle is to minimize unnecessary data (feature) transformations.

If you can process and transform data in the offline or near-line stages, do not perform them in the FG stage (online scoring service).

Follow these guidelines for better performance:

• For structured input data, prioritize using complex types from MaxCompute tables (for example, Map and Array) instead of the STRING type to reduce string parsing overhead.

  • In online scoring services (such as EasyRec Processor or TorchEasyRec Processor), use FeatureStore and FeatureDB as the online storage to enable support for complex types.

  • For a lookup_feature, using the Map type for the map field is highly recommended.

  • For sequence_feature, overlap_feature, and bm25_feature, using Array-type inputs is highly recommended.

  • Avoid using match_feature because it does not support complex types. Use lookup_feature instead by combining pkey and skey.

• Avoid the overhead of data type conversion.

  • The value_type of raw_feature should not be set to a type other than float without a specific reason.

  • For a lookup_feature, ensure that the key type of the Map<Key, Value> input matches the type of the query field.

  • If you configure feature binning of the num_buckets type, value_type must be set to int64.

  • If the optimal type for a data column varies across different scenarios, consider adding a copy of the column with a different type.

    • For example, a field must be a BIGINT when used as the lookup field for lookup_feature, but a STRING when used as part of combo_feature.

    • In this case, add a copy of the column for each required type: one BIGINT and one STRING. The following is sample SQL code:

      • SELECT int_data, int_data as str_data FROM ....

• Reuse shared logic and computations whenever possible by using feature dependencies (DAG mode).

Global configuration

Note: The preceding configurations must be consistent across all execution environments, including offline and online, as well as for training and inference. Otherwise, inconsistencies may occur between online and offline scoring.

Hash collision rate

The following shows the test results on a dataset with 26 features of different cardinalities, where the hash_bucket_size for each feature is set to 10 × its cardinality:

In summary, we recommend using std::hash + MultiplicativeHash as a combined approach to optimize model performance. std::hash is enabled by default. MultiplicativeHash is disabled by default for backward compatibility, and you must manually enable it by following the instructions below.

Additionally, CityHash is a method that theoretically offers better uniformity, but it did not show a significant advantage on this dataset. You can test it further on your own dataset.

Online scoring service configuration

Configure these settings by using server-side environment variables. Specifically, you can set them in the service configuration of EasyRec Processor or TorchEasyRec Processor.

{
  "processor_envs": [
    {
      "name": "USE_MULTIPLICATIVE_HASH",
      "value": "true"
    }
  ]
}

from pyfg105 import run_on_odps

fg_task = run_on_odps.FgTask(...)
fg_task.add_fg_setting('USE_CITY_HASH_TO_BUCKETIZE', 'false')
fg_task.add_fg_setting('USE_MULTIPLICATIVE_HASH', 'true')
fg_task.run(o)

import pyfg
pyfg.set_env('USE_MULTIPLICATIVE_HASH', 'true')
pyfg.set_env('USE_CITY_HASH_TO_BUCKETIZE', 'false')