:Built-in feature operators

Introduction

id_feature is a discrete feature. This includes single-value features, such as user IDs and item IDs, and multi-value features, such as item colors.

Configuration

{
  "feature_type": "id_feature",
  "feature_name": "item_is_main",
  "expression": "item:is_main",
  "need_prefix": true,
  "separator": "\u001D",
  "default_value": ""
}

• Supports feature binning. See Feature binning (discretization) for details.

• Supports the array type for multi-value features.

Examples

The following examples show the input and output for the item:is_main feature based on different configurations.

^] represents the multi-value separator, a symbol whose ASCII code is "\x1D" and can also be written as "\u001d".

Overview

The raw_feature operator processes a continuous feature. It supports numeric types such as int, float, and double.

Configuration

{
 "feature_type" : "raw_feature",
 "feature_name" : "ctr",
 "expression" : "item:ctr",
 "normalizer" : "method=log10"
}

• This operator supports feature binning/discretization. For configuration details, see Feature binning (discretization).

• This operator supports multi-value inputs of the array type.

Example

^] represents the multi-value separator. Note that this is a single character with the ASCII code "\x1D", not two characters.

Normalizer

Both raw_feature and match_feature support four types of normalizers: minmax, zscore, log10, and expression. Their configuration and calculation methods are as follows:

• minmax

  Example: method=minmax,min=2.1,max=2.2

  Formula: x = (x - min) / (max - min)

• zscore

  Example: method=zscore,mean=0.0,standard_deviation=10.0

  Formula: x = (x - mean) / standard_deviation

• log10

  Example: method=log10,threshold=1e-10,default=-10

  Formula: x = x > threshold ? log10(x) : default;

• expression

  Example: method=expression,expr=sign(x)

  Formula: You can configure any function or expression. The variable x represents the input value.

Overview

The expr_feature evaluates an expression and outputs the result as a specified type, such as float, double, int32, or int64. It supports batch computing and broadcasting.

Note: When you use this feature operator, all inputs must be convertible to the double type.

Configuration

{
  "feature_type" : "expr_feature",
  "feature_name" : "ctr_sigmoid",
  "value_type": "float",
  "expression" : "sigmoid(pv/(1+click))",
  "variables": ["item:pv", "item:click"]
}

When pv = 2, click = 3, the value of the expression feature is 0.6224593312.

Examples

{
    "feature_name": "expr_feat",
    "feature_type": "expr_feature",
    "value_type": "float",
    "expression": "a+b",
    "variables": ["a", "b"],
    "value_dimension": 3
}

• Scalar and vector computation (broadcasting)

  • When a=1 and b=[1, 2, 6], the result is [2, 3, 7].

• Vector-vector element-wise calculation

  • When variable a=[3, 2, 1] and variable b=[1, 2, 6], the result is [4, 4, 7].

• Temporary variables and comma expressions

  • For example: x=roundp(a),(a-x)*b. In this example, x is a temporary variable and does not need to be configured in variables.

  • A comma expression evaluates from left to right and returns the value of its rightmost sub-expression.

  • To reduce memory overhead, reuse existing variables as temporary variables where permitted by the semantics.

Combine expression and sequence features

{
  "features": [
    {
      "feature_name": "sphere_distance",
      "feature_type": "expr_feature",
      "expression": "sphere_dist(click_id_lng,click_id_lat,j_lng,j_lat)",
      "variables": ["user:click_id_lng", "user:click_id_lat", "item:j_lng", "item:j_lat"],
      "default_value": "0",
      "value_dimension": 3,
      "stub_type": true
    },
    {
      "feature_name": "time_diff",
      "feature_type": "expr_feature",
      "variables": ["user:cur_time", "user:clk_time_seq"],
      "expression": "cur_time-clk_time_seq",
      "default_value": "0",
      "separator": ";",
      "value_dimension": 3,
      "stub_type": true
    },
    {
      "sequence_name": "click_seq",
      "sequence_length": 3,
      "sequence_delim": ";",
      "sequence_pk": "user:click_item",
      "features": [
        {
          "feature_name": "spherical_distance",
          "feature_type": "raw_feature",
          "expression": "feature:sphere_distance",
          "default_value": "0.0"
        },
        {
          "feature_name": "time_diff_seq",
          "feature_type": "id_feature",
          "expression": "feature:time_diff",
          "default_value": "0.0",
          "num_buckets": 10000
        }
      ]
    }
  ]
}

Expressions

• Built-in functions (scalar)

  Function name	Number of parameters	Description
  rnd	0	Generates a random number in the range [0, 1).
  isnan	1	Returns 1.0 if the input is NaN, and 0.0 otherwise. This function requires version 1.0.5 or later.
  sin	1	Returns the sine of a number.
  cos	1	Returns the cosine of a number.
  tan	1	Returns the tangent of a number.
  asin	1	Returns the arcsine of a number.
  acos	1	Returns the arccosine of a number.
  atan	1	Returns the arctangent of a number.
  sinh	1	Returns the hyperbolic sine of a number.
  cosh	1	Returns the hyperbolic cosine of a number.
  tanh	1	Returns the hyperbolic tangent of a number.
  asinh	1	Returns the inverse hyperbolic sine of a number.
  acosh	1	Returns the inverse hyperbolic cosine of a number.
  atanh	1	Returns the inverse hyperbolic tangent of a number.
  log2	1	Returns the base-2 logarithm of a number.
  log10	1	Returns the base-10 logarithm of a number.
  log	1	Returns the natural logarithm (base e) of a number.
  ln	1	Returns the natural logarithm (base e) of a number.
  exp	1	Raises Euler's number (e) to the power of a number.
  sqrt	1	Returns the square root of a number.
  sign	1	Returns the sign of a number: -1 for negative, 1 for positive, or 0 for zero.
  abs	1	Returns the absolute value of a number.
  rint	1	Rounds a number to the nearest integer.
  round	1	Rounds a number to the nearest integer, using the "round half away from zero" method.
  roundp	2	Rounds a number to a specified precision. For example, roundp(3.14159, 2) returns 3.14.
  mod	2	Returns the remainder of a division.
  floor	1	Rounds a number down to the nearest integer.
  ceil	1	Rounds a number up to the nearest integer.
  trunc	1	Truncates a number by removing its fractional part.
  sigmoid	1	Returns the sigmoid of a number.
  sphere_dist	4	Returns the spherical distance between two GPS points. Arguments: lng1, lat1, lng2, lat2.
  haversine	4	Returns the Haversine distance between two GPS points. Arguments: lng1, lat1, lng2, lat2.
  min	Variable	Returns the minimum value from a list of arguments.
  max	Variable	Returns the maximum value from a list of arguments.
  sum	Variable	Returns the sum of all arguments.
  avg	Variable	Returns the average value of all arguments.

  Note: These built-in functions support batch computing and broadcasting.

• Built-in vector operation functions

  Function name	Number of parameters	Description
  len	1	Returns the length (number of elements) of a vector.
  l2_norm	1	Returns the L2-normalized vector.
  squared_norm	1	Returns the squared L2 norm of a vector.
  dot	2	Returns the dot product of two vectors.
  euclid_dist	2	Returns the Euclidean distance between two vectors.
  corr	2	Returns the Pearson correlation coefficient between two vectors.
  std_dev	1	Returns the sample standard deviation of a vector (divides by n-1).
  pop_std_dev	1	Returns the population standard deviation of a vector (divides by n).
  variance	1	Returns the sample variance of a vector (divides by n-1).
  pop_variance	1	Returns the population variance of a vector (divides by n).
  reduce_min	1	Returns the minimum value in a vector.
  reduce_max	1	Returns the maximum value in a vector.
  reduce_sum	1	Returns the sum of all elements in a vector.
  reduce_mean	1	Returns the average value of all elements in a vector.
  reduce_prod	1	Returns the product of all elements in a vector.

  Note: If an expression contains one of these built-in vector operation functions, all other variables in the expression must be scalars.

• Built-in binary operators

  Operator	Description	Priority
  =	Assignment *	0
  ||	Logical OR	1
  &&	Logical AND	2
  |	Bitwise OR	3
  &	Bitwise AND	4
  <=	Less than or equal to	5
  >=	Greater than or equal to	5
  !=	Not equal to	5
  ==	Equal to	5
  >	Greater than	5
  <	Less than	5
  +	Addition	6
  -	Subtraction	6
  *	Multiplication	7
  /	Division	7
  %	Modulo	7
  ^	Raises x to the power of y.	8

  * The assignment operator is special: it modifies one of its arguments and applies only to variables.

• Built-in ternary operator

  This operator provides if-else functionality.

  Lazy evaluation evaluates only the required branch of an expression.

  Operator	Description	Syntax
  ?:	If-then-else operator	condition ? value_if_true : value_if_false

• Built-in constants

  Constant	Description	Value
  _pi	The mathematical constant pi (π).	3.141592653589793
  _e	The mathematical constant e, also known as Euler's number.	2.718281828459045

Overview

The combo_feature operator creates a synthetic feature by calculating the cartesian product of multiple source fields. This process is also known as feature crossing. The id_feature operator is a special case of combo_feature that involves only a single source field. Typically, the source fields for feature crossing originate from different tables, such as combining a user feature with an item feature.

Configuration

{
  "feature_type" : "combo_feature",
  "feature_name" : "comb_age_item",
  "expression" : ["user:age_class", "item:item_id"],
  "need_prefix": true,
  "separator": "\u001D",
  "default_value": ""
}

• Supports feature binning. For configuration instructions, see feature binning (discretization).

• Supports multi-value inputs of the array type.

Examples

The ^] symbol represents the multi-value separator, which is a single character with the ASCII code "\x1D", not two characters.

The number of output values is calculated as follows:

|F1| * |F2| * ... * |Fn|

Where |Fn| represents the number of values in the nth source field.

Overview

Similar to match_feature, the lookup_feature operator finds a key within a set of key-value pairs and returns the corresponding value.

This operator depends on two parameters: map and key.

• The map parameter is a dictionary or a multi-value string. In a multi-value string, each element is a key-value pair in the "k1:v1" format.

• The key parameter can be of any data type. For multiple keys, an array is the recommended input type. To generate a feature, the operator retrieves the value for the key parameter, converts it to the key type of the map, and then finds the corresponding value in the map.

Configuration

{
  "feature_type": "lookup_feature",
  "feature_name": "item_match_item",
  "map": "item:item_attr",
  "key": "item:item_value",
  "need_discrete": true,
  "need_key": true
}

• Supports binning operations. For the configuration method, see feature binning (discretization).

• The map parameter supports dictionary-type inputs, and the key parameter supports array-type inputs.

Example

For the configuration above, assume a document has the following data:

item_attr : "k1:v1^]k2:v2^]k3:v3"

The ^] symbol represents the multi-value separator. It is a single character with the ASCII code "\x1D", not two characters. To enter this character, press C-q C-5 in emacs or C-v C-5 in vim. Here, item_attr is a multi-value string.

When the map parameter represents multiple key-value pairs as a string, it must be a multi-value string, not a standard string.

item_value : "k2"

The resulting feature is item_match_item_k2_v2.

feature_name: fg
map: {"k1:123", "k2:234", "k3:3"}
key: {"k1"}
Result: feature={"fg_123"}

map: {"k1:123", "k2:234", "k3:3"}
key: {"k1"}
Result: feature={123}

Combine lookup results

When you provide multiple keys, you can use the combiner parameter to aggregate the retrieved values.

To use the combiner, you must set need_discrete to false. In this case, the looked-up values must be numeric or strings convertible to numbers.

Overview

The match_feature operator is typically used for feature-to-feature matching. At its core, it performs a two-level map lookup.

Configuration

The configuration file uses the JSON format.

{
  "feature_name": "user__l1_ctr_1",
  "feature_type": "match_feature",
  "category": "ALL",
  "need_discrete": false,
  "item": "item:category_level1",
  "user": "user:l1_ctr_1",
  "match_type": "hit"
}

• user: A nested dictionary.

  • The user field uses a string to represent the two-level map.

  • For the first-level map, | is the separator between items, and ^ is the separator between a key and its value.

  • For the second-level map, , is the separator between items, and : is the separator between a key and its value.

  • The first-level dictionary also supports a Map<K, string> input, where K can be of the types string,int32,int64. The map's value is a string that represents the inner dictionary and uses the same separators.

• category: The primary key for the first-level map lookup.

  ALL is a wildcard character that matches all keys at this level.

• item: The secondary key for the second-level map lookup.

  ALL is a wildcard character that matches all keys at this level.

• need_discrete

  • true: The model uses the generated feature name from the output and ignores the feature value. By default, this parameter is false.

  • false: The model uses the matched feature value from the output and ignores the feature name.

• match_type

  • hit: Outputs a single matched feature. The operator first queries the first-level map using the category value, and then queries the resulting second-level map using the item value to retrieve the final value. To perform single-level matching, set the first-level map key to ALL and set the category parameter in the feature generation (FG) configuration to "ALL".

  • multihit: Matches and outputs multiple values when the ALL wildcard character is used in the category and item fields.

• normalizer

  The normalization method. This parameter has the same meaning as in the raw_feature operator and takes effect only when need_discrete=false.

• show_category

  Specifies whether to prefix the output with the category value. Defaults to true if need_discrete=true and match_type=hit, and false otherwise.

• show_item

  Specifies whether to prefix the output with the item value. Defaults to true if need_discrete=true and match_type=hit, and false otherwise.

• value_type

  The output data type. Defaults to string.

• separator

  Optional. The multi-value separator for the string-type key field. Defaults to "\u001D".

• default_value

  Optional. The default value for null inputs.

• value_dimension

  Optional. Defines the output's data structure. Defaults to 0. If set to 1, the output is a single value of type value_type; otherwise, it is an array of type array<value_type>.

• stub_type

  Optional. If set to true, the output is treated as an intermediate result and not sent to the model. Default: false.

Examples

{
  "50011740": {
    "50011740": 0.2,
    "36806676": 0.3,
    "122572685": 0.5
  },
  "50006842": {
    "16788": 0.1
  }
}

{
  "feature_name": "brand_hit",
  "feature_type": "match_feature",
  "category": "item:auction_root_category",
  "need_discrete": true,
  "item": "item:brand_id",
  "user": "user:user_brand_tags_hit",
  "match_type": "hit"
}

Overview

Outputs information on term matches between two strings. For example, you can use this feature to determine if a query is in an item's title.

The paper "An Exploration of Proximity Measures in Information Retrieval" describes the calculation methods for these features.

Assume the term sequence of a title (document) is: t1,t2,t1,t3,t5,t4,t2,t3,t4

• MinCover measures the length of the shortest document segment that covers each query term at least once.

• MinDist (Minimum pairwise distance): The minimum distance found between any pair of matching query terms. For example, if a query Q=t1,t2,t3 has pairwise distances of 1, 2, and 3 in a document, the MinDist is min(1,2,3)=1.

• MaxDist (Maximum pairwise distance): The maximum distance found between any pair of matching query terms. For the same example, MaxDist=max(1,2,3)=3.

• AveDist (Average pairwise distance): The average of the pairwise distances between all matching query terms. For the same example, AveDist=(1+2+3)/3=2.

Note that all aggregate operators (MinDist, MaxDist, and AveDist) are defined based on the pairwise distances between matching query terms. If a document contains only one matching query term, the value for MinDist, AveDist, and MaxDist is the length of the document.

Configuration

{
  "feature_type" : "overlap_feature",
  "feature_name" : "is_contain",
  "query" : "user:attr1",
  "title" : "item:attr2",
  "method" : "is_contain",
  "separator" : " ",
  "normalizer" : ""
}

The overlap_feature returns a float value.

Example 1

Given a query of "high,high2,fiberglass,abc" and a title of "high,quality,fiberglass,tube,for,golf,bag":

Example 2

method=index_of, title=the cat sat on the mat.

Overview

A user's historical behavior is a critical feature. This behavior is typically represented as a sequence, such as a series of clicks or purchases. The entities that make up the sequence can be items themselves or their attributes.

Configuration

For example, to process a user's click sequence with a maximum length of 50, you can extract the item_id, price, and ts features for each item. Here, ts is the difference between the request time (request_time) and the event time (event_time). The following example shows the configuration.

{
  "sequence_name": "click_50_seq",
  "sequence_length": 50,
  "sequence_delim": ";",
  "sequence_pk": "user:click_50_seq",
  "features": [
    {
        "feature_name": "item_id",
        "feature_type": "id_feature",
        "value_type": "string",
        "expression": "item:item_id"
    },
    {
        "feature_name": "price",
        "feature_type": "raw_feature",
        "expression": "item:price"
    },
    {
        "feature_name": "ts",
        "feature_type": "raw_feature",
        "expression": "user:ts"
    },
    {
      "feature_name": "time_diff_seq",
      "feature_type": "custom_feature",
      "operator_name": "SeqExpr",
      "operator_lib_file": "3rdparty/lib64/libseq_expr.so",
      "expression": ["user:cur_time", "user:clk_time_seq"],
      "formula": "cur_time - clk_time_seq",
      "sequence_fields": ["clk_time_seq"],
      "default_value": "0",
      "value_type": "double",
      "is_op_thread_safe": false,
      "value_dimension": 1
    }
  ]
}

• sequence_name: The name of the sequence.

• sequence_length: The maximum length of the sequence.

• sequence_delim: The delimiter that separates elements in the sequence.

• sequence_pk: The sequence primary key. An example is user:click_50_seq, which stores the 50 most recent itemIDs clicked by a user. The model inference service uses this field as a key to query side info.

  • The request parameters for the Online Inference Service (EAS Processor) must include a feature whose key is sequence_pk.

    • For example: click_50_seq: 5410233389955966;1832586 (The separator is the value specified for sequence_delim)

      • In the example above, the value of the click_50_seq feature is 5410233389955966;1832586

  • Item-side sub-features of the sequence do not need to be included in the request to the model inference service.

    • The model inference service uses this field as a key to query the item's side info.

    • For example, in this configuration, the item_id, price sequence features are not required in the request to the inference service. Instead, they are retrieved from the item cache of the Processor and concatenated by using the fg SDK. This process ensures that the format is consistent with the one used during offline training.

  • User-side sub-features of the sequence must be included in the request to the model inference service.

    • The feature name is ${sequence_name}__${input_name}, for example: click_50_seq__ts.

    • ${input_name} is typically configured with the expression option, but the configuration may vary for different sub-feature types. ${input_name} does not include an input domain prefix (such as item: or user:).

• Features: The side info of a sequence, which includes information such as an item's static attribute values and behavioral time information.

  • sequence_fields: Specifies the field names of the input sequence. The value is a string or a [string] array.

    • When a feature operator has only one input field, the content of that field must be a sequence. In this case, you do not need to configure sequence_fields.

    • When a feature operator has multiple input fields, if you do not configure sequence_fields, all item-side features (such as item:XXX) are assumed to be sequence input fields.

  • The input table for an offline task must include columns corresponding to all sub-features.

    • When the column is a sequence (see the rules for sequence_fields), it is named ${sequence_name}__${input_name}.

      • For example, in this configuration, the offline table requires 4 columns: click_50_seq__item_id, click_50_seq__price, click_50_seq__ts, and click_50_seq__clk_time_seq.

      • The recommended type for a column in an offline table is array (for better performance), while the string type using sequence_delim as the element separator is also supported.

    • If the column is not a sequence, name it ${input_name} without a prefix.

      • For example, in this configuration, the offline table requires one non-sequence column: ${cur_time}

    • You can use the global configuration input_alias to set a shorter alias for a long column name (see the example below).

  • Feature binning is supported. For configuration details, see Feature binning (discretization). When binning is configured, the output element type is int64, and the shape is determined by the value_dimension configuration.

  • value_dimension (or value_dim): The dimension of each element in the Sequence. For a sequence_raw_feature, the output type is 1 when this parameter is set to array<float>, and array<array<float>> for other values. For a sequence_id_feature, the output type is array<string> when this parameter is set to 1, and array<array<string>> for other values. The default value is 0.

Any feature type can be configured as a sequence sub-feature. The following is an example:

{
  "features": [
    {
      "sequence_name": "common_seq",
      "sequence_length": 50,
      "sequence_delim": ";",
      "sequence_pk": "user:click_50_seq",
      "features": [
        {
          "feature_name": "item_id",
          "feature_type": "id_feature",
          "value_type": "String",
          "expression": "item:item_id",
          "value_dimension": 1
        },
        {
          "feature_name": "price",
          "feature_type": "raw_feature",
          "expression": "item:price"
        },
        {
          "feature_name": "ts",
          "feature_type": "raw_feature",
          "expression": "user:ts"
        },
        {
          "feature_name": "expr_feat",
          "feature_type": "expr_feature",
          "expression": "a > b",
          "variables": ["item:a", "item:b"],
          "sequence_fields": "a",
          "default_value": "0",
          "value_dimension": 1
        },
        {
          "feature_name": "lookup_feat",
          "feature_type": "lookup_feature",
          "map": "user:dict",
          "key": "item:prop",
          "separator": ",",
          "default_value": "0",
          "value_type": "float",
          "combiner": "sum",
          "boundaries": [0.0, 0.15, 0.5]
        },
        {
          "feature_name": "match_feat",
          "feature_type": "match_feature",
          "user": "user:nested_dict",
          "category": "item:pkey",
          "item": "item:skey",
          "separator": "\u001D",
          "default_value": "0",
          "matchType": "hit",
          "value_type": "float",
          "value_dimension": 1
        },
        {
          "feature_name": "bm25_score",
          "feature_type": "bm25_feature",
          "separator": " ",
          "default_value": "0",
          "query": "user:query",
          "document": "item:document",
          "sequence_fields": "query",
          "document_number": 100,
          "avg_doc_length": 6,
          "term_doc_freq_dict": {
            "this": 30,
            "example": 10,
            "document": 15
          }
        },
        {
          "feature_name": "overlap_feat",
          "feature_type": "overlap_feature",
          "query": "user:query2",
          "title": "item:title2",
          "sequence_fields": "query2",
          "method": "index_of",
          "separator": " ",
          "default_value": "-1"
        },
        {
          "feature_type": "kv_dot_product",
          "feature_name": "query_doc_sim",
          "query": "user:query3",
          "document": "item:title",
          "sequence_fields": "query3",
          "separator": "|",
          "default_value": "0"
        },
        {
          "feature_name": "seg_feat",
          "feature_type": "tokenize_feature",
          "expression": "input_a",
          "default_value": "0",
          "output_type": "word",
          "tokenizer_type": "sentencepiece",
          "vocab_file": "spmodel.model"
        },
        {
          "feature_name": "txt_norm",
          "feature_type": "text_normalizer",
          "expression": "input",
          "default_value": "<oov>",
          "parameter": 28
        },
        {
          "feature_name": "seq_combo_feat",
          "feature_type": "combo_feature",
          "expression": ["user:tags", "item:cat"],
          "sequence_fields": ["tags"],
          "separator": "_",
          "default_value": "0",
          "value_dimension": 1
        },
        {
          "feature_name": "norm_str",
          "feature_type": "str_replace_feature",
          "expression": ["user:profile"],
          "default_value": "",
          "replace_file": "synonyms.txt",
          "replacements": {
            "|": "",
            "aa": "x",
            "a": "X"
          },
          "value_dimension": 1
        },
        {
          "feature_name": "query_tokens",
          "feature_type": "regex_replace_feature",
          "expression": ["user:query_tokens"],
          "default_value": "",
          "value_type": "string",
          "regex_pattern": [ "\\|", "#", "\\(.*\\)" ],
          "replacement": "",
          "value_dimension": 1
        },
        {
          "feature_name": "slice",
          "feature_type": "slice_feature",
          "value_type": "int32",
          "expression": ["context:array"],
          "slice": "0:3",
          "value_dimension": 3,
          "num_buckets": 100000
        },
        {
          "feature_name": "mask_feature",
          "feature_type": "bool_mask_feature",
          "value_type": "float",
          "expression": [
            "user:click_items",
            "item:is_valid"
          ]
        },
        {
          "feature_name": "time_diff_seq",
          "feature_type": "custom_feature",
          "operator_name": "SeqExpr",
          "operator_lib_file": "3rdparty/lib64/libseq_expr.so",
          "expression": ["user:cur_time", "user:clk_time_seq"],
          "formula": "cur_time - clk_time_seq",
          "sequence_fields": ["clk_time_seq"],
          "default_value": "0",
          "value_type": "double",
          "is_op_thread_safe": false,
          "value_dimension": 1
        }
      ]
    }
  ],
  "input_alias": {
    "common_seq__clk_time_seq": "clk_time_seq"
  }
}

Note: The input_alias parameter specifies an alias for an input field. The format is "origin_field": "alias_field", which allows you to use a shorter name to replace the original input field name.

Flattened format

Typically, you can obtain the sequence version by adding the sequence_ prefix to a non-sequence feature type (feature_type). Note that for sequence features, you must generally configure a default_value.

Examples:

• sequence_id_feature: The output value is always a string. If you need a different type, use slice_feature instead.

• sequence_raw_feature: The output value type is fixed to float. If you need other types, use slice_feature instead.

• sequence_match_feature

• sequence_lookup_feature

• sequence_expr_feature

• sequence_combo_feature

• sequence_overlap_feature

• sequence_bm25_feature

• sequence_kv_dot_product

• sequence_text_normalizer

• sequence_tokenize_feature

• sequence_combine_feature

Special case 1: Some feature transformation types have both sequence and non-sequence versions.

To activate the corresponding version, set is_sequence: true/false.

In this case, the feature_type parameter does not require the sequence_ prefix.

Examples:

• str_replace_feature

• regex_replace_feature

• custom_feature

Special case 2: Some feature transformation types only have a sequence version.

In this case, the feature_type parameter does not require the sequence_ prefix.

Examples:

• slice_feature

• bool_mask_feature

For these two special cases, you can add the following optional parameters:

• sequence_length: The maximum length of the sequence. Elements beyond this length are truncated. The default value is -1, which means the sequence is not truncated.

• sequence_delim: The separator between sequence elements. The default value is ;.

The following code provides a configuration example.

{
  "feature_name": "clk_seq__item_id",
  "feature_type": "sequence_id_feature",
  "sequence_name": "clk_seq",
  "sequence_length": 50,
  "sequence_delim": ";",
  "expression": "item:clk_item_seq",
  "separator": "\u001D",
  "default_value": ""
},
{
  "feature_name": "clk_seq__item_price",
  "feature_type": "sequence_raw_feature",
  "sequence_name": "clk_seq",
  "sequence_length": 50,
  "sequence_delim": ";",
  "expression": "item:clk_item_prices",
  "separator": "\u001D",
  "default_value": "0"
},
{
  "feature_name": "test",
  "feature_type": "sequence_lookup_feature",
  "map": "user:prefer_tags",
  "key": "item:tags",
  "sequence_length": 2,
  "separator": ",",
  "default_value": "-1024",
  "value_type": "int32",
  "normalizer": "method=expression,expr=x+1",
  "combiner": "sum",
  "default_bucketize_value": 50,
  "num_buckets": 10000
},
{
  "feature_name": "test",
  "feature_type": "sequence_combo_feature",
  "separator": "_",
  "default_value": "0",
  "expression": ["user:f1", "item:f2"],
  "hash_bucket_size": 10000
}

In the preceding example, the input fields clk_item_seq and clk_item_prices must be a Sequence, which can be an array or a string with elements separated by the character configured by sequence_delim.

• With this configuration, the online service (Processor) does not query side info. You must provide the complete input.

• The input field name for a sequence feature in flattened format is the same as the configured name and is not prefixed with ${sequence_name}__.

Online feature generation

You can obtain behavior sideinfo in two ways. One way is to retrieve the sideinfo from the item cache of EasyRec Processor. The field specified by sequence_pk is used as the primary key to look up item attribute information in the item cache. The other way is to populate the corresponding field values in the request. For example, the "ts" field in the preceding configuration represents (request_time - event_time), which is the recommendation request time minus the user behavior time. This value changes with each request and therefore must be obtained from the request.

user_features {
  key: "click_50_seq"
  value {
    string_feature: "9008721;34926279;22487529;73379;840804;911247;31999202;7421440;4911004;40866551"
  }
}

user_features {
  key: "click_50_seq__ts"
  value {
    string_feature: "23;113;401363;401369;401375;401405;486678;486803;486922;486969"
  }
}

Introduction

The combine_feature operator aggregates multiple values from an input feature into a single value using a specified combination strategy (combiner).

Its sequence version, sequence_combine_feature, aggregates values within each element of a sequence feature, transforming a multi-value sequence into a single-value sequence.

Key capabilities

• Multi-value combination: Aggregates multiple values from a feature into a single value.

• Flexible combination strategy: Supports various strategies, such as sum, mean, max, min, and count.

• Value map: Converts string identifiers to numeric values, ideal for processing a behavioral event sequence.

• Dual-separator support: Lets you configure both a sequence delimiter and a multi-value separator.

Configuration

Basic configuration (numeric combination)

{
  "feature_name": "combine_feat",
  "feature_type": "combine_feature",
  "expression": "user:behavior_seq",
  "combiner": "sum",
  "separator": "|"
}

You can configure the sequence version in two ways:

• Set the feature_type parameter to sequence_combine_feature.

• Add the parameter "is_sequence": true.

{
  "feature_name": "seq_combine_feat",
  "feature_type": "sequence_combine_feature",
  "expression": "user:behavior_seq",
  "combiner": "sum",
  "separator": "|",
  "sequence_delim": ";"
}

Or:

{
  "feature_name": "seq_combine_feat",
  "feature_type": "combine_feature",
  "expression": "user:behavior_seq",
  "combiner": "sum",
  "is_sequence": true,
  "separator": "|",
  "sequence_delim": ";"
}

Value map configuration (behavioral events)

{
  "feature_name": "behavior_score",
  "feature_type": "sequence_combine_feature",
  "expression": "user:action_events",
  "combiner": "sum",
  "separator": "|",
  "sequence_delim": ";",
  "value_map": {
    "expo": 1,
    "click": 2,
    "buy": 4
  }
}

The operator first applies the value map, then combines the resulting values.

Parameters

Examples

Example 1: Basic numeric combination (sum)

Configuration:

{
  "feature_name": "score_sum",
  "feature_type": "sequence_combine_feature",
  "expression": "user:scores",
  "combiner": "sum",
  "separator": ",",
  "sequence_delim": ";"
}

Input and Output:

Example 2: Behavioral event sequence (with a value map)

Configuration:

{
  "feature_name": "behavior_weight",
  "feature_type": "sequence_combine_feature",
  "expression": "user:actions",
  "combiner": "sum",
  "separator": "|",
  "sequence_delim": ";",
  "value_map": {
    "expo": 1,
    "click": 2,
    "buy": 4
  }
}

Input and Output:

Overview

The tokenize_feature operator tokenizes an input string, returning either the tokenized string or the corresponding token IDs. This operator supports vocabulary files in the tokenizer.json format from the tokenizers-cpp library.

For more information about the vocabulary file format, see the following resources:

1. https://github.com/huggingface/tokenizers

2. https://github.com/mlc-ai/tokenizers-cpp

Configuration

{
    "feature_name": "title_token",
    "feature_type": "tokenize_feature",
    "expression": "item:title",
    "default_value": "",
    "vocab_file": "tokenizer.json",
    "tokenizer_type": "sentencepiece",
    "output_type": "word_id",
    "output_delim": ","
}

Example

When output_type is word_id, the operator returns a string of token IDs, separated by the character specified in output_delim.

Vocabulary examples

Overview

The text_normalizer operator performs text normalization. Its features include case conversion, Traditional-to-Simplified Chinese conversion, full-width to half-width character conversion, special character filtering, GBK/UTF-8 encoding conversion, and Chinese character splitting.

Configuration

{
    "feature_name": "txt_norm",
    "feature_type": "text_normalizer",
    "expression": "item:title",
    "stop_char_file": "stop_char.txt",
    "max_length": 256,
    "parameter": 0,
    "remove_space": false,
    "is_gbk_input": false,
    "is_gbk_output": false
}

Note:

• The stop_char_file file must use GBK encoding.

• Each line in the stop_char_file file can contain only one character. Otherwise, filtering will fail.

Text normalization options

The parameter parameter specifies the sum of one or more of the following numbers.

For example, if the required functions are uppercase to lowercase conversion, full-width to half-width conversion, Traditional to Simplified Chinese conversion, and special character filtering, then parameter=4+8+16+32=60.

The default value of the parameter parameter is 60.

#define __NORMALIZED_LOWER2UPPER__ 		2 			/* Convert lowercase to uppercase. */
#define __NORMALIZED_UPPER2LOWER__ 		4 			/* Convert uppercase to lowercase. */
#define __NORMALIZED_SBC2DBC__ 			8 			/* Convert full-width to half-width characters. */
#define __NORMALIZED_BIG52GBK__			16 			/* Convert Traditional to Simplified Chinese. */
#define __NORMALIZED_FILTER__ 			32 			/* Filter special characters. */
#define __NORMALIZED_SPLITCHARS__		512 		/* Split Chinese characters into single characters, separated by spaces. */

Example

{
  "feature_name": "txt_norm",
  "feature_type": "text_normalizer",
  "expression": "input_a",
  "parameter": 28
}

• inputs=["Regular Expression Code Generator", "HTML Filtering Tool", "Regular Expression Syntax Cheatsheet", "The Cat/"]

• outputs=["regex code generator", "HTML filtering tool", "regular expression syntax quick reference", "the cat/"]

Features

The BM25 (Best Matching) algorithm is a leading text matching algorithm in information retrieval, used to calculate search relevance scores. The algorithm first parses a query into terms $$q_i$$. Then, for each search result D, it calculates the relevance score of each term $$q_i$$ for D. Finally, it calculates a weighted sum of the relevance scores of each term $$q_i$$ to produce the final relevance score between the query and D.

For Chinese, you can approach query tokenization as morpheme analysis, treating each term as a morpheme$$q_i$$.

The general formula for the BM25 algorithm is:

$$score(Q,d)=\sum _{i=1}^n{w}_{i}R(q_i,d)$$

Where $$Q$$ represents a query, qi is the $$i$$-th query term, $$d$$ is a document, $$w_i$$ is the weight of $$q_i$$, and R(q_i, d) is the relevance score of $$q_i$$ for document $$d$$.

Term importance

There are several methods for determining a term's relevance to a document. One of the most common is inverse document frequency (IDF). The formula is as follows:

$$IDF(q_i)=log\frac{N-n(q_i)+0.5}{n(q_i)+0.5}$$

Here, $$N$$ represents the total number of documents in the corpus, and $$n(q_i)$$ represents the total number of documents in the corpus that contain qi.

According to the definition of IDF, for a given document collection, the more documents that contain the term $$q_i$$, the lower the weight of $$q_i$$. In other words, when many documents contain $$q_i$$, the discriminating power of $$q_i$$ is low. Therefore, $$q_i$$ is less important for determining relevance.

Term relevance

In BM25, the relevance score between a term $$q_i$$ and a document $$d$$, denoted $$R(q_i,d)$$, is:

$$R(q_i,d)=\frac{f_i\cdot (k_1+1)}{f_i+K}\cdot \frac{q{f}_i\cdot (k_2+1)}{q{f}_i+k_2}$$

$$K=k_1\cdot \left(1-b+b\cdot \frac{dl}{avgdl}\right)$$

$$k_1,k_2,b$$ are adjustment factors, typically set based on experience. Common values are $$k_1=1.2,b=0.75,\; and{k}_2=0$$. $$f_i$$ is the frequency of term $$q_i$$ in document $$d$$, and $$q{f}_i$$ is the frequency of term $$q_i$$ in the query. $$dl$$ is the length of document $$d$$, and $$avgdl$$ is the average document length across all documents. Because a term $$q_i$$ typically appears only once in a query ($$q{f}_i=1$$), the formula simplifies to:

$$R(q_i,d)=\frac{f_i\cdot (k_1+1)}{f_i+K}$$

From the definition of $$K$$, you can see that the parameter $$b$$ adjusts the impact of document length on relevance. The larger the $$b$$ value, the greater the effect of document length on the relevance score, and vice versa. The longer the relative length of a document, the larger the $$K$$ value, which lowers the relevance score. This is because a longer document is more likely to contain $$q_i$$. Therefore, for the same $$f_i$$ value, a long document's relevance to $$q_i$$ is weaker than a short document's relevance to $$q_i$$.

The BM25 algorithm's relevance score formula can be summarized as follows:

$$score(Q,d)=\sum _{j=1}^{n}IDF(q_i)\frac{f_i\cdot (k_1+1)}{f_i+k_1\cdot (1-b+b\cdot \frac{dl}{avgdl})}$$

The BM25 formula demonstrates that using different methods for tokenization, term weighting, and determining term-document relevance can produce various search relevance scoring methods, offering great flexibility for algorithm design.

Configuration

{
  "feature_type": "bm25_feature",
  "feature_name": "query_doc_relevance",
  "query": "user:query",
  "document": "item:title",
  "term_doc_freq_file": "term_doc_freq.txt",
  "document_number": 1000,
  "avg_doc_length": 100.0,
  "k1": 1.2,
  "b": 0.75,
  "separator": "\u001D",
  "default_value": ""
}

• Specify either term_doc_freq_file or term_doc_freq_dict. The former takes precedence and is used if both are specified.

• When you use this feature in the online service, place the term_doc_freq_file file and fg.json in the same directory.

Overview

Calculates the dot product of the vectors of two key-value indexes, or the size of the intersection of two sets.

Configuration

{
  "feature_type": "kv_dot_product",
  "feature_name": "query_doc_sim",
  "query": "user:query",
  "document": "item:title",
  "separator": "|",
  "default_value": "0"
}

• This feature supports complex types, such as array and map. For optimal performance, use complex types.

• When the input does not have a value part, the default value is 1.0. You can use this property to find the size of the intersection of two sets.

• If you do not configure default_value, the default value is set to 0.

Examples

Overview

The str_replace_feature operator replaces all matched substrings in an input string with specified replacements.

Overlapping matches are replaced greedily.

Configuration

{
  "feature_name": "norm_str",
  "feature_type": "str_replace_feature",
  "expression": ["user:query"],
  "default_value": "",
  "replacements": {
    "brown": "box",
    "dogs": "jugs",
    "fox": "with",
    "jumped": "five",
    "over": "dozen",
    "quick": "my",
    "the": "pack",
    "the lazy": "liquor",
    "|": "",
    "aa": "x",
    "a": "X"
  },
  "value_dimension": 1
}

• You can configure both replace_file and replacements. The replacement dictionaries from both are merged, and replacements has a higher priority.

• This operator supports feature binning. For configuration details, see Feature Binning (Discretization):

  • hash_bucket_size: Hashes the feature transformation result and applies a modulo operation.

  • vocab_list: Bins the input based on a vocabulary and maps each value to its index in the list.

  • vocab_dict: Bins the input by mapping each feature value to its corresponding value in the vocab_dict.

  • vocab_file: Reads the vocab_list or vocab_dict from a file.

• This operator supports multi-value inputs of type array.

Example

The following table shows the output for the preceding configuration example.

Overview

The regex_replace_feature operator replaces substrings that match a regular expression with a specified replacement string.

You can configure multiple patterns. The operator replaces any substring that matches one of the specified patterns.

Configuration

{
  "feature_name": "query",
  "feature_type": "regex_replace_feature",
  "expression": ["user:query"],
  "regex_pattern": "\\|",
  "replacement": " ",
  "default_value": ""
}

• This operator supports feature binning. For configuration details, see the feature binning (discretization) documentation. The following parameters are available:

  • hash_bucket_size: Hashes the feature's value and applies a modulo operation.

  • vocab_list: Bins the feature's value based on a vocabulary list and maps it to its index in the list.

  • vocab_dict: Maps each feature value to a corresponding value in the vocab_dict.

  • vocab_file: Reads the vocab_list or vocab_dict from a file.

• Supports multi-valued array inputs.

Example

Introduction

Filters elements from a sequence using a boolean mask, similar to tf.boolean_mask(tensor, mask).

It is a type of sequence feature.

Configuration

{
  "feature_name": "mask_feature",
  "feature_type": "bool_mask_feature",
  "value_type": "float",
  "expression": [
    "user:click_items",
    "item:is_valid"
  ],
  "sequence_delim": ","
}

• Supports feature binning. For configuration details, see Feature binning (discretization).

• Supports multi-value inputs as arrays or nested arrays.

Examples

Usage with expression features

{
  "features": [
    {
      "feature_name": "mask",
      "feature_type": "expr_feature",
      "expression": "price>100",
      "variables": ["item:price"],
      "value_dimension": 3
    },
    {
      "feature_name": "filter_list",
      "feature_type": "bool_mask_feature",
      "expression": [
        "user:click_items",
        "feature:mask"
      ],
      "num_buckets": 10000
    }
  ]
}

Overview

Slices an input array using Python-style slice syntax or retrieves an element at a specific index.

This is a type of sequence feature.

Configuration

{
  "feature_name": "test_feature",
  "feature_type": "slice_feature",
  "value_type": "float",
  "expression": [
    "user:click_items"
  ],
  "slice": "2:4"
}

• This operator supports feature binning. For configuration details, see feature binning (discretization).

• This operator supports multi-value input, including arrays and nested arrays.

Example

When you set sequence_delim="," and value_dimension=1, the input and output are as follows: