Faker connector

Background information

The Faker connector is a built-in connector for Realtime Compute for Apache Flink. It generates test data based on the Java Faker expressions that you provide for each field in a table. Use this connector to generate test data for validating business logic during development or testing.

Prerequisites

None

Limits

• The Faker connector is supported only in Realtime Compute for Apache Flink VVR 4.0.12 or later.

• The connector supports only the following data types: CHAR(n), VARCHAR(n), STRING, TINYINT, SMALLINT, INT, BIGINT, FLOAT, DOUBLE, DECIMAL, BOOLEAN, TIMESTAMP, ARRAY, MAP, MULTISET, and ROW.

• When used as a dimension table in a JOIN operation, the Faker connector does not perform an actual dimension table lookup. Instead, it generates a result directly based on the lookup key passed from the source table.

Syntax

CREATE TABLE faker_source (
  `name` STRING,
  `age` INT
) WITH (
  'connector' = 'faker',
  'fields.name.expression' = '#{superhero.name}',
  'fields.age.expression' = '#{number.numberBetween ''0'',''1000''}'
);

WITH parameters

Examples

Dimension table example

CREATE TEMPORARY TABLE datagen_source (
  `character_id` INT,
  `location` STRING,
  `datagen_name` STRING,
  `user_fullname` ROW<first_name STRING, last_name STRING>,
  `user_data` ARRAY<STRING>,
  `user_score` Map<STRING, INT>,
  `user_books` MULTISET<STRING>,
  `proctime` AS PROCTIME()
) WITH (
  'connector' = 'faker',
  'fields.character_id.expression' = '#{number.numberBetween ''0'',''10000''}',
  'fields.location.expression' = '#{harry_potter.location}',
  'fields.datagen_name.expression' = '#{superhero.name}',
  'fields.user_fullname.first_name.expression' = '#{superhero.prefix}',
  'fields.user_fullname.last_name.expression' = '#{superhero.suffix}',
  'fields.user_data.expression' = '#{harry_potter.character}',
  'fields.user_data.length' = '2',
  'fields.user_score.key.expression' = '#{harry_potter.character}',
  'fields.user_score.value.expression' = '#{number.numberBetween ''10'',''100''}',
  'fields.user_score.length' = '2',
  'fields.user_books.expression' = '#{book.title}',
  'fields.user_books.length' = '2',
  'number-of-rows' = '5'
);

CREATE TEMPORARY TABLE faker_dim (
  `character_id` INT,
  `faker_name` STRING
) WITH (
  'connector' = 'faker',
  'fields.character_id.expression' = '#{number.numberBetween ''0'',''100''}',
  'fields.faker_name.expression' = '#{harry_potter.characters}'
);
    
SELECT
  l.character_id,
  l.location,
  l.datagen_name,
  l.user_fullname,
  l.user_data,
  l.user_score,
  l.user_books,
  c.faker_name
FROM datagen_source AS l
JOIN faker_dim FOR SYSTEM_TIME AS OF proctime AS c
ON l.character_id = c.character_id;

Field expressions

• Format

  Each field defined in the DDL statement requires a corresponding expression in the WITH clause. The expression must be in the format 'fields.<field>.expression' = '#{className.methodName ''parameter'', ...}'. The following table describes the components in this format.

  Parameter	Description
  field	The name of a field in the DDL statement.
  className	The name of the Faker class. Java Faker provides about 80 Faker classes for generating field expressions. Select a class based on your business requirements. Note Faker class names are case-insensitive.
  methodName	The name of the method. Note Method names are case-insensitive.
  parameter	The input parameters for the method. Note Enclose input parameters for the method in two consecutive single quotation marks ('') to correctly escape them within the expression string. Use a comma (,) to separate multiple parameters.

• Example

  The following steps demonstrate how to construct a field expression, using the age field expression 'fields.age.expression' = '#{number.numberBetween ''0'',''1000''}' from the Syntax section as an example:

  1. In the Java Faker API documentation, find the Number class.

  2. In the Number class, find the numberBetween method and view its description.

     The method returns a value within a specified numeric range.

  3. Based on the class name (Number), method name (numberBetween), and input parameters (0 and 1000), construct the SQL expression for the age field: 'fields.age.expression' = '#{number.numberBetween ''0'',''1000''}'.

     This expression generates a value between 0 and 1000 for the age field.