Machine learning models that rely on gradient descent—such as linear models, neural networks, and support vector machines—are sensitive to the scale of input features. When features have different units or magnitudes, training can converge slowly or produce biased results. The Standardization component rescales numeric feature columns to zero mean and unit variance, eliminating dimensional differences so that each feature contributes equally to the model.
Not all algorithms require standardization. Tree-based models (such as random forests and gradient boosting) are scale-invariant and do not benefit from this step. If your data contains significant outliers, standardization may distort the scaled values because the mean and standard deviation are sensitive to extreme values.
How it works
Standardization applies the following formula to each selected column independently:
(X − Mean) / Standard deviation
where:
Mean is the sample mean of that column.
Standard deviation is the sample standard deviation (Bessel's correction applied).
The component saves the computed mean and standard deviation for each column to a parameter table (outputParaTableName). Pass this parameter table to inputParaTableName when standardizing new data to ensure training and inference use the same scaling statistics.
Standardized columns are prefixed with stdized_. The component supports DOUBLE and BIGINT column types.
Configure the component
Method 1: Configure the component on the pipeline page
Add the Standardization component to your pipeline in Machine Learning Designer. Configure the parameters in the right panel.
Fields Setting tab
| Parameter | Description |
|---|---|
| All Selected by default | Selects all columns by default. Non-selected columns pass through unchanged and do not affect prediction results. |
| Reserve original columns | When enabled, standardized columns are added alongside the originals with the stdized_ prefix. When disabled, all columns are reserved without renaming. Supports DOUBLE and BIGINT types. |
| Input sparse matrix | Enable if the input data is in sparse key-value format. Exposes three additional parameters: KV Pair Delimiter, KV Delimiter, and KV Index. |
Tuning tab
| Parameter | Description |
|---|---|
| Cores | Number of CPU cores. Allocated automatically based on input data volume. |
| Memory size per core | Memory per core in MB. Allocated automatically based on input data volume. |
Method 2: Use PAI commands
Run the Standardization component via PAI commands in the SQL Script component. For more information, see Scenario 4: Execute PAI commands within the SQL script component.
All commands use PAI -name Standardize.
Dense data
PAI -name Standardize
-project algo_public
-DkeepOriginal="false"
-DoutputTableName="test_5"
-DinputTablePartitions="pt=20150501"
-DinputTableName="bank_data_partition"
-DselectedColNames="euribor3m,pdays"Sparse data
PAI -name Standardize
-project projectxlib4
-DkeepOriginal="true"
-DoutputTableName="kv_standard_output"
-DinputTableName=kv_standard_test
-DselectedColNames="f0,f1,f2"
-DenableSparse=true
-DoutputParaTableName=kv_standard_model
-DkvIndices=1,2,8,6
-DitemDelimiter=",";PAI command parameters
| Parameter | Required | Default | Description |
|---|---|---|---|
inputTableName | Yes | — | Name of the input table. |
selectedColNames | No | All columns | Comma-separated list of columns to standardize. Supports INT and DOUBLE types for dense data, and STRING for sparse data. |
inputTablePartitions | No | All partitions | Partitions to read from the input table. Supported formats: partition_name=value (single level) and name1=value1/name2=value2 (multi-level). Separate multiple partitions with commas. |
outputTableName | Yes | — | Name of the output table. |
outputParaTableName | Yes | — | Name of the output parameter table. Stores the mean and standard deviation for each standardized column. |
inputParaTableName | No | — | Name of an existing parameter table to apply. Use this to standardize new data with the statistics computed from a training set. |
keepOriginal | No | false | Whether to retain original columns alongside standardized columns. true: adds standardized columns with the stdized_ prefix. false: reserves all columns without renaming them. |
lifecycle | No | — | Lifecycle of the output table. |
coreNum | No | Auto-assigned | Number of CPU cores. |
memSizePerCore | No | Auto-assigned | Memory per core in MB. |
enableSparse | No | false | Set to true to process data in sparse key-value format. |
itemDelimiter | No | , | Delimiter between key-value pairs in sparse data. |
kvDelimiter | No | : | Delimiter between keys and values in sparse data. |
kvIndices | No | — | Feature indexes to standardize in the key-value formatted table. |
Examples
The following examples demonstrate the complete standardization workflow: fitting a scaler on training data, saving the scaling parameters, and applying those parameters to new data. Using the same statistics for both training and inference prevents data leakage and ensures consistent predictions.
Step 1: Generate input data
drop table if exists standardize_test_input;
create table standardize_test_input(
col_string string,
col_bigint bigint,
col_double double,
col_boolean boolean,
col_datetime datetime);
insert overwrite table standardize_test_input
select
*
from
(
select
'01' as col_string,
10 as col_bigint,
10.1 as col_double,
True as col_boolean,
cast('2016-07-01 10:00:00' as datetime) as col_datetime
union all
select
cast(null as string) as col_string,
11 as col_bigint,
10.2 as col_double,
False as col_boolean,
cast('2016-07-02 10:00:00' as datetime) as col_datetime
union all
select
'02' as col_string,
cast(null as bigint) as col_bigint,
10.3 as col_double,
True as col_boolean,
cast('2016-07-03 10:00:00' as datetime) as col_datetime
union all
select
'03' as col_string,
12 as col_bigint,
cast(null as double) as col_double,
False as col_boolean,
cast('2016-07-04 10:00:00' as datetime) as col_datetime
union all
select
'04' as col_string,
13 as col_bigint,
10.4 as col_double,
cast(null as boolean) as col_boolean,
cast('2016-07-05 10:00:00' as datetime) as col_datetime
union all
select
'05' as col_string,
14 as col_bigint,
10.5 as col_double,
True as col_boolean,
cast(null as datetime) as col_datetime
) tmp;Step 2: Fit the scaler and standardize training data
This command fits the scaler on the training data and writes the scaling parameters (mean and standard deviation per column) to standardize_test_input_model_output.
drop table if exists standardize_test_input_output;
drop table if exists standardize_test_input_model_output;
PAI -name Standardize
-project algo_public
-DoutputParaTableName="standardize_test_input_model_output"
-Dlifecycle="28"
-DoutputTableName="standardize_test_input_output"
-DinputTableName="standardize_test_input"
-DselectedColNames="col_double,col_bigint"
-DkeepOriginal="true";Step 3: Apply the saved parameters to new data
Pass the parameter table from Step 2 to inputParaTableName to apply the same scaling statistics to a new dataset.
drop table if exists standardize_test_input_output_using_model;
drop table if exists standardize_test_input_output_using_model_model_output;
PAI -name Standardize
-project algo_public
-DoutputParaTableName="standardize_test_input_output_using_model_model_output"
-DinputParaTableName="standardize_test_input_model_output"
-Dlifecycle="28"
-DoutputTableName="standardize_test_input_output_using_model"
-DinputTableName="standardize_test_input";Input
standardize_test_input
| col_string | col_bigint | col_double | col_boolean | col_datetime |
|---|---|---|---|---|
| 01 | 10 | 10.1 | true | 2016-07-01 10:00:00 |
| NULL | 11 | 10.2 | false | 2016-07-02 10:00:00 |
| 02 | NULL | 10.3 | true | 2016-07-03 10:00:00 |
| 03 | 12 | NULL | false | 2016-07-04 10:00:00 |
| 04 | 13 | 10.4 | NULL | 2016-07-05 10:00:00 |
| 05 | 14 | 10.5 | true | NULL |
Output
standardize_test_input_output (Step 2 — original columns retained alongside stdized_ columns)
| col_string | col_bigint | col_double | col_boolean | col_datetime | stdized_col_bigint | stdized_col_double |
|---|---|---|---|---|---|---|
| 01 | 10 | 10.1 | true | 2016-07-01 10:00:00 | -1.2649110640673518 | -1.2649110640683832 |
| NULL | 11 | 10.2 | false | 2016-07-02 10:00:00 | -0.6324555320336759 | -0.6324555320341972 |
| 02 | NULL | 10.3 | true | 2016-07-03 10:00:00 | NULL | 0.0 |
| 03 | 12 | NULL | false | 2016-07-04 10:00:00 | 0.0 | NULL |
| 04 | 13 | 10.4 | NULL | 2016-07-05 10:00:00 | 0.6324555320336759 | 0.6324555320341859 |
| 05 | 14 | 10.5 | true | NULL | 1.2649110640673518 | 1.2649110640683718 |
standardize_test_input_model_output (scaling parameters saved in Step 2)
| feature | json |
|---|---|
| col_bigint | {"name": "standardize", "type":"bigint", "paras":{"mean":12, "std": 1.58113883008419}} |
| col_double | {"name": "standardize", "type":"double", "paras":{"mean":10.3, "std": 0.1581138830082909}} |
standardize_test_input_output_using_model (Step 3 — same statistics applied to new data)
| col_string | col_bigint | col_double | col_boolean | col_datetime |
|---|---|---|---|---|
| 01 | -1.2649110640673515 | -1.264911064068383 | true | 2016-07-01 10:00:00 |
| NULL | -0.6324555320336758 | -0.6324555320341971 | false | 2016-07-02 10:00:00 |
| 02 | NULL | 0.0 | true | 2016-07-03 10:00:00 |
| 03 | 0.0 | NULL | false | 2016-07-04 10:00:00 |
| 04 | 0.6324555320336758 | 0.6324555320341858 | NULL | 2016-07-05 10:00:00 |
| 05 | 1.2649110640673515 | 1.2649110640683716 | true | NULL |
standardize_test_input_output_using_model_model_output (scaling parameters from Step 3)
| feature | json |
|---|---|
| col_bigint | {"name": "standardize", "type":"bigint", "paras":{"mean":12, "std": 1.58113883008419}} |
| col_double | {"name": "standardize", "type":"double", "paras":{"mean":10.3, "std": 0.1581138830082909}} |