Correlation coefficient matrix

The Correlation Coefficient Matrix component computes pairwise correlation coefficients across all selected feature columns and returns them as a square matrix. Use it during feature engineering to detect linear dependencies and multicollinearity before building a model.

In most cases, the Pearson correlation coefficient is used to measure linear relationships.

Configure the component

In Machine Learning Designer, add the Correlation Coefficient Matrix component to your pipeline. You can configure it using the Designer UI or a PAI command.

Method 1: Configure on the pipeline page (recommended)

Set the following parameters on the component's settings panel.

Tab	Parameter	Description
Fields Setting	All Selected by Default	Feature columns included in the calculation. By default, all feature columns are selected.
Tuning	Cores	Number of cores. Must be set together with Memory Size.
Tuning	Memory Size	Memory allocated per core. Must be set together with Cores.

Method 2: Use PAI commands

Submit the component as a PAI command using the SQL Script component. For details, see Scenario 4: Execute PAI commands within the SQL script component.

PAI -name corrcoef
    -project algo_public
    -DinputTableName=maple_test_corrcoef_basic12x10_input
    -DoutputTableName=maple_test_corrcoef_basic12x10_output
    -DcoreNum=1
    -DmemSizePerCore=110;

Parameter	Required	Default	Description
`inputTableName`	Yes	—	Name of the input table.
`inputTablePartitions`	No	—	Partitions to read from the input table. Supported formats: `partition_name=value`; `name1=value1/name2=value2` for multi-level partitions. Separate multiple partitions with commas — for example, `name1=value1,value2`.
`outputTableName`	Yes	—	Name of the output table.
`selectedColNames`	No	All columns	Columns to include in the calculation.
`lifecycle`	No	—	Lifecycle of the output table.
`coreNum`	No	System default	Number of cores. Must be set together with `memSizePerCore`. Valid values: 1–9999.
`memSizePerCore`	No	System default	Memory per core, in MB. Must be set together with `coreNum`. Valid values: 1024–65536.

Example

Input data

Generate a test table with 10 columns and 12 rows:

col0:double	col1:bigint	col2:double	col3:bigint	col4:double	col5:bigint	col6:double	col7:bigint	col8:double	col9:double
19	95	33	52	115	43	32	98	76	40
114	26	101	69	56	59	116	23	109	105
103	89	7	9	65	118	73	50	55	81
79	20	63	71	5	24	77	31	21	75
87	16	66	47	25	14	42	99	108	57
11	104	38	37	106	51	3	91	80	97
84	30	70	46	8	6	94	22	45	48
35	17	107	64	10	112	53	34	90	96
13	61	39	1	29	117	112	2	82	28
62	4	102	88	100	36	67	54	12	85
49	27	44	93	68	110	60	72	86	58
92	119	0	113	41	15	74	83	18	111

Run the PAI command

PAI -name corrcoef
    -project algo_public
    -DinputTableName=maple_test_corrcoef_basic12x10_input
    -DoutputTableName=maple_test_corrcoef_basic12x10_output
    -DcoreNum=1
    -DmemSizePerCore=110;

Output

Each value in the output matrix is a Pearson correlation coefficient in the range [−1, 1]:

1 on the diagonal indicates that each column is perfectly correlated with itself.
Values close to 1 or −1 indicate a strong positive or negative linear relationship.
Values close to 0 indicate little to no linear relationship.

columnsnames	col0	col1	col2	col3	col4	col5	col6	col7	col8	col9
col0	1	-0.2115657251820724	0.0598306259706561	0.2599903570684693	-0.3483249188225586	-0.28716254396809926	0.47880162127435116	-0.13646519484213326	-0.19500158764680092	0.3897390240949085
col1	-0.2115657251820724	1	-0.8444477377898585	-0.17507636221594533	0.40943384150571377	0.09135976026101403	-0.3018506374626574	0.40733726912808044	-0.11827739124590071	0.12433851389455183
col2	0.0598306259706561	-0.8444477377898585	1	0.18518346647293102	-0.20934839228057014	-0.1896417512389659	0.1799377498863213	-0.3858885676469948	0.20254569203773892	0.13476160753756655
col3	0.2599903570684693	-0.17507636221594533	0.18518346647293102	1	0.03988018649854009	-0.43737887418329147	-0.053818296425267184	0.2900856441586986	-0.3607547910075688	0.4912019074930449
col4	-0.3483249188225586	0.40943384150571377	-0.20934839228057014	0.03988018649854009	1	0.1465605209246875	-0.5016030364347955	0.5496024325711117	0.013743256115394122	0.07497231559184887
col5	-0.28716254396809926	0.09135976026101403	-0.1896417512389659	-0.43737887418329147	0.1465605209246875	1	0.16729809310873522	-0.29890655828796964	0.3618518101014617	-0.1713960957286885
col6	0.47880162127435116	-0.3018506374626574	0.1799377498863213	-0.053818296425267184	-0.5016030364347955	0.16729809310873522	1	-0.8165019880156462	-0.11173420918721436	-0.10363860378347944
col7	-0.13646519484213326	0.40733726912808044	-0.3858885676469948	0.2900856441586986	0.5496024325711117	-0.29890655828796964	-0.8165019880156462	1	0.07435907471544469	0.11711976051999162
col8	-0.19500158764680092	-0.11827739124590071	0.20254569203773892	-0.3607547910075688	0.013743256115394122	0.3618518101014617	-0.11173420918721436	0.07435907471544469	1	-0.18463012549540175
col9	0.3897390240949085	0.12433851389455183	0.13476160753756655	0.4912019074930449	0.07497231559184887	-0.1713960957286885	-0.10363860378347944	0.11711976051999162	-0.18463012549540175	1

In this example, col1 and col2 have a strong negative correlation (−0.844), and col6 and col7 also have a strong negative correlation (−0.817), which suggests potential multicollinearity between those pairs.

col0:double	col1:bigint	col2:double	col3:bigint	col4:double	col5:bigint	col6:double	col7:bigint	col8:double	col9:double
19	95	33	52	115	43	32	98	76	40
114	26	101	69	56	59	116	23	109	105
103	89	7	9	65	118	73	50	55	81
79	20	63	71	5	24	77	31	21	75
87	16	66	47	25	14	42	99	108	57
11	104	38	37	106	51	3	91	80	97
84	30	70	46	8	6	94	22	45	48
35	17	107	64	10	112	53	34	90	96
13	61	39	1	29	117	112	2	82	28
62	4	102	88	100	36	67	54	12	85
49	27	44	93	68	110	60	72	86	58
92	119	0	113	41	15	74	83	18	111

col0:double	col1:bigint	col2:double	col3:bigint	col4:double	col5:bigint	col6:double	col7:bigint	col8:double	col9:double
19	95	33	52	115	43	32	98	76	40
114	26	101	69	56	59	116	23	109	105
103	89	7	9	65	118	73	50	55	81
79	20	63	71	5	24	77	31	21	75
87	16	66	47	25	14	42	99	108	57
11	104	38	37	106	51	3	91	80	97
84	30	70	46	8	6	94	22	45	48
35	17	107	64	10	112	53	34	90	96
13	61	39	1	29	117	112	2	82	28
62	4	102	88	100	36	67	54	12	85
49	27	44	93	68	110	60	72	86	58
92	119	0	113	41	15	74	83	18	111