Empirical Probability Density Chart - Model Training| Alibaba Cloud Documentation Center

This topic describes the Empirical Probability Density Chart component provided by Machine Learning Studio.

The Empirical Probability Density Chart component uses empirical distribution and kernel density estimation functions.

Empirical distribution function
If accurate parametric distribution cannot be obtained, the empirical distribution function is used to estimate probability distribution based on data and generate non-parametric distribution. For more information, see Empirical distribution function.
Kernel density estimation function
The kernel density estimation function is used to estimate the probability density of sample data. Similar to a histogram, kernel distribution indicates the distribution of sample data. The difference is that kernel distribution is a smooth and continuous curve, whereas a histogram shows discrete data distribution. If the kernel density estimation function is used, the probability density of non-sample data points is not 0. Instead, the probability density is an overlay of weighted probability densities of all the sampling points in specific kernel distribution. The Empirical Probability Density Chart component uses Gaussian distribution as the kernel density estimation function. For more information, see Kernel density estimation function.

Configure the component

You can configure the Empirical Probability Density Chart component by using one of the following methods:

Machine Learning Platform for AI (PAI) console


Tab	Parameter	Description
Fields Setting	Input Columns	The input columns. You can select only BIGINT- or DOUBLE-type columns.
Fields Setting	Label Column	The label column.
Parameters Setting	Number of Calculation Intervals	A large value indicates high accuracy. The value of this parameter is calculated based on the range of values in each column.
Tuning	Cores	The number of cores that you want to use for computing. The value of this parameter must be a positive integer.
Tuning	Memory Size	The memory size of each core. Valid values: 1 to 65536. Unit: MB.

PAI command

PAI -name empirical_pdf
-project algo_public
-DinputTableName="test_data"
-DoutputTableName="test_epdf_out"
-DfeatureColNames="col0,col1,col2"
-DinputTablePartitions="ds='20160101'"
-Dlifecycle=1
-DintervalNum=100


Parameter	Required	Description	Default value
inputTableName	Yes	The name of the input table.	No default value
outputTableName	Yes	The name of the output table.	No default value
featureColNames	Yes	The names of the feature columns that you want to select from the input table for training.	No default value
labelColName	No	The name of the label column in the input table.	No default value
inputTablePartitions	No	The partitions that you want to select from the input table for training. Specify this parameter in one of the following formats: Partition_name=value Multi-level partition: name1=value1/name2=value2 Note If you specify multiple partitions, separate them with commas (,).	No default value
intervalNum	No	The number of calculation intervals. A large number indicates a high accuracy. Valid values: [1,1E14).	No default value
lifecycle	No	The lifecycle of the output table.	No default value
coreNum	No	The number of cores that you want to use for computing. The value of this parameter must be a positive integer.	Automatically allocated
memSizePerCore	No	The memory size of each core. Valid values: 1 to 65536. Unit: MB.	Automatically allocated

Example

Execute the following SQL statement to generate input data:

    drop table if exists epdf_test;
    create table epdf_test as
    select
      *
    from
    (
      select 1.0 as col1 from dual
        union all
      select 2.0 as col1 from dual
        union all
      select 3.0 as col1 from dual
        union all
      select 4.0 as col1 from dual
        union all
      select 5.0 as col1 from dual
    ) tmp;

Run the following PAI command:

PAI -name empirical_pdf
-project algo_public
-DinputTableName=epdf_test
-DoutputTableName=epdf_test_out
-DfeatureColNames=col1;

Input
You can select multiple columns to calculate. You can also select label columns and group these columns by label value. For example, the label columns contain the values 0 and 1. The columns are divided into two groups: label=0 and label=1. Then, the probability density of each group is provided.

Note A maximum of 100 label columns can be specified.

Output

A diagram and a result table are generated. The following table lists the columns that are contained in the result table. If no label columns are specified, NULL is generated for the label column in the output table.


Column name	Data type
colName	STRING
label	STRING
x	DOUBLE
pdf	DOUBLE

Output table

    +------------+------------+------------+------------+
    | colname    | label      | x          | pdf        |
    +------------+------------+------------+------------+
    | col1       | NULL       | 1.0        | 0.12775155176809325 |
    | col1       | NULL       | 1.0404050505050506 | 0.1304256933829622 |
    | col1       | NULL       | 1.0808101010101012 | 0.13306325897429525 |
    | col1       | NULL       | 1.1212151515151518 | 0.1356613897616418 |
    | col1       | NULL       | 1.1616202020202024 | 0.1382173796574596 |
    | col1       | NULL       | 1.202025252525253 | 0.1407286844875733 |
    | col1       | NULL       | 1.2424303030303037 | 0.14319293014274642 |
    | col1       | NULL       | 1.2828353535353543 | 0.14560791960033242 |
    | col1       | NULL       | 1.3232404040404049 | 0.14797163876379316 |
    | col1       | NULL       | 1.3636454545454555 | 0.1502822610772349 |
    | col1       | NULL       | 1.404050505050506 | 0.1525381508819247 |
    | col1       | NULL       | 1.4444555555555567 | 0.1547378654919243 |
    | col1       | NULL       | 1.4848606060606073 | 0.1568801559764068 |
    | col1       | NULL       | 1.525265656565658 | 0.15896396664681753 |
    | col1       | NULL       | 1.5656707070707085 | 0.16098843325768245 |
    | col1       | NULL       | 1.6060757575757592 | 0.1629528799404685 |
    | col1       | NULL       | 1.6464808080808098 | 0.16485681490034038 |
    | col1       | NULL       | 1.6868858585858604 | 0.16669992491584543 |
    | col1       | NULL       | 1.727290909090911 | 0.16848206869138338 |
    | col1       | NULL       | 1.7676959595959616 | 0.17020326912168932 |
    | col1       | NULL       | 1.8081010101010122 | 0.17186370453638117 |
    | col1       | NULL       | 1.8485060606060628 | 0.17346369900080946 |
    | col1       | NULL       | 1.8889111111111134 | 0.17500371175692428 |
    | col1       | NULL       | 1.929316161616164 | 0.17648432589456017 |
    | col1       | NULL       | 1.9697212121212146 | 0.17790623634938396 |
    | col1       | NULL       | 2.0101262626262653 | 0.1792702373286898 |
    | col1       | NULL       | 2.050531313131316 | 0.18057720927022053 |
    | col1       | NULL       | 2.0909363636363665 | 0.18182810544221673 |
    | col1       | NULL       | 2.131341414141417 | 0.18302393829491406 |
    | col1       | NULL       | 2.1717464646464677 | 0.18416576567472337 |
    | col1       | NULL       | 2.2121515151515183 | 0.1852546770123305 |
    | col1       | NULL       | 2.252556565656569 | 0.18629177959496213 |
    | col1       | NULL       | 2.2929616161616195 | 0.18727818503109434 |
    | col1       | NULL       | 2.33336666666667 | 0.18821499601297229 |
    | col1       | NULL       | 2.3737717171717208 | 0.18910329347850022 |
    | col1       | NULL       | 2.4141767676767714 | 0.18994412426940221 |
    | col1       | NULL       | 2.454581818181822 | 0.19073848937711185 |
    | col1       | NULL       | 2.4949868686868726 | 0.19148733286168018 |
    | col1       | NULL       | 2.535391919191923 | 0.1921915315221827 |
    | col1       | NULL       | 2.575796969696974 | 0.19285188538972659 |
    | col1       | NULL       | 2.6162020202020244 | 0.19346910910630113 |
    | col1       | NULL       | 2.656607070707075 | 0.19404382424446043 |
    | col1       | NULL       | 2.6970121212121256 | 0.1945765526142701 |
    | col1       | NULL       | 2.7374171717171762 | 0.19506771059517916 |
    | col1       | NULL       | 2.777822222222227 | 0.19551760452158667 |
    | col1       | NULL       | 2.8182272727272775 | 0.19592642714194602 |
    | col1       | NULL       | 2.858632323232328 | 0.1962942551623821 |
    | col1       | NULL       | 2.8990373737373787 | 0.1966210478770638 |
    | col1       | NULL       | 2.9394424242424293 | 0.1969066468790639 |
    | col1       | NULL       | 2.97984747474748 | 0.19715077683721793 |
    | col1       | NULL       | 3.0202525252525305 | 0.19735304731663747 |
    | col1       | NULL       | 3.060657575757581 | 0.19751295561309964 |
    | col1       | NULL       | 3.1010626262626317 | 0.19762989056457925 |
    | col1       | NULL       | 3.1414676767676823 | 0.19770313729675995 |
    | col1       | NULL       | 3.181872727272733 | 0.19773188285349683 |
    | col1       | NULL       | 3.2222777777777836 | 0.19771522265793107 |
    | col1       | NULL       | 3.262682828282834 | 0.19765216774530828 |
    | col1       | NULL       | 3.303087878787885 | 0.19754165270453194 |
    | col1       | NULL       | 3.3434929292929354 | 0.19738254426210697 |
    | col1       | NULL       | 3.383897979797986 | 0.19717365043938664 |
    | col1       | NULL       | 3.4243030303030366 | 0.19691373021193162 |
    | col1       | NULL       | 3.4647080808080872 | 0.1966015035982942 |
    | col1       | NULL       | 3.505113131313138 | 0.19623566210464843 |
    | col1       | NULL       | 3.5455181818181885 | 0.19581487945135703 |
    | col1       | NULL       | 3.585923232323239 | 0.19533782250778076 |
    | col1       | NULL       | 3.6263282828282897 | 0.1948031623623475 |
    | col1       | NULL       | 3.6667333333333403 | 0.1942095854560816 |
    | col1       | NULL       | 3.707138383838391 | 0.19355580470939734 |
    | col1       | NULL       | 3.7475434343434415 | 0.19284057057394655 |
    | col1       | NULL       | 3.787948484848492 | 0.19206268194364004 |
    | col1       | NULL       | 3.8283535353535427 | 0.19122099686158253 |
    | col1       | NULL       | 3.8687585858585933 | 0.19031444296253852 |
    | col1       | NULL       | 3.909163636363644 | 0.1893420275936375 |
    | col1       | NULL       | 3.9495686868686946 | 0.18830284755928747 |
    | col1       | NULL       | 3.989973737373745 | 0.1871960984396676 |
    | col1       | NULL       | 4.030378787878796 | 0.18602108343567092 |
    | col1       | NULL       | 4.070783838383846 | 0.18477722169674377 |
    | col1       | NULL       | 4.111188888888897 | 0.1834640560916829 |
    | col1       | NULL       | 4.151593939393948 | 0.1820812603860928 |
    | col1       | NULL       | 4.191998989898998 | 0.18062864579383914 |
    | col1       | NULL       | 4.232404040404049 | 0.179106166873458 |
    | col1       | NULL       | 4.272809090909099 | 0.17751392674406796 |
    | col1       | NULL       | 4.31321414141415 | 0.17585218159888508 |
    | col1       | NULL       | 4.353619191919201 | 0.17412134449794325 |
    | col1       | NULL       | 4.394024242424251 | 0.1723219884250765 |
    | col1       | NULL       | 4.434429292929302 | 0.17045484859762067 |
    | col1       | NULL       | 4.4748343434343525 | 0.16852082402064342 |
    | col1       | NULL       | 4.515239393939403 | 0.1665209782808102 |
    | col1       | NULL       | 4.555644444444454 | 0.16445653957824907 |
    | col1       | NULL       | 4.596049494949504 | 0.16232889999798905 |
    | col1       | NULL       | 4.636454545454555 | 0.16013961402571825 |
    | col1       | NULL       | 4.6768595959596055 | 0.1578903963157465 |
    | col1       | NULL       | 4.717264646464656 | 0.15558311872216193 |
    | col1       | NULL       | 4.757669696969707 | 0.1532198066072439 |
    | col1       | NULL       | 4.798074747474757 | 0.1508026344442397 |
    | col1       | NULL       | 4.838479797979808 | 0.14833392073462115 |
    | col1       | NULL       | 4.878884848484859 | 0.14581612226291346 |
    | col1       | NULL       | 4.919289898989909 | 0.1432518277151203 |
    | col1       | NULL       | 4.95969494949496 | 0.1406437506896507 |
    | col1       | NULL       | 5.00010000000001 | 0.13799472213247665 |
    +------------+------------+------------+------------+