The topic model is a type of statistical model that is used to discover abstract topics from a collection of documents. In Machine Learning Platform for AI (PAI), you can set the Topics parameter for the PLDA component to abstract different topics for each document.

Latent Dirichlet allocation (LDA) is a topic model that provides topics of each document based on probability distribution. LDA is an unsupervised learning algorithm. You need only to specify the number of topics in a document set by using K. You do not need to manually annotate training sets. K is the Topics parameter of the PLDA component.

LDA is a technique that was developed by David M. Blei, Andrew Y. Ng, and Michael I. Jordan in 2003. It is used to recognize texts, classify texts, and calculate the similarity between texts in the text mining field.

Configure the component

You can use one of the following methods to configure the PLDA component.

Method 1: Configure the component on the pipeline page

You can configure the parameters of the PLDA component on the pipeline page of Machine Learning Designer of Machine Learning Platform for AI (PAI). Machine Learning Designer is formerly known as Machine Learning Studio. The following table describes the parameters.
Table 1. Parameters
TabParameterDescription
Fields SettingFeature ColumnsThe feature columns that are used for training.
Parameters SettingTopicsThe number of topics that are generated by LDA.
AlphaThe prior Dirichlet distribution parameter of P(z/d).
BetaThe prior Dirichlet distribution parameter of P(w/z).
Burn-in IterationsThe number of burn-in iterations. The value of this parameter must be smaller than the total number of iterations. Default value: 100.
Total IterationsOptional. The total number of iterations. The value must be a positive integer. Default value: 150.

Method 2: Use PAI commands

Configure the component parameters by using PAI commands. You can use the SQL Script component to call PAI commands. For more information, see SQL Script. 
pai -name PLDA
    -project algo_public
    -DinputTableName=lda_input
    –DtopicNum=10
    -topicWordTableName=lda_output;
ParameterRequiredDescriptionTypeDefault value
inputTableNameYesThe name of the input table. STRINGNo default value
inputTablePartitionsNoThe partitions selected from the input table for training. The following formats are supported:
  • Partition_name=value
  • name1=value1/name2=value2: multi-level partitions
Note If you specify multiple partitions, separate them with commas (,).
STRINGAll partitions
selectedColNamesNoThe names of the columns selected from the input table for LDA. STRINGAll columns
topicNumYesThe number of topics. Valid values: 2 to 500. Positive integerNo default value
kvDelimiterNoThe delimiter used to separate keys and values. Valid values:
  • Space
  • Comma (,)
  • Colon (:)
STRINGColon (:)
itemDelimiterNoThe delimiter used to separate keys. Valid values:
  • Space
  • Comma (,)
  • Colon (:)
STRINGSpace
alphaNoThe prior Dirichlet distribution parameter of P(z/d). Valid values: (0, ∞). FLOAT0.1
betaNoThe prior Dirichlet distribution parameter of P(w/z). Valid values: (0, ∞). FLOAT0.01
topicWordTableNameYesThe name of the topic-word frequency contribution table. STRINGNo default value
pwzTableNameNoThe name of the P(w/z) output table. STRINGThe P(w/z) table is not generated.
pzwTableNameNoThe name of the P(z/w) output table. STRINGThe P(z/w) table is not generated.
pdzTableNameNoThe name of the P(d/z) table. STRINGThe P(d/z) table is not generated.
pzdTableNameNoThe name of the P(z/d) output table. STRINGThe P(z/d) table is not generated.
pzTableNameNoThe name of the P(z) output table. STRINGThe P(z) table is not generated.
burnInIterationsNoThe number of burn-in iterations. The value of this parameter must be smaller than the value of the totalIterations parameter. Positive integer100
totalIterationsNoThe total number of iterations.
Note z indicates the topic, w the word, and d the document.
Positive integer150
enableSparseNoSpecifies whether the data in the input table is key-value pairs. The data can be key-value pairs or word segmentation results. Valid values:
  • true: key-value pairs
  • false: word segmentation results
BOOLtrue
coreNumNoThe parameter and the memSizePerCore parameter must be used in pair. By default, the system calculates the number of cores based on the amount of the input data. Default value: -1. Positive integer-1
memSizePerCoreNoThe memory size of each core. Unit: MB. Valid values: [1024,65536]. By default, the system automatically calculates the memory size of each core. Default value: -1. Positive integer-1

Input and output settings

  • Input

    The data must be in the format of a sparse matrix. You can use the Convert Row, Column, and Value to KV Pair component to convert the data.

    Input format shows the input format.
    Figure 1. Input format
    Input format
    • Column 1: the ID of a document
    • Column 2: key-value data of words and word frequencies
  • Output

    The following tables are generated in sequence: topic-word frequency contribution table, P(w/z) table, P(z/w) table, P(d/z) table, P(z/d) table, and P(z) table.

    Output format shows the output format of the topic-word frequency contribution table.
    Figure 2. Output format
    Output format