Configure the N-gram Counting component - Platform For AI - Alibaba Cloud - Platform For AI

N-gram Counting generates and counts n-gram frequencies across an entire corpus. The component scans all input sentences, identifies every sequence of n consecutive words (an n-gram), and tallies how often each sequence appears. The result is a global frequency count over the full corpus — not per document — which you can use directly as input to language model training.

How it works

Read each sentence from the input table column you specify.
Slide a window of width 1 through order (maximum n-gram length) across each sentence, producing all n-grams up to that length. For example, with order=3, the component produces unigrams, bigrams, and trigrams.
Accumulate counts across all sentences. If a bag-of-words table is provided, words outside the vocabulary are replaced with <unk> before counting.
Write the accumulated n-gram counts to the output table. Each row contains one n-gram and its total frequency across the corpus.

Configure the component

Two methods are available: the visual pipeline editor in Machine Learning Designer, or PAI commands via the SQL Script component.

Method 1: Configure on the pipeline page

In Machine Learning Designer (formerly Machine Learning Studio) on Platform for AI (PAI), drag the N-gram Counting component onto the canvas and configure the parameters in the right panel.

Fields Setting

Parameter	Description
Column of Sentences in Input Table	The column that contains the sentences to process.
Column of Words in the Bag-of-Words	The column that contains the vocabulary words. Words absent from this vocabulary are replaced with `<unk>` in the output.
Words Column in Input Counting Result Table	The word column in an existing n-gram count table to merge into the output.
Count Column in Input Counting Result Table	The count column in an existing n-gram count table to merge into the output.
Sentence Weight Column	The column that contains per-sentence weights. Leave blank to apply a uniform weight of 1 to all sentences.

Parameters Setting

Parameter	Description	Default
Maximum N-gram Length	The maximum n-gram length to generate. For example, setting this to 3 produces unigrams, bigrams, and trigrams — all orders from 1 up to the value you set.	3

Tuning

Parameter	Description
Optional. The number of cores.	The number of cores to allocate. Leave blank to let the system decide.
Optional. Memory size per core.	Memory per core, in MB. Leave blank to let the system decide.

Method 2: Use PAI commands

Run the ngram_count algorithm from the algo_public project using a SQL Script component. For more information, see SQL Script.

PAI -name ngram_count
    -project algo_public
    -DinputTableName=pai_ngram_input
    -DoutputTableName=pai_ngram_output
    -DinputSelectedColNames=col0
    -DweightColName=weight
    -DcoreNum=2
    -DmemSizePerCore=1000;

Parameter	Required	Default	Description
`inputTableName`	Yes	—	The name of the input table.
`outputTableName`	Yes	—	The name of the output table. Each row in the output contains one n-gram and its total frequency across the corpus.
`inputSelectedColNames`	No	First STRING column	The columns to read from the input table.
`weightColName`	No	1	The weight column. A value of 1 applies uniform weighting across all sentences.
`inputTablePartitions`	No	All partitions	The partitions to read from the input table.
`countTableName`	No	—	An existing n-gram count table to merge into the output.
`countWordColName`	No	Second column	The word column in the count table specified by `countTableName`.
`countCountColName`	No	Third column	The count column in the count table specified by `countTableName`.
`countTablePartitions`	No	—	The partitions to read from the count table.
`vocabTableName`	No	—	The bag-of-words table. Words absent from this table are replaced with `<unk>`.
`vocabSelectedColName`	No	First STRING column	The vocabulary column in the bag-of-words table.
`vocabTablePartitions`	No	—	The partitions to read from the bag-of-words table.
`order`	No	3	The maximum n-gram length. For example, `order=3` produces unigrams, bigrams, and trigrams — all orders from 1 up to the value you set.
`lifecycle`	No	—	The lifecycle of the output table, in days.
`coreNum`	No	—	The number of cores to allocate.
`memSizePerCore`	No	—	Memory per core, in MB.