All Products
Search

This Product

    Platform For AI:String similarity

    Document Center

    Platform For AI:String similarity

    Last Updated:Apr 01, 2026

    The String Similarity component computes a pairwise similarity or distance score between two string columns in a table. It supports five algorithms spanning edit-distance, sequence-matching, kernel, cosine, and locality-sensitive hashing approaches. Use it to identify, match, or cluster similar text records in information retrieval, natural language processing (NLP), and bioinformatics workflows.

    Algorithm description

    The component supports five algorithms. Select the algorithm that matches your data type and task, then choose whether you need a distance or a similarity output.

    Distance vs. similarity: For Levenshtein and LCS, similarity = 1 − distance. For SimHash_Hamming, similarity = 1 − distance / 64.0. A similarity of 1 means the strings are identical; 0 means completely different.

    Edit-based

    Levenshtein Distance counts the minimum number of single-character edits (insertions, deletions, substitutions) needed to transform one string into the other. Use it for short strings such as product codes, usernames, or spell-check candidates.

    • Distance output: levenshtein

    • Similarity output: levenshtein_sim

    Sequence-based

    LCS (Longest Common Substring) measures the length of the longest common contiguous substring shared by two strings. Use it when the order and contiguity of characters matter, such as comparing DNA sequences or file paths.

    • Distance output: lcs

    • Similarity output: lcs_sim

    Kernel-based

    SSK (String Subsequence Kernel) scores similarity based on shared subsequences, with a penalty for non-contiguous matches controlled by the lambda parameter. Use it for flexible pattern matching where gaps between matching characters are acceptable, such as comparing log messages or command strings.

    • Similarity output: ssk

    Token-based

    Cosine splits each string into n-grams of length k, represents them as frequency vectors, and computes the cosine of the angle between the vectors. Use it for longer texts where term frequency matters, such as document deduplication or search ranking.

    • Similarity output: cosine

    Hash-based

    SimHash_Hamming uses the SimHash algorithm to map each string to a 64-bit binary fingerprint. The Hamming Distance is then used to calculate the number of different characters at the same position in the binary fingerprints. Use it for near-duplicate detection at scale, such as deduplicating web pages or product descriptions.

    • Distance output: simhash_hamming

    • Similarity output: simhash_hamming_sim (formula: 1 − distance / 64.0)

    Component configuration

    Method 1: Use the GUI

    Add the String Similarity component to the Designer workflow, then configure the parameters in the right panel.

    Fields setting

    ParameterDescription
    Columns to append to output tableAdditional columns from the input table to carry through to the output.
    First column for similarity calculationThe first string column to compare. Default: the first STRING-type column in the table.
    Second column for similarity calculationThe second string column to compare. Default: the second STRING-type column in the table.
    Similarity column in output tableThe name of the output column that holds the computed score.

    Parameters setting

    ParameterDescription
    Similarity calculation methodThe algorithm and output format. Valid values: levenshtein, levenshtein_sim, lcs, lcs_sim, ssk, cosine, simhash_hamming, simhash_hamming_sim. Default: levenshtein_sim.
    Substring lengthThe n-gram size k used to split strings before comparison. A value of 2 compares character bigrams; increase it to match longer patterns. Required when the method is ssk, cosine, simhash_hamming, or simhash_hamming_sim. Valid values: (0, 100). Default: 2.
    Weight of matching stringThe lambda decay factor that penalizes non-contiguous subsequence matches in SSK. Lower values penalize gaps more heavily; higher values are more lenient. Required when the method is ssk, simhash_hamming, or simhash_hamming_sim. Valid values: (0, 1). Default: 0.5.

    Execution tuning

    ParameterDescription
    Number of cores for computingDefault: system-assigned.
    Memory size per core (MB)Default: system-assigned.

    Method 2: Use PAI commands

    Run the component via PAI commands using the SQL Script component. For more information, see SQL Script.

    PAI -name string_similarity
    -project algo_public
    -DinputTableName="pai_test_string_similarity"
    -DoutputTableName="pai_test_string_similarity_output"
    -DinputSelectedColName1="col0"
    -DinputSelectedColName2="col1";
    ParameterRequiredDefaultDescription
    inputTableNameYes—Name of the input table.
    outputTableNameYes—Name of the output table.
    inputSelectedColName1NoFirst STRING-type columnName of the first column for similarity calculation.
    inputSelectedColName2NoSecond STRING-type columnName of the second column for similarity calculation.
    inputAppendColNamesNoNoneColumns to append to the output table.
    inputTablePartitionsNoAll partitionsPartitions of the input table to process.
    outputColNameNooutputName of the similarity score column in the output table. Only letters (a–z, A–Z), digits, and underscores (_) are allowed; must start with a letter; max 128 bytes.
    methodNolevenshtein_simAlgorithm and output format. Valid values: levenshtein, levenshtein_sim, lcs, lcs_sim, ssk, cosine, simhash_hamming, simhash_hamming_sim.
    kNo2N-gram size for splitting strings. Controls the granularity of token comparison: smaller values match shorter patterns, larger values match longer patterns. Required when method is ssk, cosine, simhash_hamming, or simhash_hamming_sim. Valid values: (0, 100).
    lambdaNo0.5Decay factor controlling how much non-contiguous gaps are penalized in SSK. Lower values penalize gaps more heavily; higher values are more lenient. Required when method is ssk. Valid values: (0, 1).
    lifecycleNoNoneLifecycle of the output table in days. Must be a positive integer.
    coreNumNoSystem-assignedNumber of cores for computing.
    memSizePerCoreNoSystem-assignedMemory size per core.

    What's next