Word Splitting

pai -name split_word_model
    -project algo_public
    -DoutputModelName=aliws_model
    -DcolName=content
    -Dtokenizer=TAOBAO_CHN
    -DenableDfa=true
    -DenablePersonNameTagger=false
    -DenableOrgnizationTagger=false
    -DenablePosTagger=false
    -DenableTelephoneRetrievalUnit=true
    -DenableTimeRetrievalUnit=true
    -DenableDateRetrievalUnit=true
    -DenableNumberLetterRetrievalUnit=true
    -DenableChnNumMerge=false
    -DenableNumMerge=true
    -DenableChnTimeMerge=false
    -DenableChnDateMerge=false
    -DenableSemanticTagger=true

If you use a regular table, we recommend that you do not set coreNum or memSizePerCore. The Word Splitting component automatically configures the parameters by default.

def CalcCoreNumAndMem(row, col, kOneCoreDataSize=1024):
    """Calculates the number of cores and memory size for each core.
       Args:
           row: the number of rows in the input table.
           col: the number of columns in the input table.
           kOneCoreDataSize: the amount of data that can be computed by each core. Unit: MB. The value must be a positive integer. Default value: 1024.
       Return:
           coreNum, memSizePerCore
       Example:
           coreNum, memSizePerCore = CalcCoreNumAndMem(1000,99, 100, kOneCoreDataSize=2048)
    """
    kMBytes = 1024.0 * 1024.0
    #Number of cores involved in computing
    coreNum = max(1, int(row * col * 1000/ kMBytes / kOneCoreDataSize))
    #Memory size per core = Data amount
    memSizePerCore = max(1024,  int(kOneCoreDataSize*2))
    return coreNum,  memSizePerCore

create table pai_aliws_test
as select
    1 as id,
    'Today is a good day. The weather is fine and sunny.' as content
from  dual;

pai -name split_word
    -project algo_public
    -DinputTableName=pai_aliws_test
    -DselectedColNames=content
    -DoutputTableName=doc_test_split_word

• The words in the tokenization column of the input table are split and then returned. The rest columns are returned without changes.
• When a custom dictionary is used, the system splits words based on the custom dictionary and context.