This topic describes the Word Splitting (Generate Models) component provided by Machine Learning Studio.
The Word Splitting (Generate Models) component is based on Alibaba Word Segmenter (AliWS). The component is used to generate a word segmentation model based on parameters and custom dictionaries.
The component supports only Chinese Taobao word segmentation and Internet word segmentation.
- The Word Splitting component splits texts into words.
- The Word Splitting (Generate Models) component generates a word segmentation model. To split texts, you must deploy a model and make a prediction or call an API operation.
Configure the component
- Machine Learning Platform for AI console
Tab Parameter Description Fields Setting Columns The column that is used to generate a model. Parameters Setting Recognition Options The types of content for recognition. Valid values: - Recognize Simple Entities
- Recognize Individual Names
- Recognize Organization Names
- Recognize Telephone Numbers
- Recognize Times
- Recognize Dates
- Recognize Alphanumeric Characters
By default, the following options are selected: Recognize Simple Entities, Recognize Telephone Numbers, Recognize Times, Recognize Dates, and Recognize Alphanumeric Characters.
Merge Options The types of content for merging. Valid values: - Merge Chinese Numbers
- Merge Arabic Numerals
- Merge Chinese Dates
- Merge Chinese Times
Default value: Merge Arabic Numbers.
Tokenizer The type of the filter. Valid values: TAOBAO_CHN and INTERNET_CHN. Default value: TAOBAO_CHN. Pos Tagger Specifies whether to perform part-of-speech tagging. By default, part-of-speech tagging is not performed. Semantic Tagger Specifies whether to perform semantic role labeling. By default, semantic role labeling is not performed. Filter Out Words That Contain Only Numbers Specifies whether to filter out words whose word segmentation results are numbers. By default, this option is cleared. Filter Out Words That Contain Only English Letters Specifies whether to filter out words whose word segmentation results are English letters. By default, this option is cleared. Filter Out Words That Contain Only English Punctuations Specifies whether to filter out words whose word segmentation results are punctuation marks. By default, this option is cleared. Tuning Cores The number of cores used for calculation. The value is automatically allocated. Memory Size per Core The size of memory required by each core. The value is automatically allocated. - PAI command
pai -name split_word_model -project algo_public -DoutputModelName=aliws_model -DcolName=content -Dtokenizer=TAOBAO_CHN -DenableDfa=true -DenablePersonNameTagger=false -DenableOrgnizationTagger=false -DenablePosTagger=false -DenableTelephoneRetrievalUnit=true -DenableTimeRetrievalUnit=true -DenableDateRetrievalUnit=true -DenableNumberLetterRetrievalUnit=true -DenableChnNumMerge=false -DenableNumMerge=true -DenableChnTimeMerge=false -DenableChnDateMerge=false -DenableSemanticTagger=true
Parameter Required Description Default value userDictTableName No Specifies whether to use a custom dictionary. A custom dictionary has only one column, and each row contains only one word. No default value outputModelName Yes The name of the output model. No default value colName No The column name of the prediction text. context dictTableName No Specifies whether to use a custom dictionary. A custom dictionary has only one column, and each row contains only one word. No default value tokenizer No The type of the filter. Valid values: TAOBAO_CHN and INTERNET_CHN. TAOBAO_CHN enableDfa No Specifies whether to recognize simple entities. Valid values: True and False. True enablePersonNameTagger No Specifies whether to recognize individual names. Valid values: True and False. False enableOrgnizationTagger No Specifies whether to recognize organization names. Valid values: True and False. False enablePosTagger No Specifies whether to perform part-of-speech tagging. Valid values: True and False. False enableTelephoneRetrievalUnit No Specifies whether to recognize telephone numbers. Valid values: True and False. True enableTimeRetrievalUnit No Specifies whether to recognize time expressions. Valid values: True and False. True enableDateRetrievalUnit No Specifies whether to recognize date expressions. Valid values: True or False. True enableNumberLetterRetrievalUnit No Specifies whether to recognize alphanumeric characters. Valid values: True and False. True enableChnNumMerge No Specifies whether to merge Chinese numbers into a retrieval unit. Valid values: True and False. False enableNumMerge No Specifies whether to merge Arabic numerals into a retrieval unit. Valid values: True and False. True enableChnTimeMerge No Specifies whether to merge Chinese time expressions into a semantic unit. Valid values: True and False. False enableChnDateMerge No Specifies whether to merge Chinese date expressions into a semantic unit. Valid values: True and False. False enableSemanticTagger No Specifies whether to perform semantic role labeling. Valid values: True and False. False
Examples
- PAI command
pai -name split_word_model -project algo_public -DoutputModelName=aliws_model
- Model deployment
create onlinemodel ning_test_aliws_model_2 -offlinemodelName ning_test_aliws_model -instanceNum 1 -cpu 100 -memory 4096;
- Online word segmentation
KVJsonRequest request = new KVJsonRequest(); Map<String, JsonFeatureValue> row = request.addRow(); row.put(col_name, new JsonFeatureValue("The big data algorithm platform is new")); KVJsonResponse res = predictClient.syncPredict(new JsonPredictRequest(project_name, model_name, request)); List<ResponseItem> ri = res.getOutputs(); for (ResponseItem item : ri) { System.out.println(item.getOutputLabel()); }
- Offline word segmentation
pai -name prediction -DmodelName=ning_test_aliws_model -DinputTableName=ning_test_aliws -DoutputTableName=ning_test_aliws_offline_predict;