MaxCompute:Use a PyODPS node to segment Chinese text based on Jieba

Preparation: Download the open source Jieba package

Download the open source Jieba package from its GitHub repository at https://github.com/fxsjy/jieba. Click the Code button in the upper-right corner and select Download ZIP to download the source code archive.

Practice 1: Use the open source Jieba package to segment Chinese text

1. Create a workflow. For more information, see Create a workflow.

2. Create a MaxCompute resource and upload the jieba-master.zip package.

   1. Right-click the workflow you created and choose Create Resource > MaxCompute > Archive.

   2. In the Create Resource dialog box, configure the parameters and click Create.

      In this example, set Engine Type to MaxCompute, Engine Instance to doc_test, and Resource Type to Archive. Select Upload as ODPS Resource.

      Parameter	Description
      Upload File	Click Click Upload and select the downloaded jieba-master.zip file.
      Name	The name of the resource. It must follow naming conventions but does not need to match the uploaded file name. In this example, the name is set to jieba-master.zip.

   3. Click the icon in the top toolbar to commit the resource.

3. Create a table named jieba_test and a table named jieba_result. The jieba_test table is used to store test data. The jieba_result table is used to store the test result.

   Right-click the workflow you created and choose Create Table > MaxCompute > Table. Create the tables and use the DDL mode to configure their fields. After the tables are created, commit them to the development environment. For more information about creating tables, see Create and use MaxCompute tables.

   The following table describes the DDL statements that are used to configure fields in the two tables.

   Table	DDL statement	Description
   jieba_test	CREATE TABLE jieba_test ( `chinese` string, `content` string );	Stores test data.
   jieba_result	CREATE TABLE jieba_result ( `chinese` string ) ;	Stores the test result.

4. Download test data and import the test data to the jieba_test table.

   1. Download the jieba_test.csv file that contains test data to your on-premises machine.

   2. On the DataStudio page, click the icon.

   3. In the Data Import Wizard dialog box, enter jieba_test as the destination table, select it, and click Next.

   4. Upload the jieba_test.csv file from your computer, configure the upload settings, preview the data, and click Next.

   5. Select By Name and click Import Data.

5. Create a PyODPS 3 node.

   1. Right-click the workflow you created and choose Create Node > MaxCompute > PyODPS 3.

   2. In the Create Node dialog box, enter a Name (for example, Name) and click OK.

6. Use the open source Jieba package to run segmentation code.

   Run the following sample code on the PyODPS 3 node to segment the test data in the jieba_test table and return the first 10 rows of segmentation result data:

   def test(input_var):
       import jieba
       result = jieba.cut(input_var, cut_all=False)
       return "/ ".join(result)
   # odps.stage.mapper.split.size can be used to increase parallelism.
   hints = {
       'odps.isolation.session.enable': True,
       'odps.stage.mapper.split.size': 64,
   }
   libraries =['jieba-master.zip']  # Reference your jieba-master.zip archive.
   src_df = o.get_table('jieba_test').to_df()  # Reference the data in your jieba_test table.
   result_df = src_df.chinese.map(test).persist('jieba_result', hints=hints, libraries=libraries)
   print(result_df.head(10))  # View the first 10 rows of the segmentation results. For more data, query the jieba_result table.

   Note

   odps.stage.mapper.split.size can be used to improve the execution parallelism. For more information, see Flag parameters.

7. View the result.

   You can view the Jieba segmentation results by using one of the following methods:

   • Method 1: View the results in the Operational Logs area at the bottom of the page.

   • Method 2: In the left-side navigation pane, click Ad Hoc Query to create an Ad Hoc Query node. Then, view the data in the jieba_result table.

     select * from jieba_result;

Practice 2: Use a custom dictionary to segment Chinese text

If the default Jieba dictionary does not meet your requirements, you can use a custom dictionary to segment Chinese text. The following example demonstrates this approach.

1. Create a MaxCompute resource.

   PyODPS user-defined functions (UDFs) can read resources uploaded to MaxCompute, including tables and files. The UDF must be written as a closure function or a callable class.

   Note

   You can create MaxCompute functions in DataWorks to reference complex UDFs. For more information, see Create and use a MaxCompute function.

   The following example uses a closure function to reference the custom dictionary file key_words.txt uploaded to MaxCompute.

   1. Create a MaxCompute function of the File type.

      Right-click the workflow you created and choose Create Resource > MaxCompute > File. Enter the resource name key_words.txt and click Create.

   2. On the configuration tab of the key_words.txt resource, enter the content of the custom dictionary and save and commit the resource.

      The following content is the example content of the custom dictionary. You can enter the content of the custom dictionary based on your test requirements.

2. Use the custom dictionary to run segmentation code.

   Run the following sample code on the PyODPS 3 node to segment the test data in the jieba_test table and return the first 10 rows of segmentation result data:

   def test(resources):
       import jieba
       fileobj = resources[0]
       jieba.load_userdict(fileobj)
       def h(input_var):  # In the nested function h(), load the dictionary and perform segmentation.
           result = jieba.cut(input_var, cut_all=False)
           return "/ ".join(result)
       return h
   # odps.stage.mapper.split.size can be used to increase parallelism.
   hints = {
       'odps.isolation.session.enable': True,
       'odps.stage.mapper.split.size': 64,
   }
   libraries =['jieba-master.zip']  # Reference your jieba-master.zip archive.
   src_df = o.get_table('jieba_test').to_df()  # Reference the data in your jieba_test table.
   file_object = o.get_resource('key_words.txt') # get_resource() references a MaxCompute resource.
   mapped_df = src_df.chinese.map(test, resources=[file_object])  # The map function calls the function and passes the resources parameter.
   result_df = mapped_df.persist('jieba_result2', hints=hints, libraries=libraries)
   print(result_df.head(10))  # View the first 10 rows of the segmentation results. For more data, query the jieba_result2 table.

   Note

   odps.stage.mapper.split.size can be used to improve the execution parallelism. For more information, see Flag parameters.

3. View the result.

   You can view the custom dictionary segmentation results by using one of the following methods:

   • Method 1: View the results in the Operational Logs area at the bottom of the page.

   • Method 2: In the left-side navigation pane, click Ad Hoc Query to create an Ad Hoc Query node. Then, view the data in the jieba_result2 table.

     select * from jieba_result2;