MaxCompute:PyODPS overview

PyODPS is used in the following scenarios:

• Quick start.

• Read data from partitioned tables using PyODPS.

• Pass parameters in PyODPS.

• Use third-party packages with PyODPS.

• View hash partitions using PyODPS.

• Perform conditional queries using PyODPS.

• Use Sequence and execution operations in PyODPS.

PyODPS runs in local environments, DataWorks, and PAI Notebooks.

• Local environment: You can install and use PyODPS in your local environment. For more information, see Use PyODPS in a local environment.

• DataWorks: PyODPS is pre-installed on PyODPS nodes in DataWorks. You can directly develop and periodically run PyODPS jobs on these nodes. For more information, see Use PyODPS in DataWorks.

• PAI Notebooks: You can install and run PyODPS in the PAI Python environment. PyODPS is pre-installed in the built-in PAI images, such as the custom Python component of PAI-Designer, and is ready to use. Using PyODPS in PAI Notebooks is similar to its standard usage. For more information, see Basic operations overview and DataFrame (not recommended).

PyODPS is an SDK that runs on various clients, including PCs, DataWorks PyODPS nodes in Data Studio, and PAI Notebook environments.PyODPS provides several convenient operations to pull data to a local machine, such as the tunnel download operation, execute operation, and to_pandas operation. As a result, many new users try to pull data locally, process it, and then upload it back to MaxCompute. However, this method is often highly inefficient. Pulling data locally prevents you from taking advantage of MaxCompute's large-scale parallel computing capabilities.

The following tokenization example compares the code for both methods.

• Example scenario

  You need to extract information by analyzing daily log strings. You have a table that contains a single column of the string type. You must use the jieba library to tokenize the Chinese text, find the keywords you want, and store the keywords in an information table.

• Inefficient processing code demo

  import jieba
  t = o.get_table('word_split')
  out = []
  with t.open_reader() as reader:
      for r in reader:
          words = list(jieba.cut(r[0]))
          #
          # Processing logic to generate processed_data
          #
          out.append(processed_data)
  out_t = o.get_table('words')
  with out_t.open_writer() as writer:
      writer.write(out)

  This approach follows a single-machine processing mindset: read data row by row, process it row by row, and then write it to the destination table row by row. The entire process consumes a large amount of time for data download and upload. The machine that runs the script also needs a large amount of memory to process all the data. For users of DataWorks nodes, this approach can easily cause an OOM error by exceeding the default allocated memory.

• Efficient processing code demo

  from odps.df import output
  out_table = o.get_table('words')
  df = o.get_table('word_split').to_df()
  
  # Assume the following fields and types need to be returned
  out_names = ["word", "count"]
  out_types = ["string", "int"]
  
  @output(out_names, out_types)
  def handle(row):
      import jieba
      words = list(jieba.cut(row[0]))
      #
      # Processing logic to generate processed_data
      #
      yield processed_data
  df.apply(handle, axis=1).persist(out_table.name)

  Use the apply method for distributed execution:

  • The complex logic is placed in the handle function. This function is automatically serialized to the server-side to be used as a UDF, where it is invoked and executed. Because the handle function also processes data row by row during server-side execution, the logic is identical. The difference is that when this program is submitted to MaxCompute for execution, multiple machines process the data simultaneously. This saves a large amount of time.

  • Calling the persist interface writes the generated data directly to another MaxCompute table. All data generation and consumption occurs within the MaxCompute cluster. This saves local network and memory resources.

  • This example also uses a third-party package. MaxCompute supports third-party packages, such as jieba in this example, in UDFs. Therefore, you do not need to worry about the cost of code changes. You can leverage the large-scale computing capabilities of MaxCompute with almost no changes to your main logic.