PyODPS supports file and table resources. This topic describes the common operations on resources in PyODPS.

Basic operations

  • list_resources(): lists all resources in a specific project.
  • exist_resource(): checks whether a specific resource exists.
  • delete_resource(): deletes a specific resource. You can also call the drop() method to delete a resource.
  • create_resource(): creates a resource.
  • open_resource(): reads a resource.

Manage file resources

Files of basic file types and the .py, .jar, and .archive types are supported.

  • Create a file resource

    To create a file resource, you can call the create_resource() method with a resource name, a file type, and a file-like object or string specified.

    resource = o.create_resource('test_file_resource', 'file', file_obj=open('/to/path/file'))  # Create a file resource with a file-like object.
    resource = o.create_resource('test_py_resource', 'py', file_obj='import this')  # Create a file resource with a string.
  • Read and modify a file resource
    You can use one of the following methods to open a file resource:
    • Call the open() method for the file resource.
    • Call the open_resource() method of a MaxCompute entry object.
    An opened file resource is a file-like object. Similar to the open() method that is built in Python, the preceding two methods also allow you to specify the opening mode of a file resource.
    with'r') as fp:  # Open the file in read mode.
         content =  # Read all content of the file.  # Move the file pointer to the beginning of the file.
         lines = fp.readlines()  # Read multiple lines.
         fp.write('Hello World')  # Error. Data cannot be written to a file in read mode.
     with o.open_resource('test_file_resource', mode='r+') as fp:  # Open the file in read/write mode.
         fp.tell()  # Locate the current position of the file pointer.
         fp.truncate()  # Truncate the file to the specified length.
         fp.writelines(['Hello\n', 'World\n'])  # Write multiple lines to the file.
         fp.write('Hello World')
         fp.flush()  # Manually call the method to submit the update to MaxCompute.

    PyODPS supports the following file opening modes:

    • r: read mode. The file can be opened but data cannot be written to it.
    • w: write mode. Data can be written to the file, but data in the file cannot be read. If a file is opened in write mode, the file content is cleared first.
    • a: append mode. Data can be added to the end of the file.
    • r+: read/write mode. You can read data from and write data to the file.
    • w+: This mode is similar to r+, but the file content is cleared first when the file is opened.
    • a+: This mode is similar to r+, but data can be written only to the end of the file.
    In PyODPS, you can also open a file resource in binary mode. For example, some compressed files must be opened in binary mode.
    • rb: binary read mode
    • r+b: binary read/write mode

Table resources

  • Create a table resource
    o.create_resource('test_table_resource', 'table', table_name='my_table', partition='pt=test')
  • Update a table resource
    table_resource = o.get_resource('test_table_resource')
    table_resource.update(partition='pt=test2', project_name='my_project2')
  • Obtain a table and a partition of the table
    table_resource = o.get_resource('test_table_resource')
    table = table_resource.table
    partition = table_resource.partition
  • Read data from and write data to a table
    table_resource = o.get_resource('test_table_resource')
    with table_resource.open_writer() as writer:
        writer.write([0, 'aaaa'])
        writer.write([1, 'bbbbb'])
        with table_resource.open_reader() as reader:
            for rec in reader: