PyODPS supports file and table resources. This topic describes the common operations on resources in PyODPS.
list_resources(): lists all resources in a specific project.
exist_resource(): checks whether a specific resource exists.
delete_resource(): deletes a specific resource. You can also call the
drop()method to delete a resource.
create_resource(): creates a resource.
open_resource(): reads a resource.
Manage file resources
Files of basic file types and the .py, .jar, and .archive types are supported.
- Create a file resource
To create a file resource, you can call the
create_resource()method with a resource name, a file type, and a file-like object or string specified.
resource = o.create_resource('test_file_resource', 'file', file_obj=open('/to/path/file')) # Create a file resource with a file-like object. resource = o.create_resource('test_py_resource', 'py', file_obj='import this') # Create a file resource with a string.
- Read and modify a file resource
You can use one of the following methods to open a file resource:
An opened file resource is a file-like object. Similar to the
- Call the
open()method for the file resource.
- Call the
open_resource()method of a MaxCompute entry object.
open()method that is built in Python, the preceding two methods also allow you to specify the opening mode of a file resource.
with resource.open('r') as fp: # Open the file in read mode. content = fp.read() # Read all content of the file. fp.seek(0) # Move the file pointer to the beginning of the file. lines = fp.readlines() # Read multiple lines. fp.write('Hello World') # Error. Data cannot be written to a file in read mode. with o.open_resource('test_file_resource', mode='r+') as fp: # Open the file in read/write mode. fp.read() fp.tell() # Locate the current position of the file pointer. fp.seek(10) fp.truncate() # Truncate the file to the specified length. fp.writelines(['Hello\n', 'World\n']) # Write multiple lines to the file. fp.write('Hello World') fp.flush() # Manually call the method to submit the update to MaxCompute.
PyODPS supports the following file opening modes:
In PyODPS, you can also open a file resource in binary mode. For example, some compressed files must be opened in binary mode.
r: read mode. The file can be opened but data cannot be written to it.
w: write mode. Data can be written to the file, but data in the file cannot be read. If a file is opened in write mode, the file content is cleared first.
a: append mode. Data can be added to the end of the file.
r+: read/write mode. You can read data from and write data to the file.
w+: This mode is similar to
r+, but the file content is cleared first when the file is opened.
a+: This mode is similar to
r+, but data can be written only to the end of the file.
rb: binary read mode
r+b: binary read/write mode
- Call the
- Create a table resource
o.create_resource('test_table_resource', 'table', table_name='my_table', partition='pt=test')
- Update a table resource
table_resource = o.get_resource('test_table_resource') table_resource.update(partition='pt=test2', project_name='my_project2')
- Obtain a table and a partition of the table
table_resource = o.get_resource('test_table_resource') table = table_resource.table print(table.name) partition = table_resource.partition print(partition.spec)
- Read data from and write data to a table
table_resource = o.get_resource('test_table_resource') with table_resource.open_writer() as writer: writer.write([0, 'aaaa']) writer.write([1, 'bbbbb']) with table_resource.open_reader() as reader: for rec in reader: print(rec)