Manage task instances with PyODPS - MaxCompute - Alibaba Cloud - MaxCompute

A task, such as an SQL task, is the basic unit of computation in MaxCompute. This topic describes the basic operations for task instances in PyODPS.

Basic operations

When a task runs, it is instantiated as a MaxCompute instance, which is also referred to as an instance. The following are common operations for instances:

list_instances(): Retrieves all instances in a project.
exist_instance(): Checks whether an instance exists.
get_instance(): Retrieves an instance.
stop_instance(): Stops an instance. You can stop only running instances. An error is reported if you try to stop an instance that is in any other state.
Note
Calling the stop method on an Instance object also stops the instance. For example: o.get_instance('instance id').stop.

Examples

Retrieve the ID of an instance

for instance in o.list_instances():
    print(instance.id)

Check whether an instance exists

print(o.exist_instance('my_instance_id'))

Get the Logview address

For tasks such as SQL tasks, you can call the get_logview_address method on an Instance object to retrieve the Logview address.

# Get the Logview address from an existing instance object.
instance = o.run_sql('desc pyodps_iris')
print(instance.get_logview_address())

# Get the Logview address from an instance ID.
instance = o.get_instance('my_instance_id')
print(instance.get_logview_address())

For PAI tasks, you can first enumerate its subtasks, and then retrieve the Logview address for each subtask.

instance = o.run_xflow('AppendID', 'algo_public', {'inputTableName': 'input_table', 'outputTableName': 'output_table'})
for sub_inst_name, sub_inst in o.get_xflow_sub_instances(instance).items():
    print('%s: %s' % (sub_inst_name, sub_inst.get_logview_address()))

Task instance status

The status of an instance can be Running, Suspended, or Terminated. The following methods are related to the instance status:

status: A property to retrieve the status.
is_terminated: Returns whether the instance has finished running.
is_successful: Returns whether the instance ran successfully. It returns False if the task is running or has failed.

The following code provides examples.

Retrieve the status of an instance.

instance=o.get_instance('my_instance_id')
print(instance.status)
print(instance.status.value)

Sample return values:

Status.TERMINATED
Terminated

Check whether the current instance has finished running.

instance=o.get_instance('my_instance_id')
from odps.models import Instance
print(instance.status == Instance.Status.TERMINATED)

A return value of True indicates that the instance has finished running.

Note

Calling the wait_for_completion method blocks the process until the instance finishes running. The wait_for_success method also blocks the process but throws an exception if the task fails.

Subtask operations

An instance can contain one or more subtasks at runtime. These subtasks are also called tasks. Note that this `Task` is different from the basic unit of computation in MaxCompute. The following methods are available for these tasks:

get_task_names: Retrieves the names of all subtasks. This method returns a list of the names.
```
instance=o.get_instance('my_instance_id')
instance.get_task_names()
```
Sample return value:
```
['jdbc_sql_task']
```

get_task_result: Retrieves the execution result of a specified task. This method returns the execution result of each task as a dictionary. The following code provides examples:

Retrieve the subtask name.

instance=o.execute_sql('select*frompyodps_irislimit1')
print(instance.get_task_names())

Sample return value:

['AnonymousSQLTask']

Retrieve the subtask execution result.

print(instance.get_task_result('AnonymousSQLTask'))

Sample return value:

"sepallength","sepalwidth","petallength","petalwidth","name"
4.9,3.0,1.4,0.2,"Iris-setosa"

Retrieve the subtask results.

print(instance.get_task_results())

Sample return value:

OrderedDict([('AnonymousSQLTask', '"sepallength","sepalwidth","petallength","petalwidth","name"\n4.9,3.0,1.4,0.2,"Iris-setosa"\n')])

get_task_progress: Retrieves the current progress of a task while the instance is running.

instance=o.get_instance('20160519101349613gzbzufck2')
while not instance.is_terminated():
    for task_name in instance.get_task_names():
        print(instance.id, instance.get_task_progress(task_name).get_stage_progress_formatted_string())
        time.sleep(10)

Sample return value:

20160519101349613gzbzufck2 2016-05-19 18:14:03 M1_Stg1_job0:0/1/1[100%]