MaxCompute:MaxFrame partial job submission function

Obtaining partial results for failed jobs

When you enable the partial submission function, you can obtain the result data of successfully executed Fuxi Instances through the following methods even if the current job ultimately fails.

You need to set the following parameter at the job level to enable the partial submission function.

options.sql.settings = {
    "odps.sql.always.commit.result": "true"  # Enable partial submission function
}

• Write the final results to the destination table (Recommended)

  As shown in the following example job, the final result data is written to the destination table data_output. You can view the destination table to obtain the result data of partially successful executions.

  df.mf.flatmap(
      ... # Customer business logic
  ).to_odps_table("data_output").execute()

  You can also view the successful Fuxi Instances in the job Logview:

• Query through temporary tables

  If the final result data is not written to the destination table, you can query the Logview to obtain the temporary table name automatically generated by MaxFrame to access the successfully executed result data.

  df.mf.flatmap(
      ...
  ).execute()

  View the MaxFrame temporary table name in Logview:

Setting job timeout to obtain partial results

For jobs that might have medium to long tails or infinite loops, you can actively set a job timeout period. After the timeout, the job will be forcibly terminated, but the result data of successfully executed Fuxi Instances will still be retained. You can still obtain the partially successful result data from the destination table or temporary table.

Enable the partial submission function and set the job timeout period:

options.sql.settings = {
    "odps.sql.always.commit.result": "true"  # Enable partial submission function
    "odps.sql.job.max.time.hours": "1"  # Set timeout period to 1 hour
}

Pay-as-you-go

If a MaxFrame job runs in pay-as-you-go mode, you are billed based on the actual CU-hours used.

You can download the specific usage information of the job through Usage Details. For more information, see MaxCompute bill usage details analysis.

Subscription

If a MaxFrame job runs in subscription mode, the resources of the subscription resource group will be occupied according to the requested amount when the MaxFrame job is running. There are no additional fees.