Progressive computation combines stream processing and batch processing to process only incremental data between runs. Instead of reprocessing an entire time range on every execution, it captures new data, computes results for the new partition, and merges them with stored intermediate results from previous runs.
How it works
In traditional batch processing, a sliding window query -- such as calculating the weekly total sales of a product -- reprocesses all data within the window on every run. Starting from day n+1 (n>=7), the data spanning day n-5 to day n is redundantly calculated each time.
Progressive computation eliminates this redundancy. On the first run, it computes results for each day in the window and stores them as intermediate results. On subsequent runs, only the new day's data is computed and merged with the stored results. This reduces the computational load by 70% in each run after the first.
Tradeoffs
First run takes longer. The first run calculates intermediate results for each day in the range, which may take longer than standard batch processing. Run the initial job manually in advance to make sure the results are ready when baseline-period scheduling begins.
Additional storage for intermediate results. Intermediate results are stored until they are no longer used. The storage cost is offset by the elimination of redundant calculation, making progressive computation more cost-efficient overall.
When to use progressive computation
Use progressive computation when MaxCompute jobs meet these conditions:
The query operates on a sliding time window or accumulation window (for example, last 7 days, last 3 hours, month-to-date).
The same time range is reprocessed on every scheduled run, creating redundant computation.
Query latency or resource consumption needs to be reduced without changing the underlying logic.
Progressive computation is not necessary for one-time queries or jobs where the entire dataset changes between runs.
Configuration reference
The following table lists all progressive computation parameters.
Parameter | Description | Valid values | Default | Required |
| Enable or disable progressive computation. |
|
| Yes |
| Computation mode (window strategy). | See Computation modes. | None | Yes |
| Time partition column names for finer granularity. | None | No (only for | |
| Store intermediate tables as cluster tables to improve shuffle performance. |
|
| No |
| Maximum concurrent instances during the first run. | Integer >= 1 |
| No |
Computation modes
Set odps.progressive.range.query.input.partition.pattern to one of the following values.
Mode | Window type | Use case |
| Hour-level or minute-level sliding window | Query data from the last 5 minutes or last 3 hours |
| Multi-day sliding window with hourly partitions | Query data from the last 3 hours or days, where the most recent window stores data in different partitions by hour |
| Multi-day sliding window | Query data from the last 3 days |
| Multi-day accumulation window with daily partitions | Query data from the last 50 days or last 3 years, where the most recent window stores data in different partitions by day |
| Monthly accumulation window | Query accumulated data from the first day of the current month to the current day |
| Yearly accumulation window | Query accumulated data from the first day of the current year to the current day |
Time partition columns
When using PASS_BY_HOUR, specify finer-grained time columns with odps.progressive.range.query.time.partition.col.names using one of the following formats.
Option 1: Apply to all tables
set odps.progressive.range.query.time.partition.col.names=default:<col_name_day>:<col_name_hour>:<col_name_minute>|<col_name_day>:<col_name_hour>:<col_name_minute>|...;Option 2: Apply to specific tables
set odps.progressive.range.query.time.partition.col.names=<table_name>:<col_name_day>:<col_name_hour>:<col_name_minute>|<col_name_day>:<col_name_hour>:<col_name_minute>|...;Option 3: Mix default and table-specific rules
set odps.progressive.range.query.time.partition.col.names=default:<col_name_day>:<col_name_hour>:<col_name_minute>,<table_name>:<col_name_day>:<col_name_hour>:<col_name_minute>,...;Parameters:
Parameter | Description |
| Keyword. Applies the column specification to all tables that meet the specified conditions. |
| Name of a specific table. Separate multiple tables with commas. |
| Time column specification. Use the pipe character ( |
Example 1: Run progressive computation every hour:
set odps.progressive.range.query.input.partition.pattern=PASS_BY_HOUR:1;Example 2: Specify finer-grained time columns:
set odps.progressive.range.query.time.partition.col.names=default:ds:hh|dt:hour;Example 3: Apply time columns to two specific tables:
set odps.progressive.range.query.time.partition.col.names=table_1:day:hour,table_2:hour:minute;Tune progressive computation
Store intermediate tables as cluster tables
During progressive computation, data in intermediate tables is shuffled before calculation. Storing intermediate tables as cluster tables improves computation performance.
set range.query.force.cluster.table=true;Set this parameter to true for best performance. When set to false (default), the system automatically selects a storage method based on whether a shuffle operation is present.
Limit concurrent instances on the first run
The first run processes all data within the specified time range, which is resource-intensive. Limit the number of concurrent instances to prevent excessive resource consumption.
set odps.progressive.combine.exec.time.limit.num=<number>;<number> is the maximum number of concurrent instances. Default value: 15. Minimum value: 1. Set this value based on actual job performance -- an inappropriate value affects the running efficiency.
Example
This example uses a multi-day sliding window (PASS_BY_DAY) to query data from a 7-day window.
First run
The first run processes the window [20200801, 20200807]. All seven days are computed, and the results are saved to seven intermediate tables -- one per day.
set odps.progressive.enable=true;
set odps.progressive.range.query.input.partition.pattern=PASS_BY_DAY;
CREATE TABLE adl_aegis_webshell_test_neoke AS
SELECT
request_datetime,host,uri,src_ip,src_port,dst_ip,dst_port,method,post_data,user_agent,ret_code,cookie,
referer,x_forward_for,rsp_content_type,rqs_content_type,content_length,jump_location,set_cookie,ttl,
get_bigwebshell_uri(uri) AS nopar_uri,internet_ip
FROM secbase.adl_aegis_webshell_newadd_beaverlog
WHERE ds >= TO_CHAR(DATEADD(TO_DATE('20200807','yyyymmdd'),-6,'dd') ,'yyyymmdd')
AND ds <= '20200807';Second run
On the next day, the window shifts to [20200802, 20200808]. Only the data for 20200808 is computed separately and then combined with the results in the seven intermediate tables to produce the final result.
Best practices
Test in the development environment first
Before enabling progressive computation in production, test its performance in the development environment. Verify that the query produces correct results and that the performance improvement meets expectations.
Run the first execution in advance
The first run computes intermediate results for every partition in the time range, making it more resource-intensive than subsequent runs. Schedule the first run during a low-traffic period or trigger it manually before production scheduling begins.
Store intermediate tables as cluster tables
Set range.query.force.cluster.table=true to store intermediate tables as cluster tables. This improves shuffle performance and reduces computation time on subsequent runs.