Documentation
Home Page MaxCompute User Guide Data + AI and data science MaxFrame: The distributed AI compute engine MaxFrame flag configuration guide

MaxFrame flag configuration guide

Updated at:
Copy as MD
Product

MaxFrame exposes two layers of configuration: MaxCompute flags (SQL-level settings that control how individual jobs execute) and MaxFrame options (runtime settings that govern sessions, resources, and client behavior). This guide covers both layers — their purpose, default values, and when to change them.

Important

Many advanced flags require whitelisting before they take effect. Contact the MaxCompute technical support team before configuring expert-level options.

How to set flags

All MaxCompute SQL flags go into the options.sql.settings dictionary. MaxFrame options are set directly using options.<name>.

from maxframe import options

# MaxCompute SQL flags — set via options.sql.settings
options.sql.settings = {
    "odps.sql.job.max.time.hours": 72,
    "odps.session.image": "common",
    "odps.sql.split.dop": '{"*":50000}',
    "odps.sql.executionengine.batch.rowcount": 1024,
}

# MaxFrame options — set directly
options.session.logview_hours = 24
options.retry_times = 3
options.sql.enable_mcqa = True

Find flags by problem

Use this table to jump to the flags most relevant to a specific issue.

Problem Relevant flags
Job splits are slow odps.sql.split.dop, odps.sql.split.cluster.parallel_explore
Out-of-memory (OOM) errors odps.stage.mapper.mem, odps.stage.reducer.mem, odps.stage.joiner.mem, odps.sql.executionengine.batch.rowcount
Large-scale shuffles failing or slow fuxi_streamline_x_EnableNormalCheckpoint, fuxi_ShuffleService_client_CheckpointMaxCopy, odps.sql.sys.flag.fuxi_JobMaxInternalFolderSize
Data skew in GROUP BY or JOIN odps.stage.reducer.num, odps.stage.joiner.num
Long-running UDF (User-Defined Function) killed unexpectedly odps.sql.runtime.flag.fuxi_EnableInstanceMonitor, fuxi_InstanceMonitorTimeout, odps.sql.udf.timeout, odps.function.timeout
Column-store statistics (CMF) not generated after writing to dynamic partitioned tables Table writes and CMF fixed combination
Job fails due to transient machine errors odps.job.instance.retry.times
Python UDF log truncated odps.sql.runtime.flag.executionengine_PythonStdoutMaxsize
Need to use a specific runtime image or MaxCompute version odps.session.image, odps.task.major.version

MaxCompute flags

The following flags are set via options.sql.settings.

Concurrency and chunking

Parameter Description Default When to change
odps.sql.split.dop Sets the degree of parallelism (DOP) for data reads on a per-table basis, using column-store statistics (CMF) information. Takes priority over split.size. Format: {"table_name": value}. Use project.[schema.]table for a fully qualified name. Use {"*": value} to target all tables. Range: 1–99999. None When you process large tables or run large-scale tasks, explicitly enable this parameter to achieve high concurrency.
odps.stage.mapper.split.size When CMF is unavailable, splits tasks based on input table size in MB. Range: ≥1 MB. 256 MB Keep the default.

Resource and memory

Parameter Description Default When to change
odps.stage.mapper.mem / odps.stage.reducer.mem / odps.stage.joiner.mem Memory in MB allocated to a single worker for the Mapper, Reducer, and Joiner stages, respectively. Range: 1024–12288 MB. 1024 MB Increase when processing large data volumes, encountering data hot spots, or experiencing OOM errors from complex joins.
odps.stage.reducer.num / odps.stage.joiner.num Number of concurrent instances for the Reduce and Join stages. Maximum: 10,000. Dynamically calculated Increase for large-scale shuffles (GROUP BY or JOIN) or when data skew causes one instance to handle a disproportionate load.

Shuffle and output safety

Parameter Description Default When to change
odps.sql.runtime.flag.fuxi_streamline_x_EnableNormalCheckpoint & fuxi_ShuffleService_client_CheckpointMaxCopy Enables backups for intermediate Mapper output and sets the number of replicas. — For long-running jobs with large-scale shuffles, set replicas to 2: "fuxi_ShuffleService_client_CheckpointMaxCopy": 2. This significantly improves fault tolerance and data read stability during shuffle.
odps.sql.sys.flag.fuxi_JobMaxInternalFolderSize Maximum total size in MB for intermediate shuffle data generated by a single job. — Increase when a job fails with Internal data size exceeds limit during a large-table shuffle.

Compute stability and monitoring

Parameter Description Default When to change
odps.sql.runtime.flag.fuxi_EnableInstanceMonitor & fuxi_InstanceMonitorTimeout Enables heartbeat monitoring for the Fuxi scheduler and sets the timeout in seconds. Use together. Prevents the system from treating a long-running UDF as unresponsive and terminating it. — Enable when long-running UDFs are incorrectly killed. To change fuxi_InstanceMonitorTimeout, contact technical support to add it to the whitelist.
odps.job.instance.retry.times Maximum number of automatic retries for a single worker after transient failures such as machine breakdowns. Default: 3. Recommended maximum: 100. 3 Increase for jobs that fail due to transient infrastructure errors. To set above the default, contact technical support to add the parameter to the whitelist.
odps.dag2.compound.config Configures the worker reuse policy. Set to fuxi.worker.reuse.policy:NO_REUSE to disable reuse. — Disable reuse when a UDF has memory leak or state pollution risks, so each task starts in a clean environment. This slightly increases task startup overhead.

Execution efficiency and optimization

Parameter Description Default When to change
odps.sql.executionengine.batch.rowcount Batch size in rows — the basic unit for internal data processing. 1024 Decrease if a single row is very large and causes OOM. Increase for simple compute workloads to improve throughput.
odps.sql.runtime.flag.executionengine_EnableVectorizedExpr Enables the vectorized execution engine for expressions. — Enable when using rand() or running arithmetic-heavy operations.
odps.optimizer.enable.conditional.mapjoin & odps.optimizer.cbo.rule.filter.black Used together to disable HashJoin. Set cbo.rule.filter.black to "hj". — Expert option. Do not configure unless you fully understand the impact on the execution plan.
odps.sql.split.cluster.parallel_explore Reads CMF information concurrently during the task split stage. — Enable when the split stage of a job takes too long.
odps.sql.jobmaster.memory Memory size in MB for the job's Master node. — Increase for shuffle jobs involving very large tables. For example, set to 30,000 MB.

UDF and function safety

Parameter Description Default When to change
odps.sql.udf.timeout & odps.function.timeout Timeout in seconds for a data batch to execute in a UDF or function. Setting to 0 has no effect. Range: 1–3600 s. 1800 s —
odps.sql.runtime.flag.executionengine_PythonStdoutMaxsize Maximum length in MB for logs output to stdout by Python UDF print statements. Maximum: 100 MB. 20 MB Increase when Python UDF logs are truncated. Contact technical support to add to the whitelist.

Resource and environment

Parameter Description Default When to change
odps.session.image Specifies the runtime environment. The value must be the name of an existing custom image in the current tenant's MaxCompute project. — —
odps.task.major.version Locks a job to a specific major version of MaxCompute for feature and behavior stability. — Expert option. Do not configure unless you understand the impact.
odps.storage.orc.row.group.stride & odps.storage.meta.file.version Control the row group size of ORC files and the version of CMF metadata files, respectively. — Expert options. Do not configure unless you understand the underlying mechanisms.

Other general flags

Parameter Description Default When to change
odps.sql.allow.fullscan Allows a full table scan on a partitioned table without a partition filter condition. — Enable with caution. A full scan on large partitioned tables can result in high costs and long runtimes.
odps.sql.cfile2.field.maxsize Maximum storage size in bytes for a single column. Default: 8,388,608 (8 MB). Maximum: 268,435,456 (256 MB). 8,388,608 (8 MB) Increase when processing columns with very large content, such as long text, HTML, or Base64-encoded data.
odps.sql.job.max.time.hours Maximum runtime in hours for the entire SQL job. Maximum: 72 hours. 24 h —
odps.sql.always.commit.result & odps.sql.runtime.flag.executionengine_EnableWorkerCommit Used together to enable partial commit. Even if a job fails due to some processing errors, successfully processed data is still committed. — Use in ETL pipelines where partial success is acceptable.

Table writes and CMF fixed combination

When writing to dynamic partitioned tables, apply all five flags together. This ensures that column-store statistics (CMF) is generated quickly and correctly — which is critical for downstream jobs to use odps.sql.split.dop for precise data splitting.

options.sql.settings = {
    "odps.task.merge.enabled": "false",
    "odps.sql.reshuffle.dynamicpt": "false",
    "odps.sql.enable.dynaparts.stats.collection": "true",
    "odps.optimizer.dynamic.partition.is.first.nth.value.split.enable": "false",
    "odps.sql.stats.collection.aggressive": "true",
}

MaxFrame options

The following options are set directly using options.<name>.

Option Description Type Default
options.session.quota_name Quota to use for job execution. str / None None
options.local_timezone Local time zone. Affects the default behavior of date and time functions. str / None None
options.session.logview_hours Retention period in hours for LogView links. int 24
options.sql.enable_mcqa Specifies whether to enable the built-in intelligent query optimization and acceleration feature of MaxCompute. bool True
options.sql.generate_comments Automatically adds comments to generated SQL statements for traceability. bool True
options.sql.auto_use_common_image Automatically configures a common public image when the system detects that code uses libraries with extra dependencies. bool True
options.session.max_alive_seconds Maximum time to live for a session. — —
options.session.max_idle_seconds Maximum idle time for a session. If exceeded, the session is revoked. Must be less than or equal to max_alive_seconds. — —
options.session.temp_table_lifecycle Default lifecycle in days for temporary tables created by MaxFrame. int 1
options.session.auto_purge_temp_tables Automatically cleans up all temporary tables created in the current session when the session ends. bool False
options.function.default_running_options Default resource configuration for functions registered with the @remote decorator. Keys: cpu, memory, gpu. dict —
options.dpe.settings = {"substep.public_network_whitelist": ["<ip_or_cidr>"]} Adds a public network access whitelist for UDFs in the DPE engine. — —
options.dpe.settings = {"substep.internal_network_whitelist": ["<ip_or_cidr>"]} Adds an internal network access whitelist for UDFs in the DPE engine. — —
Important

Before configuring advanced flags — particularly those that require whitelisting or depend on CMF statistics — contact the MaxCompute technical support team to confirm the prerequisites and validate your configuration.

Previous：MaxFrame-specific APIsNext：MaxFrame coding skill