Configure Dataphin development platform settings - Dataphin

The development platform lets you control lock and ANALYZE command concurrency during development. This topic describes how to configure edit locks, object submission, query acceleration, and storage update settings.

Limitations

The storage update settings feature is available for the E-MapReduce 3.x, E-MapReduce 5.x, CDH 5.x, CDH 6.x, FusionInsight 8.x, Cloudera Data Platform 7.x, AsiaInfo DP 5.3, ArgoDB, TDH 6.x, StarRocks, SelectDB, or Doris compute engines.
The SelectDB and Doris compute engines do not support table management settings or the default compute engine for standard modeling.

Permissions

To configure the development platform, you need the Manage Development Platform Settings permission or be a Super Administrator or System Administrator.

Access the development platform

In the top navigation bar of the Dataphin homepage, choose Management Center > System Settings.
In the left-side navigation pane, choose Platform Settings > Development Platform.

Directory permissions

Permission control: This setting is disabled by default. When you enable it, you can set the visibility of top-level directories for offline compute tasks to Public or Visible to Specified Members Only. This controls who can see the directory and its nested tasks. If disabled, this setting invalidates all custom directory visibility settings, and all project members can view the nested offline compute tasks.

When releasing an object across tenants, the Permission control setting of the target environment takes precedence if it differs from the source.

Edit lock

In the Edit Lock section, click the edit icon, enable exclusive edit lock, and configure the lock settings.

Parameter	Description
Exclusive edit lock	When disabled, users can override each other's locks. When enabled, other users cannot edit a locked object until the lock is released manually or expires. They can acquire the lock to begin editing only after it is released.
Lock duration	If a user is inactive for the specified lock duration, the system releases the exclusive lock, making it available to other users. The default duration is 30 minutes. The minimum is 5 minutes, and the maximum is 120 minutes.
Auto-release upon closing object	Automatically releases the lock when a user closes the object's editor tab.
Auto-release upon successful submission	Automatically releases the lock after a successful submission. The lock is not released if the submission fails.

Click OK to save your settings.
To revert to the initial system configuration, you can click Restore Defaults.

Storage update settings

When integration or real-time development tasks write directly to HDFS tables, Hive does not automatically update storage information, such as table and partition size. As a result, this information might not appear in the asset catalog. To solve this, Dataphin lets you automatically run an ANALYZE command to get the latest storage information after a table is updated. You can configure this feature in Management Center > System Settings > Development Platform.

In the Storage Update Settings section, click the edit icon, enable automatic storage update, and configure the number of concurrent connections.
- Automatic storage update: Disabled by default. When enabled, Dataphin automatically runs an ANALYZE command on the target Hive table after a task runs successfully to update storage information. If you have many integration and real-time development tasks and a high-performance Hive server, you can increase the number of concurrent connections to reduce the total runtime of the update commands. This ensures that the latest storage information is available in the asset catalog the next day. Excessively high concurrency can consume significant compute resources and affect other tasks. Configure the number of concurrent connections based on your business needs.
- Maximum connections: The maximum number of concurrent connections for running ANALYZE commands. The default value is 5. You can set an integer from 1 to 200.
  Important
  When automatic storage update is enabled, any ANALYZE command that runs for more than 24 hours is automatically terminated. This applies to both running and waiting commands and helps save compute resources.
Click OK to save the settings.
Note
- When you enable automatic storage update and click OK, the configured concurrency takes effect immediately. Excessively high concurrency can consume significant compute resources and affect other tasks. Configure the number of concurrent connections based on your business needs.
- When you disable automatic storage update, ANALYZE commands that are already running or waiting are not affected. However, for subsequently completed integration and real-time development tasks, the storage information for their target tables will no longer be updated automatically. You can manually run the ANALYZE command in Hive to update the information.

Node and task settings

In the Node and Task Settings section, click the Edit icon to configure default scheduling times for new tasks and object submission rules.

New

Parameter		Description
Default priority		The default priority for new integration tasks, compute tasks, and logical table tasks. Options are Lowest, Low, and Medium. The default is Medium.
Default scheduling time	Random within interval	The default time interval is 00:00–03:00, and the default random interval is 5 minutes. The end time must be later than the start time. Valid times are from 00:00 to 23:59 in hh:mm format. The random interval must be an integer from 1 to 30.
	Fixed time	The default time is 00:00. Valid times are from 00:00 to 23:59 in hh:mm format.
Default Python version		The default Python version for new Python compute tasks, new Python offline compute templates, and third-party Python package installations. Options are Python 2.7, Python 3.7, and Python 3.11. The default is Python 3.7.

Note

For Hadoop-based compute engines, the Default Scheduling Time is set to Random within Interval by default. You can change it to Fixed Time.
When you create a new offline task, such as an integration task, compute task, or logical table, the scheduling time is automatically set based on this configuration.
- If the default scheduling time is set to Random within Interval, a random time is assigned based on the specified rule.
- If the default scheduling time is set to Fixed Time, the specified time is used.

Run
Hide logview URL when SQL contains global variables for account and password: Global variables for accounts and passwords in SQL code are displayed in plain text in the MaxCompute logview, which can create security risks. This option is disabled by default.
If you enable this configuration item, the logview URL for SQL statements in MAXCOMPUTE_SQL and logical table tasks that reference a global variable for an account and password will be hidden in the run and data preview logs of the development environment and the O&M logs of the production environment. The logview URL is replaced with The logview URL is hidden because the current SQL statement uses the global variable "{dp_glb_xxx}", which is of the account and password type..
Note
This option is available only when the compute engine is MaxCompute.

Submit

Parameter	Description
Auto-parse dependencies for offline object submission	When enabled, the system automatically parses dependencies and updates the list of upstream dependencies every time you submit an offline development object, such as an SQL compute task or a logical table task. This helps prevent users from forgetting or missing upstream dependencies.
Validate field types for logical table submission	When enabled, the system validates whether the return type of a field's calculation logic is compatible with the defined field type during logical table submission. If the types do not match, the submission is blocked. This prevents implicit type conversions that can cause data errors.

Decommission and delete
Allow deleting published objects in development environment: When enabled, objects that have already been released to the production environment (such as compute tasks, integration tasks, logical tables, atomic metrics, business qualifiers, and derived metrics) can be deleted directly from the development environment.
Important
Objects cannot be recovered after deletion. If you delete an object in the development environment but do not release the corresponding deletion task to the production environment, you cannot apply changes to the production object because its development counterpart no longer exists.

Default dependency period and policy

You can modify the default dependency period and default dependency policy.

Default dependency period: You can select Current Cycle (Today), Previous Cycle (Previous Day), Last 24 Hours, or Previous N Days. For Previous N Days, N defaults to 2 and cannot be empty.
Default dependency policy: You can select First Instance, Latest Instance, All Instances, or Last Instance.

The initial default dependency period and policy are shown in the following table.

Current node scheduling cycle	Upstream node scheduling cycle	Upstream node self-dependency	Default dependency period	Default dependency policy
Daily/Weekly/Monthly	Daily	Yes/No	Current Cycle (Today)	Last Instance
Daily/Weekly/Monthly	Hourly/Minutely	No	Current Cycle (Today)	All Instances
Daily/Weekly/Monthly	Hourly/Minutely	Yes	Current Cycle (Today)	Last Instance
Monthly/Weekly/Daily/Hourly/Minutely	Monthly/Weekly	Yes	Current Cycle (Today)	Last Instance
Monthly/Weekly/Daily/Hourly/Minutely	Monthly/Weekly	No	Current Cycle (Today)	Last Instance
Hourly/Minutely	Daily	Yes/No	Current Cycle (Today)	Last Instance
Hourly/Minutely	Hourly/Minutely	Yes/No	Current Cycle (Today)	Last Instance

Tag value
Manage the values for task tags. You can click Add Tag to add a new tag, up to a maximum of 50 tags. The tag name can contain any characters and must be 64 characters or less.
You can click the Delete icon to remove an existing tag. If you delete a tag, it is removed from all tasks that use it. If you re-add a tag with the same name, it is restored on those tasks.

After you finish configuring the settings, click OK.
To revert to the initial system configuration, you can click Restore Defaults.

Table management settings

Note

The StarRocks, GaussDB (DWS), Doris, and SelectDB compute engines do not support table management settings.

In the Table Management Settings section, click the Edit icon to configure Auto-create item to be released for table deletion via SQL and Create item to be released for table deletion in Table Management.
- Auto-create item to be released for table deletion via SQL: Enabled by default. When enabled, the system automatically creates an item to be released for table deletion when you run a drop table statement in an ad-hoc query or SQL compute task in the development environment. If this is disabled, no item to be released is created when you run a drop table table_name statement in the development environment.
- Create item to be released for table deletion in Table Management: Enabled by default. When enabled, the system creates a corresponding item to be released when you delete a table in Table Management. If this is disabled, no item to be released is created.

Configure the default storage format and default external table storage format. The supported formats vary by compute engine, as shown in the following table.

Note

You cannot configure the default storage format when the compute engine is AnalyticDB for PostgreSQL.
You can configure the default external table storage format only when the compute engine is MaxCompute.
In the following table, - indicates that the format is not supported.

Engine default

hudi

delta (Delta Lake)

paimon

iceberg

kudu

parquet

avro

rcfile

orc

textfile

sequencefile

binaryfile

csv

text

json

MaxCompute

Supported

Lindorm (compute engine)

Supported

Databricks

Supported

Amazon EMR

Supported

Transwarp TDH 6.x
Transwarp TDH 9.3.x

Supported

CDH 5.x
CDH 6.x
E-MapReduce 3.x
E-MapReduce 5.x
Cloudera Data Platform 7.x
Huawei FusionInsight 8.x
AsiaInfo DP 5.3

Supported

Configure the default lifecycle for physical and logical tables in the MaxCompute compute engine. The default value is empty, which means no lifecycle is set. You can enter an integer from 1 to 36,500 or quickly select 7, 14, 30, or 360 days.
Note
This setting is available only for the MaxCompute compute engine.
After you finish configuring the settings, click OK.
To revert to the initial system configuration, you can click Restore Defaults.

Default engine for standard modeling

If your tenant's Dataphin instance uses a Hadoop compute engine, you can set a default compute engine for standard modeling. The options are Hive, Impala, and Spark. These compute engines have the following limitations:

Important

If a specific task type is not enabled for the project's compute source, the system automatically switches to the Hive compute engine. For more information, see Create a Hadoop compute source.

Hive: Cannot read source tables stored in Kudu format.
Impala: Can read from source tables stored in Kudu format but does not currently support storing logical tables in Kudu format. We do not recommend using Impala unless you have source tables in Kudu format.
Note
When the compute engine is Amazon EMR, you cannot select Impala.
Spark: Cannot read source tables stored in Kudu format.

Query acceleration

Enable or disable MCQA query acceleration. When enabled, all ad-hoc MaxCompute SQL queries and all SQL cell queries in the analytics platform can use MCQA query acceleration. When disabled, the current tenant cannot use MCQA query acceleration.

Important

Query acceleration is supported only for the MaxCompute compute engine.

Single-tenant multi-engine

In multi-engine mode, you can configure only edit locks, storage update settings, node and task settings, and table management settings. The following settings must be configured elsewhere:

Hide logview URL when SQL contains global variables for account and password: Configure this setting in Cluster Management > MaxCompute Cluster - Other Configurations. For more information, see Other Configurations.
Default storage format/Default external table storage format: Configure this setting in Cluster Management > Other Configurations. For more information, see Other Configurations.
Default compute engine for standard modeling: Configure this setting in Cluster Management > Other Configurations. For more information, see Other Configurations.
Query acceleration: Configure this setting in Cluster Management > MaxCompute Cluster - Other Configurations. For more information, see Other Configurations.