Documentation
Home Page DataWorks FAQ Data development and modeling Data development and execution

Data development and execution

Updated at:
Copy as MD
Product

Frequently asked questions about resources, nodes, workflows, tables, and scheduling in DataWorks data development.

How do I call a third-party package in PyODPS?

Use an exclusive resource group for scheduling. For details, see Call third-party packages in a PyODPS node.

How do I control whether query results can be downloaded after querying table data?

To download data in DataWorks, you must first enable the download feature. If no download option appears, your workspace has not enabled downloads. Contact your Alibaba Cloud account or workspace administrator to enable this in Workspace Management. In the DataWorks console, click Workspace Configuration in the navigation pane on the left. In the Security Settings section, toggle the Download SELECT results switch to allow or block downloads of query results. When the switch is blue (enabled), you can download SELECT query results.

After running a query, a download option appears in the bottom-right corner of the results. Click the Download button in the toolbar at the bottom of the results page to download the data.

Due to engine limitations, the DataWorks interface only allows downloading up to 10,000 rows.

How do I download more than 10,000 rows of table data?

Use MaxCompute Tunnel. For details, see Export large volumes of data using SQLTask and Tunnel.

Why does visual table creation in DataWorks fail for EMR with the error "call emr exception"?

  • Possible cause: The security group of the ECS cluster hosting EMR lacks required rules. When registering an EMR cluster, add the following security group policy to avoid this error.

    The security group of the ECS cluster hosting EMR lacks required rules. When registering an EMR cluster, add the following security group policy to avoid this error.

    • Authorization policy: Allow

    • Protocol type: Custom TCP

    • Port range: 8898/8898

    • Authorization object: 100.104.0.0/16

  • Resolution: Check the security group configuration of the ECS cluster hosting EMR and add the above policy.

How do I use resources within a node?

Right-click the target resource node and select Insert Resource Path.

##@resource_reference{"pyodps_packagetest.py"}
import sys
import os
sys.path.append(os.path.dirname(os.path.abspath('pyodps_packagetest.py'))) # Add the resource to the workspace
import pyodps_packagetest # Reference the resource
pyodps_packagetest.printname() # Call the method

How do I download resources uploaded to DataWorks?

Right-click the target resource node and select View Historical Versions. In the Version Information dialog box, the version history table includes columns for version, submitter, submission time, change type, status, remarks, and actions. In the Actions column, click Code to view the resource content for that version, or click Roll Back to revert the resource to that version. Use the Compare button at the bottom of the dialog to compare versions.

How do I upload resources larger than 30 MB?

Upload resources larger than 30 MB using Tunnel commands. After uploading, add them as MaxCompute resources in DataWorks for further use. For details, see How do I use resources uploaded via odpscmd in DataWorks?.

How do I use resources uploaded via odpscmd in DataWorks?

To use resources uploaded via odpscmd in DataWorks, add them under MaxCompute resources in DataStudio. In the navigation pane on the left of DataStudio, click MaxCompute Resources. The main panel displays all MaxCompute resources in the current workspace, including those uploaded via odpscmd. Select the target resource and click Add to Data Development in the details panel on the right to include it in your data development workflow.

How do I upload a locally developed JAR file in DataWorks and execute it?

Upload your JAR file as a resource in the data development interface. To use it in a node, right-click the resource node and select Insert Resource Path (this adds a comment line at the top of the node). Then, execute the JAR directly by its resource name. In DataStudio, expand the Resources node in the file tree on the left, right-click, and choose New > JAR to create a JAR resource. In the Upload Resource panel on the right, click the upload icon and select your local JAR file (for example, ip2region.jar). Check the Upload as ODPS Resource check box to sync the resource to MaxCompute (ODPS). After clicking Upload, submit and publish the resource. Otherwise, you will get a resource not found error at runtime.

Example: In a Shell node, use ##@resource_reference{"test.jar"} java -jar test.jar.

How do I use MaxCompute table resources in DataWorks?

DataWorks does not support uploading MaxCompute table resources through the graphical interface. See the MaxCompute UDF example: Reference table resources for how to reference table resources. To use MaxCompute table resources in DataWorks, follow these steps:

  1. In MaxCompute, add the table as a table resource using the following SQL statement. For more information, see Add Resource.

    add table <table_name> [partition (<spec>)] [as <alias>] [comment '<comment>'][-f];

  2. In DataStudio, create a Python resource named get_cache_table.py to traverse and access the table resource added in MaxCompute. For the Python code, see Develop code.

  1. On the DataWorks Data Development page, create a new function. In this example, the function is named table_udf.

    Fill in the following fields:

    • Class Name: get_cache_table.DistCacheTableExample

    • Resources: Select the Python file get_cache_table.py from the dropdown. Add the table resource in Script Mode mode.

  2. After registering the function, follow the usage example to prepare test data and call the registered function.

When submitting a task in DataWorks, I receive the error "No default or available resource group is configured." How do I configure a resource group?

To resolve this, go to the Scheduling Settings section on the right side of the node editing page. Find Resource Group for Scheduling and select your desired resource group from the dropdown. If no suitable resource group appears, bind a resource group to your workspace as follows.

  1. Log on to the DataWorks Management Console. After switching regions, click Resource Group in the navigation pane on the left to open the Resource Groups.

  2. On the Resource Groups List page, find your created resource group and click Associate Workspace in the Actions column.

  3. On the Associate Workspace page, find your DataWorks workspace and click Associate in the Actions column.

After completing these steps, return to the node editing page. In Scheduling Settings on the right, select your desired scheduling resource group from the dropdown.

Can a Python resource call another Python resource?

As long as two resources are in the same workspace, a Python resource can invoke another Python resource.

Does PyODPS support calling user-defined functions to use third-party packages?

Yes. Besides using the test function through the map method of a DataFrame, PyODPS supports directly invoking user-defined functions to import third-party packages. For more information, see Use third-party packages in PyODPS.

When calling a Pickle file in PyODPS 3, I get the error: _pickle.UnpicklingError: invalid load key, '\xef

If your code contains special characters, compress it into a ZIP file before uploading. Then, decompress and use it in your code.

How do I delete MaxCompute resources?

After creating a resource, you can delete it as follows. In basic mode, right-click the resource and delete it directly. In standard mode, delete the resource in the development environment first, then delete it in the production environment. The following example shows how to delete a resource in production.

Note

In standard mode, development and production environments are separate. Deleting a resource in DataStudio only removes it from the development environment. To delete it from production, you must publish the deletion operation.

  1. Delete the resource in the development environment. Under the relevant workflow, go to MaxCompute > Resource. Right-click the resource to delete and click Delete. In the confirmation dialog, click Confirm.

  2. Delete a resource from the production environment. The delete operation creates a pending deployment record in the deployment interface, and the resource is removed from the production environment only after the record is deployed. You can click DataStudio in the upper-right corner of the DataStudio page, filter the change type to Unpublish, locate the deployment package that unpublishes the resource as described in the preceding step, click Actions, and then click Deploy. On the pop-up page, click Deploy. After you complete the deployment, the resource is deleted from the production environment.

After enabling Kerberos in an EMR cluster, why do I get a "DlfMetaStoreClientFactory not found" error when running spark-submit in YARN-Cluster mode in an EMR Spark node?DlfMetaStoreClientFactory not found?

  • Error details: Class com.aliyun.datalake.metastore.hive2.DlfMetaStoreClientFactory not found?

  • Cause: After enabling Kerberos in an EMR cluster, the Driver's classpath in YARN-Cluster mode does not automatically include JAR files from specified directories, causing Spark tasks to fail. For details, see EMR on ECS FAQ.

  • Solution: Manually specify DLF-related packages as follows:

    • After enabling Kerberos in the EMR cluster, when submitting a task in YARN-Cluster mode using spark-submit, add the --jars parameter. Include all JAR files from the /opt/apps/METASTORE/metastore-current/hive2 directory along with your application's dependencies.

      For YARN-Cluster mode in EMR Spark, manually specify DLF-related packages as shown below.

      Important

      In YARN-Cluster mode, all dependencies in the --jars parameter must be separated by commas. Directory paths are not supported.

      spark-submit --deploy-mode cluster --class org.apache.spark.examples.SparkPi --master yarn  --jars /opt/apps/METASTORE/metastore-current/hive2/aliyun-java-sdk-dlf-shaded-0.2.9.jar,/opt/apps/METASTORE/metastore-current/hive2/metastore-client-common-0.2.22.jar,/opt/apps/METASTORE/metastore-current/hive2/metastore-client-hive2-0.2.22.jar,/opt/apps/METASTORE/metastore-current/hive2/metastore-client-hive-common-0.2.22.jar,/opt/apps/METASTORE/metastore-current/hive2/shims-package-0.2.22.jar /opt/apps/SPARK3/spark3-current/examples/jars/spark-examples_2.12-3.4.2.jar

    • When manually specifying DLF-related packages, configure AccessKey information with permissions to access DLF and OSS. Otherwise, you may encounter an STS authentication error, such as one of the following:

      • Process Output>>> java.io.IOException: Response{protocol=http/1.1, code=403, message=Forbidden, url=http://xxx.xxx.xxx.xxx/latest/meta-data/Ram/security-credentials/}

      • at com.aliyun.datalake.metastore.common.STSHelper.getEMRSTSToken(STSHelper.java:82)

      At the task level, add the following parameters in Advanced Configuration on the right side of the node (to apply globally, configure these as Spark global parameters in cluster service settings):
"spark.hadoop.dlf.catalog.akMode":"MANUAL",
"spark.hadoop.dlf.catalog.accessKeyId":"xxxxxxx",
"spark.hadoop.dlf.catalog.accessKeySecret":"xxxxxxxxx"

How do I restore a deleted node?

After deleting a node, restore it from the recycle bin. In DataStudio, click Recycle Bin in the navigation pane on the left. In the list of deleted nodes, right-click the target node and select Restore.

How do I view a node's version?

Open the node configuration interface to view its version.

Important

Versions are generated only after submission.

On the right side of the node configuration interface, click the Versions tab to see the version list, which includes file ID, version number, submitter, and submission time. In the Actions column, click Code to view the code for a specific version, or click Roll Back to revert to that version. Select any two versions and click the Compare button at the bottom to see their differences.

How do I clone a workflow?

Use the node group feature. For details, see Use node groups.

How do I export node code from a workspace?

Use the Migration Assistant feature. For details, see Migration Assistant.

How do I check the submission status of a workflow node?

Select DataStudio > Workflow and expand the corresponding Business Process list to view the status of all nodes. If an icon icon appears to the left of the node name, the node has been submitted. If the icon does not appear, the node has not been submitted.

When a workflow contains multiple nodes, can I batch configure their scheduling information?

DataWorks does not support configuring scheduling information at the workflow level. You must configure each node individually.

If a node is deleted, are its instances affected?

The scheduling system generates one or more instances daily based on the task's time attributes. If you delete a task after it has run for some time, existing instances remain. If an instance triggers after the task is deleted, it fails because the code no longer exists.

After modifying a node task and publishing it to the production environment, does it overwrite the previous erroneous node in production?

No. Previous node instances are not overwritten. Unrun instances use the latest code. If scheduling parameters change, you must regenerate instances to run them.

How to visualize a newly created table?

You can create tables in Data Development, Table Management, or within table containers in workflows. In DataStudio, click Table Management in the navigation pane on the left. Click the New Table icon in the top toolbar. In the details panel on the right, configure the table name, MaxCompute engine instance, and table source. The Basic Properties section includes Chinese name, level-one category, level-two category, and description. In Physical Model Design, set partition type, lifecycle, level, physical classification, and table type (internal table/foreign table). At the top of the panel, use buttons such as DDL Mode, Load from Development Environment, Submit to Development Environment, Load from Production Environment, and Submit to Production Environment. Right-click a table in the table directory to perform actions like Rename Table, Import Data, or Delete Table.

How do I add fields to a production table?

An Alibaba Cloud account can add fields to a production table in the Table Management page and submit the changes to production.

A RAM user must have the O&M or project administrator role to add fields to a production table in the Table Management page and submit the changes to production.

How do I delete a table?

To delete a development table, do it in the data development interface.

To delete a production table:

  • Delete it from My Data in Data Map.

  • Create an ODPS SQL node, enter and run a DROP statement. For details on creating ODPS SQL nodes, see Develop ODPS SQL tasks. For DROP syntax, see Table operations.

On the My Data page, select the target table and click Delete in the Actions column to delete a single table. To delete multiple tables, select them and click Batch Delete at the bottom of the page.

How do I upload local data to a MaxCompute table?

In DataStudio, use the table import feature to upload local data. Click the Import icon in the toolbar to open the Data Import Wizard dialog. Configure the following parameters: Set Select Data Import Method to Upload Local File, File Format to CSV, click Browse... to select your local CSV file (only .csv files are supported), Select Separator to Comma, Source Character Set to GBK, Import Starting Row to 1, and check First Row as Header.

Table creation fails in an EMR cluster, and the interface shows: call emr exception?

  • The security group of the ECS cluster hosting EMR lacks required rules. When registering an EMR cluster, add the following security group policy to avoid this error.

    • Authorization policy: Allow

    • Protocol type: Custom TCP

    • Port range: 8898/8898

    • Authorization object: 100.104.0.0/16

  • Solution:

    Check the security group configuration of the ECS cluster hosting EMR and add the above policy.

How do I access production environment data from the development environment?

In standard mode, query production data using project_name.table_name.

If you upgraded from basic mode to standard mode, request producer role permissions before querying production data with project_name.table_name. For details on requesting permissions, see Request table permissions.

How do I obtain historical execution logs in the data development interface?

In the data development interface, find the Runtime Logs module in the sidebar on the left to view historical execution logs.

How long are data development execution histories retained?

By default, run histories in the data development interface are retained for 3 days.

Note

For log and instance retention periods in the production O&M center, see How long are logs and instances retained?.

How do I batch modify properties of nodes, resources, functions, and other objects?

In the DataStudio interface, click Batch Operation in the toolbar on the left. You can batch modify nodes, resources, and functions. After making changes, submit them in bulk and publish them in the task release interface to apply the changes to production.

In the batch operations interface, select target nodes. The action bar at the bottom provides Submit, Change Owner, and Change Engine Instance buttons. Click More to access additional actions: Change Scheduling Resource Group, Change Rerun Properties, Change Scheduling Type, Change Scheduling Cycle, Change Timeout Period, and Delete. Click Operation History in the top-right corner to view past batch operations.

How do I batch modify the scheduling resource group used by nodes under a workflow on the data development page?

In DataStudio, go to the workflow's Resource Group Orchestration tab. Here, you can batch modify the scheduling resource group for all nodes in the workflow. After making changes, click Submit to submit them in bulk, then publish them in the task release interface to apply the changes to production. In the Resource Group Orchestration tab, use the navigation tree on the left to filter nodes by type (Data Integration, MaxCompute, Hologres, etc.). The top-right section shows configuration areas for Shared Resource Groups for Scheduling and Exclusive Resource Groups. Below, the node list lets you filter by node name, node type, engine type, engine instance, or owner. Select the nodes to modify and click Switch Resource Group at the bottom to apply the change.

When connecting Power BI to MaxCompute, I encounter an error. How do I resolve this?

MaxCompute does not support Power BI connections. We recommend using Hologres instead. For details, see Endpoints.

OpenAPI call fails with access is forbidden. Please first activate DataWorks Enterprise Edition or Flagship Edition

OpenAPI requires DataWorks Enterprise Edition. For details, see Overview of DataWorks OpenAPI.

How do I obtain Python SDK usage examples?

On the relevant API page, click Debug to view Python SDK examples.

My task runs without an instance ID. How do I disable ODPS acceleration mode?

To get an instance ID, disable acceleration mode.

Note

DataWorks only supports downloading 10,000 rows. For more, use Tunnel, which requires an instance ID.

Task submitted successfully
Task running...
Task running...
Task running...
Task running...
Congratulations! Your task was selected to run in MaxCompute Query Acceleration mode
SQL executed successfully in MaxCompute Query Acceleration mode
SQL: SELECT a.real_name...

When running the task, add set odps.mcqa.disable=true; in the ODPS SQL node editor (execute it together with your SELECT statements).

Error: [202:ERROR_GROUP_NOT_ENABLE]:group is not available

During task execution, you get the error: Job Submit Failed! submit job failed directly! Caused by: execute task failed, exception: [202:ERROR_GROUP_NOT_ENABLE]:group is not available.

Possible cause: The bound resource group is unavailable.

Solution: Log on to the DataWorks console. Click Resource Groups in the navigation pane on the left. Find your resource group and check if its Status is Running. If not, restart the resource group or use another available one.

Previous：Data development and modelingNext：Schedule settings