FAQ about DataStudio

Operations based on compute engine instances in DataWorks

Tables

How-to:

• How do I create a table in a visualized manner?

• How do I add fields to a table in the production environment?

• How do I upload data from my on-premises machine to a MaxCompute table?

• How do I query production environment data from the development environment on the DataStudio page?

• How do I control whether queried table data can be downloaded?

• How do I delete a table?

Troubleshooting:

• I get the error "call emr exception" when creating a table in a workspace with an E-MapReduce (EMR) cluster registered. What do I do?

Resources

How-to:

• How do I upload a resource larger than 30 MB?

• How do I use a resource uploaded to DataWorks by using odpscmd?

• How do I reference a resource in a node?

• How do I use a MaxCompute table resource in DataWorks?

• How do I download a resource uploaded to DataWorks?

• How do I delete a MaxCompute resource?

• How do I upload a JAR package as a JAR resource and reference it in a node?

• Which resource group can I use when referencing a third-party package in a PyODPS node?

• Can a Python resource call another Python resource?

Nodes

How-to:

• How do I check whether a node is committed?

• How do I view the versions of a node?

• How do I recover a deleted node?

• How do I export the code of a node?

• Can I configure properties for all nodes in a workflow at once?

• How do I clone a workflow?

• How do I perform batch operations on multiple nodes, resources, or functions?

• How do I change the resource group for scheduling for multiple nodes in a workflow at once?

Concepts:

• After I modify, commit, and deploy a node to production, does it overwrite the existing faulty node?

• What happens to a node's instances after the node is deleted?

API calls

How-to:

• How do I get SDK for Python call examples?

Troubleshooting:

• I get the error "access is forbidden. Please first activate DataWorks Enterprise Edition or Flagship Edition" when calling an API operation. What do I do?

Node runs and run logs

Concepts:

• What are the differences in scheduling parameter value assignment between the Run, Run with Parameters, and Perform Smoke Testing in Development Environment modes?

How-to:

• How do I query historical operational logs on the DataStudio page?

• How long are operational logs on the DataStudio page retained?

Operations on compute engines

E-MapReduce (EMR) compute engine

• Hive: FAQ about Hive

• Spark: FAQ about Spark

MaxCompute compute engine

Development:

• SQL: FAQ about SQL

• Built-in functions: FAQ about built-in functions

• User-defined functions (UDFs): FAQ about MaxCompute Java UDFs and FAQ about MaxCompute Python UDFs

• MapReduce: FAQ about MaxCompute MapReduce

• PyODPS: Can PyODPS call custom functions to use third-party packages?, I get the error "_pickle.UnpicklingError: invalid load key, '\xef'" when calling a pickle file in a PyODPS 3 node. What do I do?, and FAQ about PyODPS

Security management:

• FAQ about permission management

• FAQ about security configurations

O&M management:

• FAQ about Information Schema

• FAQ about other O&M issues

Data download:

• How do I download more than 10,000 data records?

• How do I disable MaxCompute Query Acceleration (MCQA) to get the instance ID for downloading more than 10,000 data records?

BI connections:

• What do I do if an error is reported when connecting Power BI to MaxCompute?

Scheduling parameters

Typical use cases

• How do I specify a table partition in a format that contains a space, such as pt=yyyy-mm-dd hh24:mi:ss?

• How do I configure the time properties of an ODPS Spark node?

• How do I reprocess the return values of scheduling parameters if a node cannot process them?

• How are instances generated on the day when daylight saving time begins and ends?

• I run an instance at 00:00 to analyze data from the 23:00 partition on the previous day, but it analyzes data from the 23:00 partition on the current day instead. What do I do?

• A node is scheduled to run at 20:00 daily using $cyctime, but its ancestor node fails and causes it to run at 00:00 the next day. What value does $cyctime take?

• What are the differences between the return values of a MaxCompute date function and a scheduling parameter?

Testing

• How do I test scheduling parameter configurations on the DataStudio page?

• I get the error "FAILED: ODPS-0130161:[1,84] Parse exception - invalid token '$'". What do I do?

• I get the error "params format error,please check your params(key=values)". What do I do?

Monitoring and validation

• How do I check that scheduling parameter values are valid in the production environment?

• How do I verify scheduling parameter values for an instance by viewing its logs?

• I configured a scheduling parameter and deployed the node, but the return value never changes. What do I do?

Scheduling dependencies

Before you configure scheduling dependencies

• What are scheduling dependencies?

• Why are scheduling dependencies required?

• How do I configure scheduling dependencies for a node?

• What rules apply when a node depends on its ancestor nodes to run?

• What is a node's output name used for?

• Can a node have multiple output names?

• Which scenarios do not support scheduling dependencies?

How-to

• How do I delete a table that a node does not depend on?

• How do I delete multiple tables that my node does not depend on?

• Can multiple nodes share the same output name?

• How do I prevent DataWorks from parsing temporary tables when analyzing scheduling dependencies?

• How do I configure cross-workflow or cross-workspace scheduling dependencies within the same region?

• How do I configure an ancestor node for the start node of a workflow?

• How do I configure Node A, Node B, and Node C to run in sequence once per hour?

• How do I configure dependencies for a node that depends on multiple nodes?

Troubleshooting

• When committing a node, I get an error that the ancestor node's output name does not exist. What do I do?

• When committing a node, I get an error that the node's input and output are inconsistent with the data lineage in the code. What do I do?

• The system added an output name to Parent Nodes via automatic parsing, but an error says the output name does not exist. What do I do?

• The name and ID of a descendant node are empty and cannot be specified in my node's output. Why does this happen?

• Why do I find a non-existent output name of Node B when searching for ancestor nodes of Node A?

• When undeploying a node, I get an error that it has descendant nodes, but no descendant nodes appear on the Properties tab. Why?

• I configured rerun properties but the node doesn't rerun after failure, and it shows "Task Run Timed Out, Killed by System!!!". What do I do?

Cross-cycle scheduling dependencies

• When do I need to configure an instance in the current cycle to depend on the instance from the previous cycle?

• Why do some scheduling dependencies appear as dashed lines in Operation Center?

• I configure a daily node to depend on an hourly node. What effect does this have on the daily node and its descendants?

• How do I configure a scheduling dependency where a daily node depends on an hourly node?

• When does a daily node start running if I configure an hourly node as its ancestor?

• How do I configure a daily node instance to depend on a specific instance generated on the current day for an hourly node?

• How do I configure a daily node instance to depend on all instances generated on the previous day for an hourly node?

• Node B (daily) starts only after all of Node A's (hourly) instances for the current day succeed. Will Node B be affected if Node A still runs on the next day?

• Node A runs every hour. How do I configure Node B (daily) to run automatically after Node A's first instance of the day succeeds?