Documentation
Home Page DataWorks User Guide Operation Center Other O&M operations Engine O&M

Engine O&M

Updated at:
Copy as MD
Product

In DataWorks, a task instance running on the E-MapReduce (EMR) compute engine consists of one or more jobs. The engine O&M feature lets you view details for each E-MapReduce job. This helps you promptly identify and resolve failing jobs, preventing them from blocking downstream tasks and ensuring that the task instance runs smoothly.

Limitations

  • The engine O&M feature in DataWorks supports only EMR jobs. You must submit a ticket to upgrade the EMR execution package to ensure O&M data is retrieved correctly.

  • The Engine O&M menu does not appear in the left-side navigation pane of Operation Center for workspaces that do not have a registered EMR cluster.

  • If you use an exclusive resource group for scheduling, you must submit a ticket to request an upgrade. Otherwise, some fields on the Engine O&M page will display as hyphens (-).

Notes

When a YARN application is reused, its Job ID (application ID) on the engine O&M page remains the same, even if you run the task in different DataWorks modules.

Note

For example, for the EMR Kyuubi component, the share level parameter kyuubi.engine.share.level is set to USER by default. This means each user has one engine, and all engine jobs initiated by that user share a single application ID. After an EMR Kyuubi task is run in DataWorks DataStudio, an application ID is generated. If you then run the same task in DataAnalysis, it reuses the application ID from the DataStudio job instead of generating a new one. This behavior varies by EMR component, and the UI provides the definitive information.

  • The engine O&M page displays only the application ID that is generated the first time an EMR job is run in DataWorks.

  • After the DataWorks task instance for an EMR job finishes running (with a status of Successful or Failed), the YARN application may still be in the RUNNING state. For example, in Kyuubi, the idle session timeout, specified by kyuubi.session.engine.idle.timeout, typically determines whether a YARN application is retained for a period. A setting of PT30M for kyuubi.session.engine.idle.timeout means that the YARN application is retained for 30 minutes after the EMR Kyuubi job is complete. You can go to the EMR on ECS console to view the configurations of the corresponding service.

Prerequisites

You have already registered an EMR cluster with a DataWorks workspace and run the relevant EMR tasks.

  1. Register a cluster: See DataStudio (legacy): Bind EMR compute resources.

  2. Run an EMR task: See DataWorks On EMR usage notes.

Accessing the engine O&M page

  1. Log on to the DataWorks console. In the target region, click Data Development and O&M > Operation Center in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Operation Center.

  2. In the left-side navigation pane, choose Others > Engine O&M > E-MapReduce to go to the E-MapReduce engine O&M page.

Engine jobs

The E-MapReduce engine O&M page lists all E-MapReduce jobs from all DataWorks workspaces in the current region. On this page, you can view job details and perform O&M operations.

  • Filter E-MapReduce jobs

    In the top menu bar of the E-MapReduce engine O&M page, use the filters to find specific jobs by criteria such as job ID or job type.

    Note

    • By default, the Engine O&M page displays engine instance data from the last three days.

    • When filtering by DataWorks Instance ID, you can only search for IDs from Operation Center. Searches by Job ID or DataWorks Instance ID are limited to job instances from the last seven days.

  • Perform operations on E-MapReduce jobs

    In this area, you can view the details of selected jobs and perform O&M operations as needed.

    Feature

    Description

    View job details

    You can view the basic information about an E-MapReduce job, including the job ID, job status, run time, job source, and the DataWorks instance to which the job belongs.

    • Job status descriptions:

      • NEW: The job has just been created.

      • NEW_SAVING: The job is being saved.

      • SUBMITTED: A request to run the job has been submitted.

      • ACCEPTED: The scheduler has accepted the request to run the job.

      • RUNNING: The job is running.

        Note

        If a job remains in the RUNNING state for a long time, you can manually terminate the DataWorks task instance that runs the job. This prevents jobs with errors from occupying resources for extended periods and blocking downstream tasks.

      • FINISHED: The job has finished running.

      • SUCCEEDED: The job ran successfully.

      • FAILED: The job failed to run. If a job is in this state, you must promptly identify and resolve the error to avoid blocking downstream jobs and affecting normal task execution. Click the job ID or the ID of the DataWorks instance to navigate to the task details page for troubleshooting.

      • KILLED: The job was terminated by its executor or an administrator.

    • DataWorks Instance ID:

      Different E-MapReduce jobs may have the same DataWorks instance ID. However, if their Start time values are different, they belong to different DataWorks task instances. Therefore, you must use both the DataWorks Instance ID and the Start time to uniquely identify a DataWorks task instance.

      Note

      Tasks triggered from some DataWorks modules, such as Data Quality, DataStudio, and DataAnalysis, do not have instance IDs. For these tasks, the platform displays a hyphen (-).

    • EMR job type: You can view only jobs of the MAPREDUCE and SPARK types.

    • Sort by run time: You can sort by Start time or End Time in ascending or descending order to visually check the execution order and duration of jobs and understand their running status.

    • Job source: Displays the DataWorks module where the E-MapReduce job was run. You can go to the corresponding module in the Actions column to view task details.

    • Queue usage (%): The percentage of queue capacity used by the current job. This is the proportion of resources that the YARN cluster resource manager allocates to the queue when the job is running.

    Perform operations on task instances

    • Terminate a DataWorks task instance

      You should manually terminate any E-MapReduce job that remains in the RUNNING state for an unusually long time. Such jobs often fail to terminate automatically due to internal errors. Stopping these jobs prevents them from consuming resources indefinitely and blocking other tasks.

      • To terminate a single job, click Terminate in the Actions column for the corresponding job.

      • To terminate multiple jobs, select the target jobs and click Stop DataWorks Node Instances in the lower-left corner.

      Important

      • Only the Workspace Administrator, users with the O&M role, and task owners can terminate task instances.

      • If multiple E-MapReduce jobs belong to the same DataWorks task instance, terminating any of the jobs causes the DataWorks task instance to enter the FAILED state.

      • You can terminate only DataWorks task instances that are running.

      • Terminating a task instance sets its status to FAILED. A FAILED instance blocks downstream nodes from running, so proceed with caution.

    • Go to a module to view a task

      Click the module link in the Actions column (for example, Go to DataStudio) to open the originating DataWorks module and view the task's run details.

      Note

      • DataAnalysis: Only the file owner can open the SQL query file.

      • DataStudio: After you navigate to the DataStudio page, all developers in the workspace can view the task. However, only the user who ran the task can view its historical run records.
Previous：Other O&M operationsNext：Resource O&M