All Products
Search
Document Center

DataWorks:Use the engine O&M feature

Last Updated:Aug 15, 2023

A DataWorks node instance that runs on the E-MapReduce (EMR) compute engine contains multiple EMR jobs. These EMR jobs are run based on a specific sequence. You can use the engine operations and maintenance (O&M) feature of DataWorks to view the details of each EMR job and find and remove jobs that fail to be run. This prevents failed jobs from affecting the running of both the DataWorks node instances to which the jobs belong and the nodes that generate the node instances.

Limits

  • DataWorks allows you to perform O&M only for the EMR compute engine. You must upgrade your EMR execution package to obtain O&M data. If you want to use the engine O&M feature, submit a ticket to upgrade your EMR execution package.

  • Engine Maintenance is displayed in the left-side navigation pane of the Operation Center page only after you associate an EMR compute engine instance with your workspace.

Go to the EMR engine O&M page

  1. Log on to the DataWorks console. In the left-side navigation pane, choose Data Modeling and Development > Operation Center. On the page that appears, select the desired workspace from the drop-down list and click Go to Operation Center.

  2. In the left-side navigation pane, choose Engine Maintenance > E-MapReduce to go to the EMR engine O&M page.

View EMR jobs

On the EMR engine O&M page, you can view the details of the EMR jobs that are running for the EMR compute engine instance associated with your DataWorks workspace in the current region. You can perform the following operations based on your business requirements:Engine Maintenance

  • Search for an EMR job.

    You can specify different conditions in the upper part of the EMR engine O&M page to search for an EMR job.

    Condition

    Description

    instance ID

    You can search for a specific EMR job by job ID. Alternatively, you can specify an instance ID to search for all the EMR jobs that belong to the related DataWorks node instance.

    Note
    • On the EMR engine O&M page, the data of an EMR compute engine instance over the last three days is displayed.

    • If you search for a specific EMR job by job ID or instance ID, the data of the EMR compute engine instance only over the last 30 days is displayed.

    Status

    • NEW: indicates that an EMR job is newly created.

    • NEW_SAVING: indicates that the EMR job is being saved.

    • SUBMITTED: indicates that a running application has been submitted for the EMR job.

    • ACCEPTED: indicates that the scheduling system approves the running application.

    • RUNNING: indicates that the EMR job is being run.

      Note

      If an EMR job is in the RUNNING state for a long time, you can manually stop the running of the DataWorks node instance to which the EMR job belongs. This prevents the EMR job from occupying resources and affecting the running of the DataWorks node that generates the node instance. For more information, see Stop a DataWorks node instance.

    • FINISHED: indicates that the running of the EMR job is complete.

    • SUCCESSED: indicates that the EMR job is successfully run.

    • FAILED: indicates that the EMR job fails to be run. If an EMR job is in the FAILED state, you must locate and troubleshoot the issue at the earliest opportunity. This can prevent the EMR job from affecting both the running of the descendant jobs and the DataWorks node generating the node instance to which the EMR job belongs.

    • KILLED: indicates that the EMR job is stopped by the user who runs the job or the administrator.

    EMR Node Type

    Valid values: MAPREDUCE and SPARK.

  • View the details of an EMR job.

    • You can view the basic information of an EMR job, such as the ID, status, start time, and end time of the job. You can also view the DataWorks node instance to which an EMR job belongs and the ID of the DataWorks node instance.

      Note

      Different EMR jobs may belong to DataWorks node instances with the same name. However, if these EMR jobs start to run at different points in time, they actually belong to different DataWorks node instances. To determine whether EMR jobs belong to the same DataWorks node instance, you can view the ID in the Node Instance ID column that corresponds to each EMR job.

    • You can find an EMR job in the FAILED state, and click the ID of the job in the Job ID column or click the name in the DataWorks Node Instance column. On the details page of the job, view and troubleshoot issues.

    • You can click Start time or End At to arrange the EMR jobs in chronological or reverse chronological order. This way, you can clearly understand the running sequence of EMR jobs and the status of your desired EMR job.

  • Stop a DataWorks node instance.

    If an EMR job is in the RUNNING state for a long time, you can select the EMR job and click Stop DataWorks Node Instances in the lower-left corner of the EMR engine O&M page. This way, you can stop the running of the DataWorks node instance to which the EMR job belongs. An EMR job may be in the RUNNING state for a long time because of an internal error and the job cannot be automatically stopped. To prevent the job from occupying resources and affecting the running of other jobs, you must manually stop the job and troubleshoot issues at your earliest opportunity.

    Important
    • If multiple EMR jobs belong to the same DataWorks node instance and you stop one of these EMR jobs, the DataWorks node instance enters the FAILED state.

    • You can stop EMR jobs only in the RUNNING state in a DataWorks node instance.

    • After you stop a running EMR job in a DataWorks node instance, the DataWorks node instance enters the FAILED state. In this case, the DataWorks node that generates the DataWorks node instance is blocked. Exercise caution when you stop a running EMR job.