LogView is a tool used to record and display the running status of MaxCompute jobs. LogView is now compatible with MaxFrame. This topic describes how to use LogView 2.0 to view running information about MaxFrame jobs.
Overview
LogView 2.0 is compatible with MaxFrame and supports the following features:
Viewing execution records and running durations of all directed acyclic graphs (DAGs) submitted in MaxFrame sessions
Interactive viewing of execution sequence, running time, operator topology, and status relationships of sub DAGs in each DAG
Viewing settings, status, memory usage, and CPU utilization of each child instance
For information about the general capabilities of LogView, see Use LogView 2.0 to view job running information. This topic describes how to view running information about MaxFrame jobs.
MaxFrame LogView entry
When you create a session using session = new_session(...), the system automatically generates a URL that starts with https://logview.aliyun.com/logview.
After you use MaxFrame to submit a job in the DataWorks console, you can click the URL to open the MaxFrame LogView page.
When you use MaxFrame to submit a job in the local environment, you can use the
session.get_logview_address()method to obtain and print the URL. You can then copy the URL to your browser to access the page.
The following figure shows the MaxFrame LogView page:
Number | Area |
① | This section displays the job ID and job name. For more information, see Title and function section. |
② | Basic Info. For more information, see Basic Info. |
③ | DAG List section. For more information, see DAG List. |
Title and function section
This section displays the job ID and job name. The job ID uniquely identifies a job. The job name is a custom name that you enter. You can also click the icons on the right of this section to perform operations.
Icon | Feature |
| Obtain the MaxFrame LogView URL. |
| Save the job details as a file to your on-premises machine. |
| Stop the current job process. |
| Open the LogView_detail.txt file that contains job details. The file is saved on your on-premises machine. |
| Open the job list and view the job information under the specified project. |
| Set whether to enable auto-refresh. If auto-refresh is enabled, the page is refreshed every 10 seconds. |
DAG List
This section displays DAGs in descending order based on execution time. The most recently submitted job appears at the top of the list. The following table describes the basic information.
Parameter | Description |
Id | The ID of the DAG. The unique ID that is generated each time the DAG is executed. |
Detail | Click |
Start Time | The time at which the DAG is submitted. |
End Time | The end time of the DAG. If the value of this parameter for a DAG is empty, the DAG is still running. |
Duration | The running duration of the DAG, which is the period from the start time to the end time or the current time. |
Status | The status of the job. Valid values:
|
Description | The description of the DAG. |
to view the details of the corresponding sub DAG.Sub DAG details
The following figure shows the Sub DAG details page.
Number | Area |
① | Sub DAG Graph display area. For more information, see Sub DAG Graph. |
② | Sub DAG list. For more information, see Sub DAG list. |
③ | Instance list of the sub DAG. For more information, see Instance list. |
④ | Basic Info of the sub DAG. For more information, see Basic Info. |
Sub DAG graph
The sub DAG graph displays the details of the current DAG.
Number | Description |
① | The ID of the sub DAG. Click the ID to return to the DAG list page. |
② | The breadcrumb navigation for switching the task level of the sub DAG. |
③ | The troubleshooting tool. It includes Progress Chart, Input Heat Chart, Output Heat Chart, TaskTime Heat Chart, and InstanceTime Heat Chart. |
④ | In this area, you can refresh the job execution status ( |
⑤ | The zoom tool. |
⑥ | A table node, which represents a MaxCompute table. |
⑦ | A sub DAG node, which contains the following content:
|
⑧ | The data flow, which indicates the data flow direction between stages or between stages and tables. |
⑨ | The canvas preview. |
), display the job execution graph in full screen or zoom in or out (
), obtain MaxCompute Studio documentation (
), and switch to the previous level of the task (
).Sub DAG list
This section displays running information about all sub DAGs in the current DAG. The sub DAGs are sorted in chronological order.
Parameter | Description |
Sub DAG | The name of the current sub DAG. |
Failed/Terminated/ALL | The number of MaxCompute instances in different states in the current sub DAG. Note If (+X fallbacks) is displayed, MaxCompute Query Acceleration (MCQA) jobs on a specific number of instances are rolled back to offline SQL jobs. This is not a data error. X indicates the number of instances. |
Status | The state of the current sub DAG.
|
Progress | The execution progress of the sub DAG. Note If the progress reaches 100% but the sub DAG remains in the Running state, the execution of all instances is complete. This does not indicate that the execution of the sub DAG is complete. |
Start Time | The time at which the sub DAG is submitted. |
End Time | The time at which the execution of the sub DAG is complete. |
Duration | The running duration of the sub DAG. |
Timeline | The timeline preview. You can use the timeline to view the start sequence and duration. |
Instance list
The instance list shows the jobs of MaxCompute instances related to a sub DAG.
Parameter | Description |
Instance ID | The MaxCompute instance ID of the job. |
Status | The status of the job.
|
Detail | The details of the job. |
Start Time | The time at which the job is submitted. |
End Time | The time when the job is complete. |
Duration | The duration for which the job is run. |
Timeline | The timeline preview. You can use the timeline to view the start sequence and duration. |
Fuxi Sensor
Fuxi Sensor provides a comprehensive resource view of MaxFrame jobs. You can use Fuxi Sensor to view the memory usage and CPU utilization of a Fuxi instance. You can also use Fuxi Sensor to locate job issues and analyze job performance. For example, you can use Fuxi Sensor in the following scenarios:
If an out-of-memory (OOM) error occurs, analyze the memory usage.
View the following resource usage of Fuxi instances in MaxFrame jobs:
CPU utilization
The cpu_usage chart has two lines. One line indicates the number of CPUs requested (cpu_plan), and the other line indicates the number of CPUs used (cpu_usage). On the y-axis, 400 indicates four processors. Currently, you can adjust the number of CPUs requested only by adjusting the number of processors. You cannot adjust the CPU utilization.
Memory usage
The mem_usage chart has two lines. One line indicates the number of memory resources requested (mem_plan), and the other line indicates the number of memory resources used (mem_usage).
Memory details
Use cases
View the phase in which a DAG, sub DAG, or instance ran for the longest period of time
You can view the TimeLine column in the DAG list, sub DAG list, and instance list to quickly find the DAG, sub DAG, and instance that require the longest period of time to run at the current level.
If the list is too long and needs to be paginated, you can sort the list based on the values in the Duration column in descending order to quickly find the items that require the longest period of time to run.
View the cause of an error
If you find that a DAG fails to execute or the client returns an exception log, you can go to the MaxFrame LogView page and perform the following operations:
View the status of each DAG in the DAG list.
Find the most recent DAG record whose Status is Failed and click
in the Detail column to view the details.
On the DAG details page, view the sub DAG node that is displayed in red, such as
STAGE_1in the following figure.
Alternatively, view the error message on the Result tab.
Click the failed sub DAG node
STAGE_1. Then, click in the Detail column of the Instance list to view the details. You can find the failed instance in the following figure.
View the operators included in a sub DAG
You can use one of the following methods to view the operators included in a sub DAG:
Method 1: On the DAG details page, click the desired sub DAG node.
Method 2: On the DAG details page, double-click the desired sub DAG node.
The code line of the MaxFrame script that corresponds to the current operator is marked on the operator.
Some operators may have the
merged fromidentity. This usually indicates that the optimizer merges multiple operators after the DAG is submitted to generate a new operator that has higher performance and the same features.
View data of a table
View data flow structure
If you want to view information about a data flow, you can double-click the related sub DAG node on the DAG details page to go to the operator details page. Then, click the connection line between the operators to view the data type, index, and column name format.
Parameter description:
Type: dataframe indicates that the upstream operator outputs a
dataframe.DataFrame Index indicates an index.
None: int64 indicates that the current index contains an unnamed index of the
int64type.DataFrame Columns indicates the column information.
<column name>: <dtype> indicates all columns and column types contained in the current DataFrame.
DataFrame Level Names indicates the name of the DataFrame level.
View the structure of a data table
Click a MaxCompute table node in an operator graph or a sub DAG graph.
View job details
On the DAG details page, click in the Detail column of the Instance list to view the job details.