DMSNotebookOperator - Data Management - Alibaba Cloud Documentation Center

This topic describes the configuration for DMSNotebookOperator.

Overview

Runs a DMS-managed Notebook file (.ipynb).

Prerequisites

A workspace has been created.
A resource group has been created.
To reuse a session, ensure that a Notebook session is created.
To create a new session, you must have a template that contains the configuration for the Notebook session.

Parameters

Note

The file_path, run_params, profile, session_id, profile_id, cluster_id, session_name, profile_name, and cluster_name parameters support Jinja template.

Parameter	Type	Required	Description
file_path	string	Yes	The path to the Notebook file (.ipynb).
profile	dict	No	The configuration for the Notebook session. autoStopTime: The idle time after which the session automatically releases resources. mountPoints: The data storage location. The format is `[{"mntPath" : "/mnt/data*","dataPath" : "oss://test/*"},......]`, where `mntPath` is the mount path and `dataPath` is the OSS path. dependencies: Pypi package management. environments: Environment variables.
profile_id	string	No Note This parameter is required if you do not reuse a session.	profile_id: The configuration ID. profile_name: The configuration name. Note You must specify one of them. `profile_id` has a higher priority.
profile_name	string
cluster_type	string		The compute cluster type in the DMS workspace. Valid values: cpu spark
cluster_id	string		cluster_id: The compute cluster ID in the DMS workspace. cluster_name: The compute cluster name in the DMS workspace. Note You must specify one of them. `cluster_id` has a higher priority.
cluster_name	string
spec	string		The resource specification for the driver. Valid values: 1C4G: 1 vCPU, 4 GB 2C8G: 2 vCPUs, 8 GB 4C16G: 4 vCPUs, 16 GB 8C32G: 8 vCPUs, 32 GB 16C64G: 16 vCPUs, 64 GB
runtime_name	string		The image name.
session_id	string	No Note This parameter is required if you reuse a session.	Specifies the session to reuse. session_id: The session ID. session_name: The session name. Note You must specify one of them. `session_id` has a higher priority.
session_name	string
run_params	dict	No	Runtime parameters used to replace variables in the Notebook file.
timeout	int	No	The timeout period for the Notebook file execution, in seconds.
polling_interval	int	No	The interval, in seconds, at which the execution status is polled. The default value is 10. If this parameter is set to 0 or a negative number, the task is submitted without waiting for the result. Polling includes a built-in retry mechanism.

Example

Note

task_id and dag are Airflow-specific parameters. For more information, see the official Airflow documentation.

from airflow import DAG
from airflow.decorators import task
from airflow.models.param import Param
from airflow.operators.bash import BashOperator
from airflow.operators.empty import EmptyOperator

import json
from airflow.providers.alibabadms.cloud.operators.dms_notebook import DMSNotebookOperator

with DAG(
    "dms_notebook_test",
    params={
        "x":3
    },
) as dag:

    notebook_operator = DMSNotebookOperator(
        task_id='notebook_test_hz_name',
        profile_name='hansheng_profile.48',
        profile={},
        cluster_type='spark',
        cluster_name='spark_general2.218',
        spec='4C32G',
        runtime_name='Spark3.5_Scala2.12_Python3.9_General:1.0.9',
        file_path='/Workspace/code/default/test.ipynb',
        run_params={
            'a':"{{ params.x }}"
        },
        polling_interval=5,
        dag=dag
    )

    run_this_last = EmptyOperator(
        task_id="run_this_last",
        dag=dag,
    )

    notebook_operator >> run_this_last

if __name__ == "__main__":
    dag.test(
        run_conf={}
    )

Note

All DMS Airflow operators support common features, such as task cancellation and automatic retries. For more information, see Airflow DMS Operator.