Platform For AI:Notebook

How it works

The Notebook component supports two modes:

In Development mode, start a DSW instance from the Notebook node. The instance automatically loads output data and state from upstream components, so you can inspect upstream results, develop logic, and pass results downstream — all without leaving the pipeline canvas.

In Running mode, triggering the pipeline (manually or via a DataWorks schedule) starts a DLC job that runs the Notebook file converted by Jupyter nbconvert.

Component description

The Notebook component has four input ports and four output ports:

Use the pai-notebook-utils package to access port configurations and custom parameters from within the Notebook.

pip install --upgrade https://pai-sdk.oss-cn-shanghai.aliyuncs.com/pai_notebook_utils/dist/pai_notebook_utils-0.0.1-py2.py3-none-any.whl

from pai_notebook.utils.notebook_utils import NotebookUtils

node_inputs = NotebookUtils.get_inputs()      # list of input port configs
node_outputs = NotebookUtils.get_outputs()    # list of output port configs
custom_params = NotebookUtils.get_custom_params()  # dict of custom parameters

Input ports

get_inputs() returns a list. Each item contains:

MaxCompute table input — location_type is MaxComputeTable; value contains project, table, and endpoint:

from pai_notebook.utils.notebook_utils import NotebookUtils

node_inputs = NotebookUtils.get_inputs()

for node_input in node_inputs:
    if node_input.location_type == "MaxComputeTable":
        input_name = node_input.name
        table_name = node_input.value["table"]
        project = node_input.value["project"]
        endpoint = node_input.value["endpoint"]
        print(f"input_name: {input_name}, project: {project}, table_name: {table_name}, endpoint: {endpoint}")

OSS input — location_type is OSS; the OSS path is mounted to a local path. value contains key, bucket, endpoint, and mountPath:

from pai_notebook.utils.notebook_utils import NotebookUtils

node_inputs = NotebookUtils.get_inputs()

for node_input in node_inputs:
    if node_input.location_type == "OSS":
        input_name = node_input.name
        key = node_input.value["key"]
        bucket = node_input.value["bucket"]
        endpoint = node_input.value["endpoint"]
        mount_path = node_input.value["mountPath"]
        print(f"input_name: {input_name}, bucket: {bucket}, key: {key}, endpoint: {endpoint}, mount path: {mount_path}")

Output ports

get_outputs() returns a list. Each item contains:

• OSS outputs (output1, output2): data is written to the configured OSS path. If Job Output Path is not set, the path is auto-generated from the global default path. The OSS directory is mounted to /ml/output/ in the job container: output1 maps to /ml/output/output1/, and output2 maps to /ml/output/output2/. The Notebook run result (HTML) is saved to /ml/output/result/.

• MaxCompute outputs (output3, output4): the workspace must be associated with a MaxCompute compute engine.

from pai_notebook.utils.notebook_utils import NotebookUtils

node_outputs = NotebookUtils.get_outputs()

for node_output in node_outputs:
    output_name = node_output.name
    if node_output.location_type == "MaxComputeTable":
        table_name = node_output.value["table"]
        project = node_output.value["project"]
        endpoint = node_output.value["endpoint"]
        print(f"output_name: {output_name}, project: {project}, table_name: {table_name}, endpoint: {endpoint}")
    elif node_output.location_type == "OSS":
        key = node_output.value["key"]
        bucket = node_output.value["bucket"]
        endpoint = node_output.value["endpoint"]
        mount_path = node_output.value["mountPath"]
        print(f"output_name: {output_name}, bucket: {bucket}, key: {key}, endpoint: {endpoint}, mount path: {mount_path}")
    else:
        print(json.dumps(node_output.value, indent=4))

Custom parameters

Configure custom parameters as key-value pairs in the Notebook node. Reference global variables using the ${globalParamName} format. For example, set a parameter key to modelVersion and its value to ${globalModelVersion}. At runtime, get_custom_params() returns the resolved value:

from pai_notebook.utils.notebook_utils import NotebookUtils

custom_params = NotebookUtils.get_custom_params()
print(custom_params)
# Example output: {'modelVersion': '2.1.0', 'batchSize': '32'}

Custom parameters are in the key-value pair format and can be shared by a pipeline and a DSW instance, which facilitates parameter modification.

Notebook configuration

Add the Notebook component to a pipeline in Machine Learning Designer, then configure the following parameters in the right panel.

What's next

• Use DataWorks to schedule pipelines

• Configure global variables

• Billing of DSW

• Billing of DLC