Platform For AI:TensorBoard

Prerequisites

Before you begin, make sure that you have:

• A DLC training job with at least one dataset mounted (verify on the job's Overview tab). Supported storage types: Object Storage Service (OSS) and Apsara File Storage NAS

• TensorFlow's SummaryWriter integrated into the training script to write logs to a directory within the mounted dataset

Write training logs

Add a SummaryWriter to the training script. Write logs to a path inside the mounted dataset, and close the writer after training completes to prevent data loss.

Example setup:

• OSS dataset endpoint: oss://your-bucket-name.oss-region-internal.aliyuncs.com/dlc-training-data/

• Dataset mount path in the container: /mnt/data/

• Log output directory: /mnt/data/output/runs

import tensorflow as tf

# Initialize SummaryWriter with the logging directory
log_dir = "/mnt/data/output/runs"
writer = tf.summary.create_file_writer(log_dir)

# Training loop with metric logging
for epoch in range(num_epochs):
    train_loss = calculate_loss()
    train_accuracy = calculate_accuracy()

    with writer.as_default():
        # Log scalar metrics
        tf.summary.scalar("train_loss", train_loss, step=epoch)
        tf.summary.scalar("train_accuracy", train_accuracy, step=epoch)

        # Log histograms and images every 10 epochs
        if epoch % 10 == 0:
            tf.summary.histogram("model_weights", model.weights, step=epoch)
            tf.summary.image("prediction_samples", sample_predictions, step=epoch)

# Close the writer to ensure all logs are saved
writer.close()
print(f'Training completed successfully. Logs saved to: {log_dir}')

The SummaryWriter path (/mnt/data/output/runs) is the Summary Path to specify when creating the TensorBoard instance.

Create a TensorBoard instance

1. Log on to the PAI console. Select the region in the top navigation bar and select a workspace. Click Enter Deep Learning Containers (DLC).

2. In the Actions column of the target job, click TensorBoard. In the TensorBoard panel, click Create TensorBoard.

3. On the Create TensorBoard page, configure the parameters described in the following sections, then click OK.

View TensorBoard reports

1. In the left-side navigation pane of the workspace page, choose AI Computing Asset Management > Job.

2. On the TensorBoard tab, check the Status column. If the status is Running, click View TensorBoard in the Actions column.

The TensorBoard visualization page opens, displaying training metrics and logged data.

Manage TensorBoard instances

1. Log on to the PAI console. Select the region in the top navigation bar and select a workspace. Click Enter Jobs.

2. On the TensorBoard tab, perform any of the following operations.

Troubleshooting

Best practices

• Start with 2 to 4 vCPUs and 4 to 8 GiB of memory for typical workloads.

• Organize logs by experiment type and timestamp (for example, /mnt/data/output/runs/experiment_type/date_time/).

• Clean up obsolete training logs regularly to reduce storage costs and improve performance.

• Use RAM role assignments to restrict TensorBoard instance access to authorized team members.

References

• Create and manage TensorBoard instances: Create a TensorBoard instance from the AI Computing Asset Management > Job page.