Platform For AI:TensorBoard visualization

Prerequisites

Before you begin, make sure the following requirements are met:

• A DLC training job with at least one dataset mounted (verify on the Overview tab). Supported storage types: Object Storage Service (OSS) and Apsara File Storage NAS.

• TensorFlow's SummaryWriter integrated into the training script to write logs to a directory within the mounted dataset.

Write training logs

Add SummaryWriter to your training script to write logs to a path inside the mounted dataset. Close the writer after training to ensure all log data is flushed.

Example:

• OSS dataset endpoint: oss://your-bucket-name.oss-region-internal.aliyuncs.com/dlc-training-data/

• Dataset mount path in the container: /mnt/data/

• Log output directory: /mnt/data/output/runs

import tensorflow as tf

# Initialize SummaryWriter with logging directory
log_dir = "/mnt/data/output/runs"
writer = tf.summary.create_file_writer(log_dir)

# Training loop with metric logging
for epoch in range(num_epochs):
    train_loss = calculate_loss()
    train_accuracy = calculate_accuracy()

    with writer.as_default():
        # Log scalar metrics
        tf.summary.scalar("train_loss", train_loss, step=epoch)
        tf.summary.scalar("train_accuracy", train_accuracy, step=epoch)

        # Log histograms and images every 10 epochs
        if epoch % 10 == 0:
            tf.summary.histogram("model_weights", model.weights, step=epoch)
            tf.summary.image("prediction_samples", sample_predictions, step=epoch)

# Close writer to ensure all logs are saved
writer.close()
print(f'Training completed successfully. Logs saved to: {log_dir}')

The SummaryWriter path (/mnt/data/output/runs) is the Summary Path specified when creating the TensorBoard instance.

Create a TensorBoard instance

1. Log on to the PAI console. Select the region in the top navigation bar and select a workspace. Click Enter Deep Learning Containers (DLC).

2. In the Actions column of the target job, click TensorBoard. In the panel, click Create TensorBoard.

3. On the Create TensorBoard page, configure the parameters, then click OK.

View TensorBoard reports

1. In the left-side navigation pane of the workspace page, choose AI Computing Asset Management > Job.

2. On the TensorBoard tab, check the Status column. When the status is Running, click View TensorBoard in the Actions column.

The TensorBoard page displays your training metrics and logged data, such as scalar trends, histograms, and images.

Manage TensorBoard instances

1. Log on to the PAI console. Select the region in the top navigation bar and select a workspace. Click Enter Jobs.

2. On the TensorBoard tab, perform any of the following operations:

Troubleshooting

Best practices

• Start with 2 to 4 vCPUs and 4 to 8 GiB of memory for typical workloads.

• Organize logs by experiment type and timestamp (for example, /mnt/data/output/runs/experiment_type/date_time/).

• Clean up obsolete training logs regularly to reduce storage costs and improve performance.

• Use RAM role assignments to restrict TensorBoard instance access to authorized team members.

References

• Create and manage TensorBoard instances: Create a TensorBoard instance from the AI Computing Asset Management > Job page.