Platform For AI:Create and manage distributed training jobs

Account and permission requirements

• Alibaba Cloud account: This account can perform all operations without additional authorization.

• RAM user: Grant permissions to a RAM user by adding them as a workspace member with an appropriate role. For details on the permissions for each role, see Appendix: Roles and permissions.

Create a distributed training job

You can create a DLC distributed training job on the Deep Learning Containers (DLC) tab. The procedure is as follows.

1. Go to the task management page.

   1. Log on to the PAI console.

   2. In the left-side navigation pane, click Workspaces. On the workspace list page, click the name of your target workspace.

   3. In the left-side navigation pane of the workspace, choose AI Computing Asset Management > Jobs to open the task management page.

2. On the Deep Learning Containers (DLC) tab, click Create Task.

3. On the Create Task page, configure the parameters and click OK.

   For more information about how to configure the parameters, see Create a training job.

Manage distributed training jobs

Query aggregated logs by keyword

Procedure

You can search for specific log events by keyword on the Log tab. Follow these steps:

1. In the left-side navigation pane, choose AI Computing Asset Management > Jobs. On the Deep Learning Containers (DLC) page, click a job name.

2. Click the Log tab and configure the settings.

   1. Above the Task Information section, select a time range for log collection.

      Note

      Log collection may continue after the job has finished. Select a time range that reflects this.

   2. In the Instances area, select the required instances.

   3. In the search box on the right, enter keywords to search for relevant logs or events.

Basic query rules

Keyword search in aggregated logs matches complete words. Because the underlying Simple Log Service (SLS) for DLC uses a tokenizer, an exact string match for a phrase is not guaranteed.

For example, a search for abc def returns all logs containing both abc and def, not just logs containing the exact phrase abc def.

Fuzzy query rules

You can use wildcards for fuzzy queries in aggregated logs. The supported wildcards are the asterisk (*) and the question mark (?). Other special characters are invalid for fuzzy queries.

• An asterisk (*) represents a sequence of zero or more characters. A question mark (?) represents a single character.

• The asterisk (*) and question mark (?) must be placed in the middle or at the end of a keyword; they are ineffective at the beginning.

For example, the keyword abc* finds words that start with abc. The keyword ab?d finds four-character words that start with ab and end with d.

Delimiter limitations

The underlying SLS log service for DLC uses the following characters as built-in delimiters to tokenize training logs: , '";=()[\",\"]{}?@&<>/:\n\t\r

These delimiters split log content into searchable tokens. If you search for a keyword that consists only of these delimiter characters, the service does not treat it as a complete token, and the search returns no results.

Example 1: A search for &&& will not return any relevant logs. To get relevant results, build a keyword that includes context from the surrounding text.

Example 2: To find logs containing a&b, use the full expression as your keyword instead of just &. This returns logs that contain both a and b. More specific keywords yield more accurate results.

Keyword examples