Platform For AI:AIMaster: Elastic automatic fault tolerance engine

After you enable fault tolerance monitoring for a job, to receive notifications for fault tolerance events, go to the Workspace Details page. Choose Workspace Configuration > Event Notification Configuration. Then, click Create Event Rule and set the event type to DLC Job > Automatic Fault Tolerance. For more information, see Event Center for a workspace.

When a training job encounters an exception, such as a NaN loss, you can use the AIMaster SDK in your code to send a custom notification message:

from aimaster import job_monitor as jm

job_monitor_client = jm.Monitor(config=jm.PyTorchConfig())

...

if loss == Nan and rank == 0:
  st = job_monitor_client.send_custom_message(content="The training loss of the job is NaN.")
  if not st.ok():
      print('failed to send message, error %s' % st.to_string())

Fault tolerance monitoring has built-in detection for common retriable errors. You can also define custom keywords. If a failed instance's log contains one of these keywords, AIMaster treats the error as retriable. The monitoring module scans the end of the failed instance's logs to find these keywords.

• Example of configuring custom fault tolerance keywords for a PyTorch job

  from aimaster import job_monitor as jm
  
  jm_config_params = {}
  jm_config = jm.PyTorchConfig(**jm_config_params)
  monitor = jm.Monitor(config=jm_config)
  monitor.set_retryable_errors(["connect timeout", "error_yyy", "error_zzz"])

  The parameters configured in monitor.set_retryable_errors are the custom fault tolerance keywords.

• Example of configuring custom fault tolerance keywords for a TensorFlow job

  from aimaster import job_monitor as jm
  
  jm_config_params = {}
  jm_config = jm.TFConfig(**jm_config_params)
  monitor = jm.Monitor(config=jm_config)
  monitor.set_retryable_errors(["connect timeout", "error_yyy", "error_zzz"])

By default, hang detection settings apply to the entire job. However, jobs run in stages. For example, during initialization, nodes may take longer to establish communication, while during training, logs update more frequently. To detect hangs faster during the training process, DLC provides staged hang detection, which allows you to configure different detection intervals for different stages.

monitor.reset_config(jm_config_params)

# Example:
#     monitor.reset_config(job_hang_interval=10)
#     or
#     config_params = {"job_hang_interval": 10, }
#     monitor.reset_config(**config_params)

The following is an example of staged hang detection for a PyTorch job.

import torch
import torch.distributed as dist
from aimaster import job_monitor as jm

jm_config_params = {
    "job_hang_interval": 1800 # Global 30-minute detection.
}
jm_config = jm.PyTorchConfig(**jm_config_params)
monitor = jm.Monitor(config=jm_config)

dist.init_process_group('nccl')

...

# impl these two funcs in aimaster sdk
# user just need to add annotations to their func
def reset_hang_detect(hang_seconds):
    jm_config_params = {
        "job_hang_interval": hang_seconds
    }
    monitor.reset_config(**jm_config_params)

def hang_detect(interval):
    reset_hang_detect(interval)
    ...

@hang_detect(180) # reset hang detect to 3 min, only for func scope
def train():
    ...

@hang_detect(-1) # disable hang detect temporarily, only for func scope
def test():
    ...

for epoch in range(0, 100):
    train(epoch)
    test(epoch)
    self.scheduler.step()