Integrate SchedulerX 2.0 with Simple Log Service to collect and view job logs - Microservices Engine

When jobs fail or behave unexpectedly, you need execution logs to pinpoint the root cause. By integrating Simple Log Service with SchedulerX 2.0, you can collect job execution logs -- including distributed job logs -- and view them directly in the SchedulerX console. Add a Log4j 2, Log4j, or Logback appender to start collecting logs from triggered jobs.

Prerequisites

Before you begin, ensure that you have:

A namespace created in SchedulerX
An application group created in SchedulerX
SchedulerX 2.0 Professional Edition with Simple Log Service activated

Note

Simple Log Service retains logs for up to two weeks. Expired logs are automatically deleted.

How it works

Configure a SchedulerX log appender in your logging framework (Log4j 2, Log4j, or Logback).
In your job code, write logs using a logger named schedulerx.
When a job runs, SchedulerX collects the logs and displays them in the console.

Choose a log collection method

SchedulerX supports two log collection methods. Choose one before configuring the appender.

Method	Collects	Best for	Trade-off
Job logs only	Logs written to the `schedulerx` logger	Targeted troubleshooting of specific jobs	Requires a dedicated logger in your code
All service logs	All application logs	Full visibility into the application	Mixes job logs with service logs, reducing query efficiency

Update the SchedulerX agent

Update the SchedulerX agent to the latest version. For the latest version number, see Release notes.

Add the following dependency to your pom.xml file (Spring Boot Starter example):

<dependency>
    <groupId>com.aliyun.schedulerx</groupId>
    <artifactId>schedulerx2-spring-boot-starter</artifactId>
    <version>${Latest version of the SchedulerX client}</version>
</dependency>

Configure a log appender

Select your logging framework below. Each framework supports both log collection methods.

Log4j 2

Collect job logs only

Add an appender and a dedicated logger to your Log4j 2 configuration file:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="off">
    <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout
                pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%t] %-5level %logger{36} - %m%n" />
        </Console>
        <SchedulerxLog4j2Appender name="schedulerxLog"
            timeFormat="yyyy-MM-dd'T'HH:mmZ"
            timeZone="UTC"
            ignoreExceptions="true">
            <PatternLayout pattern="%d %-5level [%thread] %logger{0}: %msg"/>
        </SchedulerxLog4j2Appender>
    </Appenders>

    <Loggers>
        <Root level="info">
            <AppenderRef ref="Console" />
        </Root>
        <Logger name="schedulerx" level="info" additivity="false">
            <AppenderRef ref="schedulerxLog" />
        </Logger>
    </Loggers>
</Configuration>

In your job class, create a logger named schedulerx and use it to write logs:

package com.hxm.test.processor;

import com.alibaba.schedulerx.worker.domain.JobContext;
import com.alibaba.schedulerx.worker.processor.JavaProcessor;
import com.alibaba.schedulerx.worker.processor.ProcessResult;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;


@Component
public class HelloWorldJob3 extends JavaProcessor {
    // Use "schedulerx" as the logger name
    private static final Logger logger = LoggerFactory.getLogger("schedulerx");

    @Override
    public ProcessResult process(JobContext context) throws Exception {
        logger.info("hello HelloWorldJob3");
        return new ProcessResult(true);
    }

}

Collect all service logs

Add the schedulerxLog appender to the root logger. All application logs are then collected by SchedulerX.

Note

This approach mixes service logs with job logs, which can make troubleshooting harder and reduce query efficiency.

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="off">
    <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout
                pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%t] %-5level %logger{36} - %m%n" />
        </Console>
        <SchedulerxLog4j2Appender name="schedulerxLog"
            timeFormat="yyyy-MM-dd'T'HH:mmZ"
            timeZone="UTC"
            ignoreExceptions="true">
            <PatternLayout pattern="%d %-5level [%thread] %logger{0}: %msg"/>
        </SchedulerxLog4j2Appender>
    </Appenders>

    <Loggers>
        <Root level="info">
            <AppenderRef ref="Console" />
            <AppenderRef ref="schedulerxLog" />
        </Root>
    </Loggers>
</Configuration>

Log4j

Collect job logs only

Add an appender and a dedicated logger to your log4j.properties file:

log4j.rootLogger = INFO,console
log4j.logger.schedulerx=schedulerxLog
log4j.appender.schedulerxLog=com.alibaba.schedulerx.worker.log.appender.SchedulerxLog4jAppender

In your job class, create a logger named schedulerx and use it to write logs. The Java code is identical to the Log4j 2 example above -- use LoggerFactory.getLogger("schedulerx").

Collect all service logs

Add the schedulerxLog appender to the root logger in log4j.properties:

log4j.rootLogger = INFO,console,schedulerxLog
log4j.appender.schedulerxLog=com.alibaba.schedulerx.worker.log.appender.SchedulerxLog4jAppender

Logback

Note

The SchedulerxLogbackAppender class is in the schedulerx2-worker.jar package. Make sure the SchedulerX agent is updated to the latest version. For the latest version, see Release notes.

Collect job logs only

Add an appender and a dedicated logger to your logback.xml file:

<appender name="schedulerxLog" class="com.alibaba.schedulerx.worker.log.appender.SchedulerxLogbackAppender">
    <timeFormat>yyyy-MM-dd'T'HH:mmZ</timeFormat>
    <timeZone>UTC</timeZone>
</appender>

<root level="INFO">
    <appender-ref ref="CONSOLE" />
</root>

<logger name="schedulerx" level="INFO">
    <appender-ref ref="schedulerxLog"/>
</logger>

In your job class, create a logger named schedulerx and use it to write logs. The Java code is identical to the Log4j 2 example above -- use LoggerFactory.getLogger("schedulerx").

Collect all service logs

Add the schedulerxLog appender to the root logger in logback.xml:

<appender name="schedulerxLog" class="com.alibaba.schedulerx.worker.log.appender.SchedulerxLogbackAppender">
    <timeFormat>yyyy-MM-dd'T'HH:mmZ</timeFormat>
    <timeZone>UTC</timeZone>
</appender>

<root level="INFO">
    <appender-ref ref="CONSOLE" />
    <appender-ref ref="schedulerxLog"/>
</root>

View job logs

Log on to the MSE SchedulerX console.
In the left-side navigation pane, click Task Management.
Find the target job and choose more > Historical records in the Operation column.
In the Task instance records panel, find the target job instance and click Log in the Operation column.

Log fields

Each log entry contains the following fields:

Field	Description
`ip`	IP address of the worker that generated the log
`executionId`	Execution ID in the format `${jobId}_${jobInstanceId}_${taskId}`
`level`	Log level (for example, INFO or ERROR)
`log`	Log content
`throwable`	Exception stack trace (present only when an error occurs)
`time`	Timestamp when the log was generated

Query historical logs

SchedulerX retains only the 60 most recent execution records per job instance. To search older records, click Log Query in the left-side navigation pane of the SchedulerX console. On the Log Query page, filter by job ID, keyword, or time range.

Troubleshooting examples

Diagnose a service failure

After integrating Simple Log Service, SchedulerX collects both job logs and service logs, including exceptions. The following example demonstrates how to identify a division-by-zero error in a service call.

Create a job that calls a service method and logs with the schedulerx logger:

package com.hxm.test.processor;

import com.alibaba.schedulerx.test.service.TestService;
import com.alibaba.schedulerx.worker.domain.JobContext;
import com.alibaba.schedulerx.worker.processor.JavaProcessor;
import com.alibaba.schedulerx.worker.processor.ProcessResult;

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

@Component
public class HelloWorldJob extends JavaProcessor {
    // Use the schedulerx logger to collect logs
    private static final Logger LOGGER = LogManager.getLogger("schedulerx");

    @Autowired
    private TestService testService;

    @Override
    public ProcessResult process(JobContext context) throws Exception {
        String parameters = context.getJobParameters();
        String tokens[] = parameters.split(" ");
        int a = Integer.valueOf(tokens[0]);
        int b = Integer.valueOf(tokens[1]);
        int c = testService.doDivision(a, b);
        LOGGER.info("testService.doDivision finished, a={}, b={}, c={}", a, b, c);
        if (c < 0) {
            return new ProcessResult(false, "result=" + c);
        }
        return new ProcessResult(true);
    }

}

Create the service class:

package com.hxm.test.service;

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.springframework.stereotype.Service;

@Service("testService")
public class TestServiceImpl implements TestService {
    // Use the schedulerx logger to collect logs
    private static final Logger LOGGER = LogManager.getLogger("schedulerx");

    @Override
    public int doDivision(int a, int b) {
        try {
             LOGGER.info("start to do division c = " + a + "/" + b);
             int c = a/b;
             LOGGER.info("c=" + c);
             return c;
        } catch (Exception e) {
            LOGGER.error("", e);
        }
        return -1;
    }

}

Configure the job in the SchedulerX console. For example, set the job parameters to 1 0 to trigger a division-by-zero error.
Run the job once, then open the Task instance records panel and view the logs. The log shows that the job failed because TestServiceImpl threw a "by zero" exception.

Diagnose a distributed job failure

Distributed jobs in SchedulerX 2.0 dispatch subtasks across multiple workers for batch processing. If a subtask fails, use logs to identify which specific task failed and why.

Create a distributed job that dispatches subtasks and logs with the schedulerx logger:

package com.hxm.test.processor;

import java.util.ArrayList;
import java.util.List;

import com.alibaba.schedulerx.worker.domain.JobContext;
import com.alibaba.schedulerx.worker.processor.MapJobProcessor;
import com.alibaba.schedulerx.worker.processor.ProcessResult;

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;

public class TestMapJobProcessor extends MapJobProcessor {
    private static final Logger LOGGER = LogManager.getLogger("schedulerx");

    @Override
    public ProcessResult process(JobContext context) throws Exception {
        String taskName = context.getTaskName();
        String parameter = context.getJobParameters();
        int dispatchNum = Integer.valueOf(parameter);
        if (isRootTask(context)) {
            LOGGER.info("start root task");
            List<String> msgList = new ArrayList<>();
            for (int i = 0; i <= dispatchNum; i++) {
                msgList.add("msg_" + i);
            }

            return map(msgList, "Level1Dispatch");
        } else if (taskName.equals("Level1Dispatch")) {
            String task = (String)context.getTask();
            if (task.equals("msg_23")) {
                LOGGER.error("msg={}, failed", task);
                return new ProcessResult(false);
            } else {
                LOGGER.info("msg={}, success", task);
            }
            return new ProcessResult(true);
        }
        return new ProcessResult(false);
    }
}

Configure the job in the SchedulerX console and trigger it.
Open the job logs as described in View job logs.
Search for keywords (such as failed or error) to locate the failed subtask and its root cause.