Connect using an agent (for script, big data, and SQL jobs) - Microservices Engine

This topic describes how to install an agent in an ECS instance or a container service and connect to MSE-XXLJOB.

Solution overview

You can dynamically write or modify scripts and SQL in the MSE console to schedule and run scheduled tasks.

Procedure

Create a general-purpose application

Log on to the XXL-JOB console and select a region from the top menu bar.
Click the target instance to view its details.
In the navigation pane on the left, choose Application Management, and then click Create Application.
Set Application Type to General-purpose Application. Configure other parameters as needed, and then click OK.

Connect to an executor using an agent

You can deploy using an installation package, Docker, or Kubernetes.

Deploy using an installation package

Prerequisites

JDK 17 or later is installed.

Download the installation package.

wget https://schedulerx3.oss-cn-hangzhou.aliyuncs.com/xxljob/schedulerx3-worker-2.4.2-bin.tar.gz

Decompress and configure the package.

# Decompress
tar -zxvf schedulerx3-worker-2.4.2-jdk17-bin.tar.gz
cd schedulerx3-worker-2.4.2-jdk17-bin

The directory structure after decompression is as follows:

schedulerx3-worker-2.4.2-jdk17-bin/
├── bin/              # Startup script folder
├── conf/             # Configuration file folder
│   ├── application.yml      # Application configuration
│   └── logback-spring.xml   # Log configuration
├── lib/              # Dependency JAR package folder
└── logs/             # Log folder (automatically created at runtime)
    ├── stdout.log    # Standard output log
    ├── stderr.log    # Standard error log
    ├── worker.log    # Application log
    ├── error.log     # Error log
    ├── gc.log        # GC log
    └── archive/      # Archived log folder

Edit the conf/application.yml configuration file and configure the following parameters based on your XXL-JOB instance:

xxl:
  job:
    admin-addresses: {Service registration address}
    access-token: {Application AccessToken}
    executor:
      appname: {Application AppName}

Start the service.

Linux/Mac

# Start in the background
./bin/start.sh

# Start in the foreground (for testing)
./bin/start.sh -f

# Stop
./bin/stop.sh

# Restart
./bin/restart.sh

# Check status
./bin/status.sh

# View logs
tail -f logs/worker.log

Windows

REM Start in the background
.\bin\start.cmd

REM Start in the foreground (for testing)
.\bin\start.cmd -f

REM Stop
.\bin\stop.cmd

REM Restart
.\bin\restart.cmd

REM Check status
.\bin\status.cmd

REM View logs
type logs\worker.log

(Optional) Configure logs.

The main log files are located in the logs/ folder. By default, job execution logs are stored in ${user.home}/applogs/xxl-job/jobhandler.

Log file	Description	Rolling policy
`stdout.log`	Standard output log (startup log)	Script redirection
`stderr.log`	Standard error log (exception stack)	Script redirection
`worker.log`	Application log (INFO and higher)	100 MB per file, retained for 30 days
`error.log`	Error log (ERROR level)	50 MB per file, retained for 60 days
`gc.log`	GC log	JVM parameter settings
`heap_dump.hprof`	Heap dump file (generated on out-of-memory (OOM))	-
`archive/`	Archived log folder (automatic .gz compression)	-

Edit conf/logback-spring.xml to adjust the log output.

<!-- Modify the root log level -->
<root level="INFO">
    <appender-ref ref="STDOUT" />
    <appender-ref ref="FILE" />
</root>
<!-- Modify the log level for a specific package -->
<logger name="com.aliyun.schedulerx" level="DEBUG" />
<logger name="com.xxl.job" level="DEBUG" />

(Optional) Configure JVM parameters. Adjust the JVM memory size based on the actual payload.

# Linux/Mac - Specify temporarily
JAVA_OPTS="-Xms2g -Xmx4g" ./bin/start.sh

# Linux/Mac - Modify permanently
vim bin/start.sh  # Edit the JAVA_OPTS variable

# Windows - Specify temporarily
set JAVA_OPTS=-Xms2g -Xmx4g
.\bin\start.cmd

# Windows - Modify permanently
notepad bin\start.cmd  # Edit the JAVA_OPTS variable

Start using Docker

Method 1: Deploy using a public image

The public image provides a runtime for common scripts and is pre-installed with Python 3, Node.js, and Go environments. You can pull and run the image directly from the image repository without building it.

# Pull the image
docker pull scx-registry.cn-hangzhou.cr.aliyuncs.com/schedulerx3/schedulerx3-worker:2.4.2-jdk17

# Run with custom configuration
docker run -d \
  --name schedulerx3-worker \
  -p 9999:9999 \
  // Configure JVM parameters as needed
  -e JAVA_OPTS="-Xms1g -Xmx2g" \
  -e SCHEDULERX3_ADMIN_ADDRESSES="{Service registration address}" \
  -e SCHEDULERX3_EXECUTOR_APPNAME="{Application AppName}" \
  -e SCHEDULERX3_ACCESS_TOKEN="{Application AccessToken}" \
  -v $(pwd)/logs:/opt/schedulerx3-worker/logs \
  --restart unless-stopped \
  scx-registry.cn-hangzhou.cr.aliyuncs.com/schedulerx3/schedulerx3-worker:2.4.2-jdk17

Method 2: Build an image based on the tar package

If your business requires external component dependencies or a custom base image, you can build an image based on the downloaded tar package and publish it to your image repository.

# Download the installation package
wget https://schedulerx3.oss-cn-hangzhou.aliyuncs.com/xxljob/schedulerx3-worker-2.4.2-bin.tar.gz
# Build the Docker image
docker build -t schedulerx3-worker:2.4.2-jdk17 -f Dockerfile .

The following is the corresponding Dockerfile:

############################################
### Install the required components in this Dockerfile based on your business needs.
############################################

# Configure the base image.
FROM hub.docker.xxx.com/library/openjdk:17.0.1-jdk-bullseye

LABEL maintainer="SchedulerX Team"
LABEL description="SchedulerX3 Worker - XXL-Job Executor"
LABEL version="2.4.2"

# Configure Alibaba Cloud traffic mirror sources.
RUN sed -i 's/deb.debian.org/mirrors.aliyun.com/g' /etc/apt/sources.list && \
    sed -i 's|security.debian.org/debian-security|mirrors.aliyun.com/debian-security|g' /etc/apt/sources.list

# Install basic tools, Python 3, Node.js, and Go.
RUN apt-get update && \
    apt-get install -y python3 python3-distutils curl wget ca-certificates nodejs npm golang-go && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# Use the official script to install pip.
RUN curl https://bootstrap.pypa.io/get-pip.py -o /tmp/get-pip.py && \
    python3 /tmp/get-pip.py && \
    rm -f /tmp/get-pip.py && \
    ln -sf /usr/bin/python3 /usr/bin/python

# Set Go environment variables.
ENV GOPATH=/root/go
ENV PATH=$GOPATH/bin:$PATH
ENV GO111MODULE=on

# Copy the tar package to the image.
# The tar package is in the schedulerx3-worker-dist/target/ folder.
COPY schedulerx3-worker-*-bin.tar.gz /tmp/schedulerx3-worker.tar.gz

# Decompress the tar package to the specified folder (remove the top-level directory).
RUN mkdir -p /opt/schedulerx3-worker && \
    tar -xzf /tmp/schedulerx3-worker.tar.gz --strip-components=1 -C /opt/schedulerx3-worker && \
    chmod +x /opt/schedulerx3-worker/bin/*.sh && \
    mkdir -p /opt/schedulerx3-worker/logs && \
    rm -f /tmp/schedulerx3-worker.tar.gz

# Set the working directory.
WORKDIR /opt/schedulerx3-worker

# Expose the port.
EXPOSE 9999

# Startup command (uses the foreground mode of start.sh).
CMD ["bin/start.sh", "-f"]

Start with Kubernetes

Create a schedulerx3-worker.yaml file and use it to create a deployment.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: schedulerx3-worker
  labels:
    app: schedulerx3-worker
spec:
  replicas: 1
  selector:
    matchLabels:
      app: schedulerx3-worker
  template:
    metadata:
      labels:
        app: schedulerx3-worker
    spec:
      containers:
        - name: schedulerx3-worker
          image: scx-registry.cn-hangzhou.cr.aliyuncs.com/schedulerx3/schedulerx3-worker:2.4.2-jdk17
          imagePullPolicy: Always
          ports:
            - containerPort: 9999
          env:
            - name: "SCHEDULERX3_ADMIN_ADDRESSES"
              value: "{Service registration address}"
            - name: "SCHEDULERX3_EXECUTOR_APPNAME"
              value: "{Application AppName}"
            - name: "SCHEDULERX3_ACCESS_TOKEN"
              value: "{Application AccessToken}"
          livenessProbe:
            tcpSocket:
              port: 9999
            timeoutSeconds: 30
            initialDelaySeconds: 30

Deploy to Kubernetes.

# Deploy
kubectl apply -f schedulerx3-worker.yaml

Create a job

Script job

In the navigation pane on the left of the product page, choose Job > Job Management.
Click Create Job. The following example shows how to configure a Shell script. Configure other parameters as needed or use the default values.
Note
If the agent is deployed on a Unix or Linux system, set File Format to Unix.
- Set Associated Application to the application that you created in the Create a general-purpose application section.
- Set Job Type to Shell.
On the Job Management page, find the job that you created and click Run Once in the Actions column to test it.
In the navigation pane on the left, choose Job Instance to view the job execution records. Click Log to view the execution details of the script.