Microservices Engine:Connect an agent (for script or HTTP jobs)

Prerequisites

• (Optional) A namespace is created. For more information, see Create a namespace.

• A resource is created. For more information, see Create a resource.

Use cases

The primary use cases for connecting an agent to SchedulerX include:

• Using a Shell script to periodically clean logs on all worker nodes or start a service.

• Using a Python script to periodically clean historical data from a database.

• Using an HTTP API to periodically scan orders.

Environment requirements

Requires Java Runtime Environment (JRE) 1.8 or later.

Manually deploy the agent

1. Download and decompress the agent package.

   If your worker node has a public IP address, you can run the following command to download the package:

   Note

   Replace X.X.X with the version number you need, for example, 1.12.5.

   wget https://schedulerx2.oss-cn-hangzhou.aliyuncs.com/agent/schedulerxAgent-X.X.X.tar

   Version	Download URL	Description
   agent 1.13.5	https://schedulerx2.oss-cn-hangzhou.aliyuncs.com/agent/schedulerxAgent-1.13.5.tar.gz	New features Enables graceful shutdown for second-level jobs. Enables passing data between upstream and downstream jobs in a workflow for script jobs. Enables collecting system performance metrics with cgroup v2.
   agent 1.13.2	https://schedulerx2.oss-cn-hangzhou.aliyuncs.com/agent/schedulerxAgent-1.13.2.tar.gz	New features Enables CPU metric collection with CGroup v2. Enables customizing the job execution thread pool size by using the share.pool.size parameter key.
   agent 1.12.5	https://schedulerx2.oss-cn-hangzhou.aliyuncs.com/agent/schedulerxAgent-1.12.5.tar	New features: Enables retrieving job system variables by using placeholders. Enables customizing the thread pool size for HTTP jobs. Allows script jobs to return the last line of standard output as the execution result.
   agent 1.11.5	https://schedulerx2.oss-cn-hangzhou.aliyuncs.com/agent/schedulerxAgent-1.11.5.tar	Bug fixes: Fixes security vulnerabilities in Netty and Logback. Fixes the "too old resource version" issue for Kubernetes jobs.
   agent 1.10.13	https://schedulerx2.oss-cn-hangzhou.aliyuncs.com/agent/schedulerxAgent-1.10.13.tar	New features: Enables deploying multiple agents on a single worker node. Enables responses larger than 1,000 bytes for HTTP jobs. Optimizes logging to reduce historical log storage.
   agent 1.10.5	https://schedulerx2.oss-cn-hangzhou.aliyuncs.com/agent/schedulerxAgent-1.10.5.tar	New features: Enables graceful shutdown. Enables application group isolation in the logging service.
   agent 1.9.8	https://schedulerx2.oss-cn-hangzhou.aliyuncs.com/agent/schedulerxAgent-1.9.8.tar	New features: Enables agent-based HTTP jobs. Allows a script job to return the last line of the log upon failure. Enables Kubernetes jobs.
   agent 1.7.10	https://schedulerx2.oss-cn-hangzhou.aliyuncs.com/agent/schedulerxAgent-1.7.10.tar.gz	Bug fixes: Fixes an issue where second-level jobs run slowly. Fixes an issue where the logging service could cause an Out of Memory (OOM) error.
   agent 1.4.2	https://schedulerx2.oss-cn-hangzhou.aliyuncs.com/agent/schedulerxAgent-1.4.2.tar.gz	New feature: Introduces the logging service.

   Important

   If the wget command times out while downloading the agent, enable a public IP address for your instance or see Upload or download files (Linux).

2. Go to the schedulerxAgent/conf directory and configure the agent.properties file.

   Configure the following parameters:

   endpoint=addr-hz-internal.edas.aliyun.com
   # For public network access, use domainName instead of endpoint.
   # domainName={domainName}
   namespace=fd2965c4-xxxx-xxxx-af52-bb62aa4f19f2
   groupId=hxm.test
   appKey=12******789

   • endpoint: The service endpoint. For more information, see Endpoints.

   • namespace: The namespace ID. You can find this on the Namespaces or page.

   • groupId and appKey: The application ID and application key. Find these on the Application Management page.

3. Go to the schedulerxAgent/bin directory and run the start.sh command to start the agent.

   Note

   To set the memory for starting the Agent, you can run start-200m.sh, start-500m.sh, or start-1g.sh.

   During startup, you can:

   • Stop the agent: Go to the schedulerxAgent/bin directory and run stop.sh.

   • View logs: The log path is ${user.home}/logs/schedulerx/worker.log.

     To get the value of user.home, run the ps aux | grep java command. If you start the process as the admin user, the log path is /home/admin/logs/schedulerx/worker.log. If you start it as the root user, the log path is /root/logs/schedulerx/worker.log.

   If the response contains Schedulerx Worker started, the agent has started successfully.

   Note

   The agent requires JDK 1.8 to 14. If you use a later version, you must remove the -XX:+UseConcMarkSweepGC option from the schedulerxAgent/bin/start.sh file in the agent installation directory.

   JAVA_OPTS="-Dproject.name=schedulerxAgent -XX:+UseConcMarkSweepGC -XX:+HeapDumpOxxx -XX:HeapDumpPath=${WORKER_LOG_DIR} -XX:+xxxlysis"
   if [ -n "${SCHEDULERX_STARTER_MODE}" ]; then
       JAVA_OPTS="${JAVA_OPTS} -Dschedulerx.stater.mode=${SCHEDULERX_STARTER_MODE}"
   fi

Deploy using a Docker image

1. Select a Docker image based on your network environment and CPU architecture.

   Note

   The following image addresses are accessible over the public internet.

   Network	x86_64	arm64
   China (Hangzhou) (Public Network)	schedulerx-registry.cn-hangzhou.cr.aliyuncs.com/schedulerx2/agent:latest-amd64	schedulerx-registry.cn-hangzhou.cr.aliyuncs.com/schedulerx2/agent:latest-arm64
   Singapore (Public Network)	schedulerx-registry.ap-southeast-1.cr.aliyuncs.com/schedulerx2/agent:latest-amd64	schedulerx-registry.ap-southeast-1.cr.aliyuncs.com/schedulerx2/agent:latest-arm64

2. Deploy the agent.

   Parameters:

   docker run  --env SCHEDULERX_ENDPOINT="addr-hz-internal.edas.aliyun.com" \
      --env SCHEDULERX_NAMESPACE="fd2965c4-xxxx-xxxx-af52-bb62aa4f19f2" \
      --env SCHEDULERX_GROUPID="hxm.test" \
      --env SCHEDULERX_APPKEY="12*****789" \
      -d {image id}

   • SCHEDULERX_ENDPOINT: The service endpoint.

   • SCHEDULERX_NAMESPACE specifies the namespace ID. You can obtain the ID from the Namespaces page in the console.

   • SCHEDULERX_GROUPID is the application ID.

   • SCHEDULERX_APPKEY is the application key. You can obtain the application key on the Application Management page in the console.

   After you start the deployment job, you can use the docker logs command to view the logs.

Deploy as a Kubernetes Deployment

If you already have a Kubernetes environment, you can use the schedulerx-agent.yaml file to deploy directly.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: schedulerx-agent
  labels:
    app: schedulerx-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: schedulerx-agent
  template:
    metadata:
      labels:
        app: schedulerx-agent
    spec:
      containers:
      - name: schedulerx-agent
        image: schedulerx-registry.cn-hangzhou.cr.aliyuncs.com/schedulerx2/agent:latest-amd64
        imagePullPolicy: Always
        resources:
          requests:
            cpu: 500m
        env:
          - name: "SCHEDULERX_ENDPOINT"
            value: "${SCHEDULERX_ENDPOINT}"
          - name: "SCHEDULERX_NAMESPACE"
            value: "${SCHEDULERX_NAMESPACE}"
          - name: "SCHEDULERX_GROUPID"
            value: "${SCHEDULERX_GROUPID}"
          - name: "SCHEDULERX_APPKEY"
            value: "${SCHEDULERX_APPKEY}"
          - name: "SCHEDULERX_STARTER_MODE"
            value: "pod"
        livenessProbe: 
          exec: 
            command: ["/bin/bash","/root/health.sh"]
          timeoutSeconds: 30
          initialDelaySeconds: 30