Schedule and manage AI script jobs with SchedulerX

Background

• LangChain is a popular open-source framework for developing applications powered by large language models (LLMs). It lets you quickly build AI applications with Python or JavaScript.

• Dify is an open-source, graphical development platform for LLM applications. It lets you quickly build AI agents and workflows with a visual, drag-and-drop canvas.

Hosting AI jobs on a scheduling system lets you manage script versions, configure time-based scheduling, improve resource utilization, implement throttling, and enhance observability.

Script management and scheduling

AI jobs are used in many business scenarios and often require time-based scheduling. Common use cases include:

• Risk monitoring: Scan risk data every minute, use an LLM to analyze potential risk events, and send timely alerts.

• Data analysis: Pull financial data daily, use an LLM for analysis, and provide decision-making advice for investors.

• Content generation: Automatically summarize daily work and generate reports.

LangChain jobs are typically implemented as Python scripts. You can use the script job feature in SchedulerX to host your scripts and configure time-based scheduling.

SchedulerX supports script version history, which lets you easily compare and roll back to previous versions. For more information, see Script version history.

The following example shows a version comparison. The current version uses a chained call with PromptTemplate and LLMChain:

from langchain.prompts import PromptTemplate
from langchain.llms import Tongyi
from langchain.chains import LLMChain
llm = Tongyi(temperature=0.9)
prompt = PromptTemplate(
    input_variables=["product"],
    template="What is a good name for a company that makes {product}"
)
chain = LLMChain(llm=llm, prompt=prompt)
results = chain.invoke("colorful socks")
print(results)

The historical version v0.0.1 uses a direct call with llm.predict:

from langchain.llms import Tongyi
def main():
    llm = Tongyi(temperature=0.9)
    request = "What would be a good c..."
    response = llm.predict(request)
    print(response)
if __name__ == "__main__":
    main()

Prompt management

Prompts play a critical role in AI jobs. To achieve optimal results, you often need to adjust them. Hardcoding prompts directly into scripts increases maintenance costs and reduces flexibility. You can use the job parameters feature in SchedulerX to manage prompts effectively. In your LangChain script, use the system parameter(#{schedulerx.jobParameters}) provided by SchedulerX to dynamically retrieve job parameters. This allows for flexible configuration by replacing hardcoded prompt or PromptTemplate values.

SchedulerX provides two ways to manage prompts: obtaining prompts through time-based scheduling and passing prompts dynamically through API calls.

Resource utilization

SchedulerX supports two modes for script execution:

• script job: Deploy theschedulerx-agent on an ECS instance. Each execution forks a child process to run the script. This mode is suitable for low-resource jobs that are scheduled frequently.

• Kubernetes job: Deploy theschedulerx-agent on a Kubernetes cluster. Each execution launches a new pod to run the script. This mode is suitable for resource-intensive jobs that run infrequently.

Combining them can significantly improve resource utilization.

The following table compares script execution on ECS instances and Kubernetes clusters.

Throttling

Use case: You have a set of offline jobs that need to run after midnight each day to process the previous day's data. The core jobs must be completed before business starts at 9:00 AM. Developers might schedule all these jobs to run at the same time, such as 00:30 every day.

When many jobs are scheduled simultaneously, they can overwhelm your ECS instances. Although running scripts on Kubernetes can alleviate some of this pressure, the traffic burst can still overload downstream services like databases. The best solution for these traffic bursts is throttling. Throttling is an effective strategy to manage uneven scheduling from time-based jobs, especially during traffic spikes, and improve resource utilization.

As shown in the preceding figure, SchedulerX supports application-level throttling:

1. Each application has a priority queue that organizes jobs by priority to ensure that core jobs are executed first. A job's priority applies only within its application and does not affect other applications.

2. Each application also has a concurrency queue that controls the number of concurrent job executions. The concurrency limits of different applications are isolated from each other.

3. When a job in the concurrency queue completes and frees up a slot, the next job is taken from the head of the priority queue and moved to the concurrency queue for execution.

Automatic retry

Large language model (LLM) calls can be unstable, causing temporary issues like token limits or backend service exceptions. These issues are often transient and can be resolved with a retry.

SchedulerX provides an automatic retry feature for failed jobs, which can be dynamically configured in the console. In practice, using this feature for LangChain scripts significantly reduces failures caused by backend LLM throttling or service unavailability, improving the job success rate by at least one 'nine' (e.g., from 99% to 99.9%).

To enable automatic retry, expand the Advanced Configuration section on the job configuration page, and set the Retry Count on Task Failure and Retry Interval on Task Failure (in seconds).

Job orchestration

SchedulerX provides visual job orchestration. If your LangChain scripts have dependencies on each other, you can easily create a workflow. You can even orchestrate different job types within the same workflow.

As shown in the figure:

1. First, a shell script pulls data from the big data platform.

2. Next, a Java job cleanses the merchant and user data.

3. Then, a LangChain job performs LLM-based analysis on the cleansed data.

4. Finally, a Python script generates a report.

Enterprise-grade observability

SchedulerX is natively integrated with various cloud services to provide enterprise-grade observability.

Contact us

If you have other requirements for AI job scheduling, join the DingTalk group (ID:23103656).