Code Harness or Natural-Language Harnesses?

This article introduces Natural-Language Agent Harnesses (NLAH), replacing traditional code-based agent control with executable natural language strategies.

By Wangchen

"If you're not the Model, you're the Harness."

This is a viewpoint raised by Langchain, which emphasizes the importance of Harnesses in the process of building Agents. Friends who have worked in backend development or designed distributed systems are certainly very familiar with glue code and middleware. Before diving into the main text of this article, let's combine two familiar concepts to further deepen our understanding of Harnesses.

Glue Code: Generally refers to code that sticks two things together, lacking its own business meaning and acting purely as a connector. It doesn't sound high-end because it implies this layer of code has no design and is a byproduct of coincidence.
Middleware: A term in distributed systems that emphasizes a layer of service located between two ends, usually referring to infrastructure such as message queues, RPC frameworks, and API gateways, which have clear engineering definitions and product forms.
Harness: In the Agent era, its definition is both precise and abstract. The precise part is that it specifically refers to the runtime control logic surrounding an LLM Agent—when to call the model, how many times to call it, how to validate the output, how to retry upon failure, how to orchestrate multiple steps, etc. It is the mastery of the Agent's runtime strategy. The abstract part is that it is difficult to give it a clear engineering definition and product form like middleware.

Although all three belong to the engineering domain, the problems they aim to solve and their solution paradigms are completely different. For Harness, the mastery layer is usually not represented as a clean, researchable object. Mastery logic is highly coupled within the controller code, such as prompts, tool adapters, parsing rules, validation scripts, artifact paths, retry logic, and context strategies. A seemingly minor change in the mastery layer can simultaneously alter calling boundaries, tool intermediaries, state carriers, validation gates, and stop semantics. This makes the mastery layer difficult to inspect, port, compare, and ablate, even though the mastery pattern itself is often the truly reusable part of the system. This sounds somewhat contradictory—difficult to master yet reusable. However, I believe friends engaged in Harness engineering design can feel the uniqueness of Harness engineering.

Recently, I read a technical paper titled "Natural-Language Agent Harnesses," from which I benefited greatly. The paper revolves around the research topic: "Can this reusable design pattern of the agent mastery layer be represented as an executable natural language object, transforming the incidental glue around the model into a scientifically represented object?" and provides its research methodology. Interested friends can read the original paper; this article attempts to provide a simple interpretation.

I. Three Ways to Control an Agent Run

Modern language model agents have become multi-step execution systems. They use tools, maintain state, recover from failures, validate intermediate results, and sometimes delegate work to other agents. These behaviors are organized by an external harness layer, which has a significant impact on measured performance. Similar concerns have also emerged in recent work on agent scaffolding, workflow generation, long-context execution, multi-agent orchestration, and tool-use agents.

Figure 1: Three ways to control an agent run.

Its spectrum ranges from "restrictive harness" to "no external harness" (i.e., self-harnessing); from left to right, external control weakens, and model autonomy strengthens. Among them:

The code harness layer exerts hard external control over the model through program logic.
Natural Language Agent Harnesses (NLAHs) + Intelligent Harness Runtime (IHR) move the harness strategy into readable natural language, while a shared runtime executes this strategy through sub-agent calls.
Self-harnessing is a possible future design: a controller model directly harnesses other models without the need for any external harness layer.

The paper mainly focuses on the second point, externalizing the harness strategy from code into executable natural language objects:

NLAH (Natural-Language Agent Harnesses): Editable documents that describe run-level harness strategies in natural language, including stages, roles, state rules, validation rules, recovery rules, and stop conditions.
IHR (Intelligent Harness Runtime): A shared runtime that interprets and executes these NLAH documents into agent calls, handoffs, state updates, validation gates, and artifact contracts.

Within this setup, natural language carries the strategy, while code and the runtime carry the precise mechanisms, such as tool execution, parsing, sandboxing, and logging.

The core contributions of the paper are:

Introducing NLAH as an explicit natural language representation of agent harness patterns, which is distinct from both runtime strategies and deterministic code hooks.
Introducing IHR, a shared, in-loop runtime that translates NLAH into auditable agent calls, handoffs, state updates, validation gates, and artifact contracts.
Exploring the boundaries of natural language and code in agent harness systems, taking a first step toward expanding the scope of natural language from local instructions to high-level harness strategies.
Providing controlled evidence across three types of benchmarks that NLAH can shape agent behavior with comparable task results, expose clean static harness strategies, and support module-level analysis.

II. The Four-Tier Architecture of NLAH and IHR

Figure 2: The NLAH+IHR framework.

Base Agent: A minimal agent loop consisting of LLM + unique tool = terminal. Through the terminal, it can read/write files, run commands, and launch sub-agents. It contains no task logic on its own and is a "tabula rasa" (blank slate).
Runtime Strategy: A fixed set of natural language instructions that turns the base agent into the Intelligent Harness Runtime (IHR). It tells it: "Your role is to read the NLAH document, schedule sub-agents according to the document, manage state files, perform call-graph recovery, and execute contract-first completion." It is generic to all tasks and does not change with specific tasks.
Natural Language Harness (NLAH) Document: This is the core innovation of the paper. A readable, editable natural language document that can be used for A/B testing, describing task stage divisions, roles and division of labor, state transitions, acceptance criteria, failure recovery, and stop conditions. It is replaceable; changing the .md file changes the task.
Scripts and Adapters: Hard logic that does not require LLM judgment, implementing JSON schema validation, diff line count calculation, format conversion, and external API call encapsulation. These are triggered by rules in the NLAH document and scheduled for execution by the IHR.

This can be considered a new paradigm for building Harnesses. To use an analogy: the base agent is the horse, the runtime strategy is the rider, the natural language harness document is the road map for today's trip (which can be changed), and the scripts and adapters are traffic lights and checkpoints (deterministic rules) along the way.

III. How to Evaluate the Performance of the New Paradigm

Once the core architecture is established, we need to design the evaluation methodology.

RQ1 (Harness Implementation)

Can NLAH shape observable agent behavior while maintaining task performance?

RQ1 compares three implementations of the same harness concept, ordered by how directly the harness layer can control execution:

Code Harness: The original code implementation of the studied agent harness family—controller code, workflow scripts, framework defaults, and tool adapters. It provides the strongest and most deterministic control, but its strategy is interwoven with implementation details.
Prompted NLAH: Providing the same NLAH content as regular prompt/instruction text to a Codex CLI agent, without the IHR's shared runtime charter and execution semantics. It tests how much control is gained "when natural language is merely a passive instruction carrier."
IHR-executed NLAH: An NLAH interpreted and executed by the IHR, with explicit runtime semantics for sub-agent lifecycles, artifact and state handling, contract gates, and stopping. It sacrifices the hard determinism of the code harness layer but provides an execution substrate for natural language strategies that can materialize roles, handoffs, state, and validation boundaries.

Across three benchmarks—SWE-bench Verified, Terminal-Bench 2.0, and OSWorld—the IHR-executed NLAH achieved performance comparable to the code harness layer, while significantly compressing static readable strategies (for example, Live-SWE dropped from 60,100 tokens of code to a 2.9k-token NLAH; MHTBA dropped from 10,500 to 0.8k). The tradeoff is a higher token/invocation overhead for the current prototype runtime.

RQ2 (Harness Mechanism Implementation)

After RQ1 demonstrated that natural language can effectively carry harness strategies, the paper asks a further, more detailed question: How faithful is the model actually to each mechanism written in the document when executing NLAH?

To do this, they decomposed and labeled the instructions in an NLAH into 8 categories of observable harness mechanisms, such as deliverable contracts, tool success gating, phase handoffs, and stop conditions. They then ran the IHR on three benchmark tasks to compare "what the document required" against "what the model actually did" on a step-by-step basis, calculating a adherence rate for each category of mechanism.

The findings were "mixed, with significant differences": contract-based, instantly verifiable mechanisms had high adherence because these instructions have clear boundaries and the model can self-check at each step. On the other hand, mechanisms relying on cross-phase, cross-sub-agent coordination showed layout lower adherence. This finding of "uneven execution quality" provided the motivation for the module ablation in RQ3.

RQ3: Module Ablation

Because NLAH is intentionally written with a structure that separates phases from mechanisms and maintains clear module boundaries, each functional module corresponds to a piece of text in the document that can be independently added or removed. This makes it possible, for the first time, to conduct classic ablation experiments at the strategy level rather than the code level.

The paper uses a leave-one-out approach, turning off only one module in the NLAH at a time (such as file-backed state persistence, self-evolution, multi-candidate search, context compression, markdown memory, etc.) while keeping the rest unchanged, and then measuring the change in success rate relative to the complete NLAH. The entire process requires editing only the document without modifying any underlying code, cleanly attributing each module's contribution individually.

Ablation revealed highly polarized module values, showing that some seemingly useful designs are actually negative assets: file-backed state persistence was the most stable positive contributor (Δ approx. +2.6 / +13.9), showing that allowing the agent to externalize intermediate states to files is far more reliable than relying on context memory. The self-evolution module brought the highest absolute performance (Live-SWE 78.8 / 52.8), but at the cost of significantly increased token consumption. Conversely, multi-candidate search actually harmed performance (distracting the model and being expensive), context compression was harmful on both tasks (saving tokens but losing critical information), and markdown memory performed inconsistently, with gains depending on the task. The overall conclusion is: with harness modules, "more is not always better." Effective engineering practices should prioritize retaining externalized states, use self-evolution cautiously, and actively eliminate negative-utility modules such as multi-candidate search and context compression. This also, for the first time, turned the question of "which harness designs to keep" into one that can be answered with experimental data.

IV. Limitations and Risks

The paper also mentions the limitations and risks of this paradigm. The limitation is the impreciseness of natural language. Since NLAHs are editable natural language strategies, semantically important constraints might be under-defined, interpreted differently by different models, or diluted when rewritten. Therefore, the paper retains precise mechanisms in code and treats executed behaviors as something that must be tested through actual runs rather than inferred solely from text.

The risk is that while externalizing harness modules can reduce development costs, improve comparability, and encourage the reuse of robust workflows, portable harness logic and scripts might also lower the barrier for distributing hazardous workflows. Because the harness layer mediates tool use, artifact handling, and delegation, they could introduce new attack surfaces for prompt injection, malicious tool grafting, or supply chain contamination. Deployment should be accompanied by provenance tracking, auditing, access control, and sandbox isolation.

Community

Code Harness or Natural-Language Harnesses?

I. Three Ways to Control an Agent Run

II. The Four-Tier Architecture of NLAH and IHR

III. How to Evaluate the Performance of the New Paradigm

RQ1 (Harness Implementation)

RQ2 (Harness Mechanism Implementation)

RQ3: Module Ablation

IV. Limitations and Risks

Read previous post:

Read next post:

Alibaba Cloud Native Community

You may also like

Comments

Alibaba Cloud Native Community

Related Products

Alibaba Cloud Model Studio

Qwen

Alibaba Cloud for Generative AI

Cloud-Native Applications Management Solution