AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces

Video

Harness Optimization on GAIA2

Step through AutoSaddler’s optimization process on GAIA2. Starting from the default agent harness (47.7%), AutoSaddler discovers prompt rules, new tools, and agent loop fixes that push dev-set accuracy to 72.3% over two epochs. Each dot is one iteration. Click any dot or use the arrow keys to inspect its patch details and code changes.

Method Overview

AutoSaddler formulates automatic harness optimization as an offline learning problem with a mini-batch training paradigm. Each iteration consists of three sessions — Diagnosis-Patch, Reflection, and Evolution — that together produce durable harness updates rather than trajectory-specific fixes.

Three Key Ingredients

AutoSaddler’s design is grounded in three requirements for effective harness optimization:

1. In-depth Diagnosis — Deep debugging of execution traces, not shallow reflection. The diagnosis agent makes an average of 6.2 additional tool calls and 5.8 file accesses per iteration to understand root causes, resulting in 13 accepted patches by Iter 25 (vs. 5 without).

2. Structured Intervention — Targeted modifications guided by a patch taxonomy across prompts, tools, and middleware. Without structure, 91.5% of patches are trivial text edits; with AutoSaddler, high-impact types like New Tools (83% acceptance) and Loop Changes (71%) are actively explored.

3. Generalization-Aware Selection — Retain broadly useful updates, not single-trajectory fixes. This is the largest ablation factor: removing it drops performance from 62.0% to 50.6%. The key mechanism is regression prevention — AutoSaddler’s regression rate trends downward (−0.24 pp/iter) vs. upward (+0.16 pp/iter) without it.

Patch Taxonomy

AutoSaddler categorizes harness modifications into a structured taxonomy:

C = Capability (modifies code/logic), S = Steering (modifies text only). AutoSaddler schedules Capability patches first, then Steering — analogous to learning-rate scheduling.

Key Results

AutoSaddler achieves state-of-the-art automatic harness optimization on three challenging agent benchmarks, outperforming both manually tuned and automated baselines.

GAIA2 Test Results

SWE-Bench Pro Test Results

Terminal-Bench 2.0 Test Results

Ablation Studies

RQ1: Does In-depth Diagnosis Matter?

Removing in-depth diagnosis drops Pass@1 from 62.0% → 57.8% (−4.2 pp). Full diagnosis uses on average 6.2 additional tool calls and 5.8 additional file accesses per step, allowing the agent to inspect execution traces, read source code, and verify hypotheses before generating patches.

RQ2: Does Structured Intervention Matter?

Removing structured intervention drops Pass@1 from 62.0% → 56.9% (−5.1 pp). Without it, the patch distribution collapses — 91.5% of patches are Steering-only (trivial text edits). With AutoSaddler’s taxonomy, Capability patches like New Tool (83% acceptance), Loop Change (71%), and Infrastructure Change (67%) are actively discovered.

RQ3: Does Generalization-Aware Selection Matter?

This is the largest ablation factor: removing generalization-aware selection drops Pass@1 from 62.0% → 50.6% (−11.4 pp). The fix rates are comparable between the two settings — the critical difference is regression prevention. Without validation-based selection, the regression rate trends upward (+0.16 pp/iter), while AutoSaddler’s regression rate trends downward (−0.24 pp/iter).

Conclusion

In this work, we introduced AutoSaddler, an automatic harness optimization framework that formulates harness improvement for LLM agents as an offline learning problem over execution traces. AutoSaddler combines in-depth failure diagnosis, structured patch generation across prompts, tools, and middleware, and generalization-aware update selection via validation and EvoDAG-based evolution. As a result, it produces durable harness updates rather than trajectory-specific fixes. On GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0, AutoSaddler consistently improves over the corresponding base harnesses by 9.0, 9.6, and 10.0 percentage points, respectively, and outperforms the strongest automated baseline on each benchmark by 7.4, 4.4, and 6.7 points. These results show that effective harness optimization requires deep debugging, targeted interventions, and explicit selection for generalization, positioning automatic harness optimization as a promising direction for building more performant and reliable long-horizon agent systems.

BibTeX

@misc{park2026autosaddlerautomaticharnessoptimization,
  title={AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces},
  author={Sungho Park and Wonjoong Kim and Rongyuan Tan and Jue Zhang and Wook-Shin Han and Pengfei Gao and Chanyoung Park and Yongqiang Yao and Rao Fu and Elsie Nallipogu and Qingwei Lin and Saravan Rajmohan and Dongmei Zhang},
  year={2026},
  eprint={2608.23041},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2608.23041},
}