Multi-LLM Harness: Making Autonomous Agents Executor-Agnostic

Thursday, April 10. Heavy sprint week. Night Shift is grinding through tasks across five projects — content drafts, code generation, QA reviews. Around 2am JST, Claude Max hits quota. Every queued task fails. Every agent exits. The pipeline goes cold.

I wake up Friday to zero completed tasks and a $50 extra-usage charge from buying my way out of the jam.

This was not an anomaly. It was an inevitability. Night Shift runs 24/7. It burns through Claude tokens at a pace that makes quota-based subscriptions a single point of failure. I had been ignoring the risk because Claude was the only executor worth using.

That changed.

---

The Discovery

Gemini 3.1 Pro Preview dropped. Benchmarks put it at 80.6% on SWE-bench — matching GPT-5.x (80%). Kimi K2.5 leads LiveCodeBench at 85%. The gap between Claude and the field has closed to the point where "Claude or nothing" is a choice, not a constraint.

But benchmarks are irrelevant if you cannot enforce quality. Night Shift does not just run models. It runs models inside a harness — the Bridle Protocol. Pre-flight planning. Tool call limits. Doom-loop detection. Pre-completion gates. If an executor cannot be harnessed, it cannot be trusted.

So the real question was: can Gemini and Codex run inside the same harness?

I started digging. Gemini CLI has an equivalent hook system: AfterTool, AfterAgent. I wrote a quick test — headless gemini -p with hooks attached. The hooks fired. Session context injection, edit tracking, pre-completion gates — roughly 90% of what Claude's hooks provide. Codex has --full-auto mode with its own constraint model.

The enforcement gap is about 10%. Meaningful, but manageable.

---

What We Built

The architecture is intentionally minimal. Three components.

1. probe_executors() — Availability probing for CLI subscriptions. Before each Night Shift cycle, the harness checks which executors are alive:

probe_executors() {
  for executor in claude gemini codex; do
    if command -v "$executor" &>/dev/null && \
       timeout 10 "$executor" --version &>/dev/null; then
      AVAILABLE+=("$executor")
    fi
  done
}

This is not API health-checking. Every published multi-LLM paper assumes API keys with rate limits and billing dashboards. We are routing across CLI subscriptions — Claude Max ($200/mo), Google AI Pro ($20/mo), OpenAI Business ($25/mo). Nobody has documented this pattern because nobody else is running autonomous agents on subscription CLIs.

2. Executor adapter pattern — Each CLI has different invocation syntax. The adapter normalizes them:

Executor	Invocation	Constraint Model
Claude	claude -p "prompt" --allowedTools ...	Hooks + allowedTools
Gemini	gemini --prompt "prompt" --yolo	AfterTool/AfterAgent hooks
Codex	codex --full-auto -p "prompt"	Built-in sandbox

One function, try_executors(), walks the available list and falls back automatically. If Claude is quota-limited, Gemini picks up the task. If Gemini is down, Codex gets it. If nothing responds, the task gets re-queued instead of failing silently.

3. Project-level executor preferences — Not every model is interchangeable for every job. The routing table assigns defaults:

• MyWritingTwin → Claude (brand voice, content quality)
• FluxDiagram → Gemini or Codex (code generation, less voice-sensitive)
• Research tasks → Gemini (1M context window, strong at document synthesis)
• Code-heavy tasks → Codex or Claude (SWE-bench caliber)

Preferences are soft. If the preferred executor is unavailable, the harness falls back rather than blocks.

---

What Nobody Published Before

I have read every multi-LLM orchestration paper and blog post I could find. LangChain's routing. CrewAI's delegation. Anthropic's own multi-model patterns. All of them assume API key access with per-token billing.

None of them address:

1. CLI-subscription routing — probing whether a CLI tool is available and responsive, not whether an API endpoint returns 200
2. Harness enforcement portability — verifying that hooks, guardrails, and quality gates work across vendor CLIs in headless mode
3. Subscription cost optimization — routing based on "which $20-200/mo plan has headroom" rather than "which API key has budget"

This is a narrow architectural niche. It exists because subscription-based AI CLIs are new (Codex launched weeks ago), and autonomous agent orchestration on those CLIs is newer still.

---

The Garry Tan Question

Garry Tan asked it directly: "Are you overcomplicating things?"

Fair question. Here is the honest answer.

The entire change is one function (try_executors()), one routing table, and three adapter wrappers. Total implementation: maybe two hours. The harness already existed. The Bridle Protocol already existed. Portability is additive, not invasive.

The alternative is paying $50 in extra-usage every time a heavy sprint burns through Claude quota. That happened once. Over a year of weekly sprints, it would happen a dozen times. At $600/year in overage alone — plus the cost of waking up to zero completed tasks — the math is obvious.

This is not complexity for its own sake. It is removing a single point of failure from a system that runs unsupervised.

The lesson I keep learning: the right time to add resilience is right after the first real failure, not after the third. April 10 was the first. I would rather spend two hours building fallback routing than lose another night of compute.

---

What's Next

The harness works. Gemini runs Night Shift tasks with ~90% enforcement parity. Codex handles code-heavy work in its sandbox. Claude remains the primary executor for anything requiring judgment, voice, or complex orchestration.

The open question is whether project-level preferences should be static or dynamic. Should the harness learn which executor produces better results per project over time? That is a future problem. Right now, the system just needs to not stop when one vendor hits a wall.

That is enough.