Instruction improvement with Harness2

harness.optimize uses Google Research Harness2's pinned Propose–Probe–Compose method inside Heimdall Audit. Google ADK still executes each isolated trial. AutoSaddler remains installed for immutable candidates, parent lineage, durable attempts and an explicit rollback search backend. Only one search backend owns an activation.

Scope and retained behavior

The default performs one sequential refinement step, with one worker at a time: baseline, general candidate, task-specific candidate, then a composed candidate. The existing model owner handles every proposal and trial; no provider server, benchmark installer or second scheduler is started. An external agent CLI runs only when the Audit Task selects a cloud session model (below).

Candidates may edit only the bound Agent identity or Runbook instruction body, using the existing whole-unit mutation validator. Article links, metadata, capability grants, model configuration and code remain outside this scope. All candidate Tool calls are intercepted by the existing ADK trial evaluator. The optimizer never reruns the real task to measure a candidate.

Harness2's preference for a final candidate is not acceptance. Publication still requires the existing repeated baseline/control comparison, training improvement, control pass floor and regression limits, Tool-count check and applicable Executive latency check. Missing evidence and execution errors cannot become success. The incumbent instructions remain when no candidate qualifies. A stage whose one correction retry still returns an edit the validator refuses keeps the current instructions, as upstream specifies; the report lists it as a resolved failure with the validator's text and the comparison sets the verdict. A session that fails to return output still makes the job incomplete.

The AutoSaddler artifact layer preserves completed attempt receipts across an interruption. Backend, installed-source manifest, model, case content, implementation and experience inputs are bound to the job. Foreground interruption cancels and joins inference and the synchronous search worker before releasing ownership. Session, trial and job budgets remain bounded across retries. An Audit cancelled before harness.optimize returns, by foreground input or a Harness stop, has published nothing: its occurrence stays pending and the next Audit resumes the same persisted job. A returned report is never resumed.

Only training evidence reaches proposals, composition or lesson extraction. Control cases and results stay outside that context. Retained lessons are bounded private optimizer evidence, not accepted Knowledge or Hindsight Memory. A case freezes its prior experience; later publications cannot change a resumed search. New lessons become eligible only after independently validated instruction publication and its committed Audit/Review evidence.

Cloud proposer sessions (Codex, Claude Code)

The model catalog also lists cloud session models offered by the installed, signed-in CLIs: codex:<model> for every model codex debug models lists for the ChatGPT login, codex:default (the model set in ~/.codex/config.toml), and claude:fable|opus|sonnet|haiku, aliases Claude Code resolves to the newest model of each family when it runs (fable is Anthropic's most capable family). Only the Audit Task can select one. Then every proposal, composition and reflection session runs through that CLI with the same system text, input and JSON schema the local model receives; the output still passes the whole-unit mutation validator, and the selected model is part of the optimizer contract, so changing it invalidates prepared cases.

Each session is one structured answer with no tools: codex exec with --ignore-user-config, a read-only sandbox, an empty working directory, shell, web, multi-agent, memory, plugin, hook and fast/ultrafast features disabled and service_tier="default"; or claude -p with --tools "", --safe-mode (no hooks, MCP servers or CLAUDE.md), no saved session, a replaced system prompt and a per-session --max-budget-usd estimate cap. Audit's reasoning maps none/low, medium, high to the same CLI levels and xhigh to max, the deepest level without delegation (Codex ultra adds sub-agent delegation, which a tool-less session cannot use). A cloud session holds no local model lease and may take up to ten minutes; the job budget still bounds the whole optimization. A CLI that answers without a complete structured result is a failed session, like invalid output.

The CLIs return no client-side function calls, so they cannot run Obsidience's native Tool loop. Audit's own short loop (harness.optimize, then completion) keeps the automatic local model. Every trial still measures the affected Agent with its own model and reasoning, because that model is what will run the published instructions. Scoring trials on the proposer's model would measure a different system and could publish instructions tuned for a model the Agent never uses.

Upstream's own channels are not used. At the pinned revision the claude_cli proposer forces Vertex AI (CLAUDE_CODE_USE_VERTEX=1, Anthropic login variables dropped) and runs --dangerously-skip-permissions over a writable copy of the context, harvesting an edited file tree (proposer.py); Codex appears only as a rollout substrate run with --dangerously-bypass-approvals-and-sandbox (codex.py), with no Codex proposer channel (spec.py). Reusing them would need a Google Cloud project, give a session shell access to the workstation and bypass the structured edit protocol. The adapter therefore keeps its proposer seam and uses the same vendor CLI with the safe flag set.

Codex reads global instructions from CODEX_HOME/AGENTS.md with no switch to skip them, and documents a separate CODEX_HOME for automation (AGENTS.md guide). Sessions therefore run with CODEX_HOME=obsidience/state/codex-home, so the owner's private ~/.codex/AGENTS.md (about 48K tokens) never reaches them: a one-line request measured 55.5K input tokens before and 9.7K after (2026-10-19). That home's auth.json is a symbolic link to ~/.codex/auth.json; with the file credential store (pinned by cli_auth_credentials_store="file") Codex opens, reloads before refreshing, and rewrites that one file in place, exactly as any other Codex process sharing the login does. Obsidience never reads or copies it. If the link is missing it is created; if it points elsewhere or has become a separate file, sessions fail closed and the file is left for the owner. A separate codex login into that home is the fully documented alternative.

Session input is training trajectories, objectives, conversation excerpts, Knowledge and Tool/Skill text from the frozen case: selecting a cloud model sends that Vault and conversation content to OpenAI or Anthropic. Usage counts against the ChatGPT or Claude plan limits; Claude Code's subscription terms assume ordinary individual use, so prefer local Audit for routine volume.

Dependency and rollback

Run from the repository root after installing ordinary Harness requirements:

python3 obsidience/scripts/install-optimization

--build-only produces the wheel without installing it. The installer verifies the exact upstream archive, applies the tracked patch, and includes a manifest of every Python source file. Runtime admission verifies the expected manifest and installed source bytes. The Apache-2.0 license is included in the wheel; the revision, source digest and patch are recorded in artifacts.lock.json.

The local 0.1.0+obsidience.1 patch admits Python 3.14 and disables implicit working-directory .env loading on import. The refinement implementation is unchanged. This is a locally validated embedded package; it does not claim upstream support for all CLI/benchmark paths on Python 3.14.

OBSIDIENCE_OPTIMIZER_BACKEND=autosaddler selects the retained original engine. Use the installation's normal idle Harness deployment procedure to change that setting. Changing backend invalidates prepared cases; prepare fresh cases under the selected backend. Failed Harness2 work never silently switches engines. Existing reports retain their original backend identity and remain historical evidence, rather than being relabeled as Harness2 results.

Upstream research informing this integration

Reviewed upstream revision 1cfe254e8d25b15f37048c3730134686abe60335 and live GitHub discussions on 2026-10-18. The fixes below were open and unmerged when reviewed. Their presence is not evidence that the pinned release includes them.

Finding	Source	Integration consequence
Selection excludes the baseline; one-step selection takes the single final	selection, scheduler	Independent comparison and not_improved remain authoritative.
Proposal cache omits model/channel configuration; resume can omit task/source and experience changes	PR 48, issue 65, PR 21, PR 36	Use the existing content-bound job and attempt records, not upstream provider/scheduler caches.
Interrupted audit fragments and unparseable trajectories can be treated as usable evidence	PR 69, PR 72	Require complete structured responses and actual validated trial records.
Concurrent experience streams can prune each other's inputs	PR 54	One owner, immutable case snapshots, no upstream mutable library writer.
Rollout spend and earlier partial-run costs are not fully accounted for	issue 4, issue 50, PR 74	Preserve model accounting and bounded durable attempts/deadlines; native recursion counts are not a spend cap.
Rejudging can retain an old grade after a run changes	issue 44, PR 45	Bind comparison and publication to immutable candidate observations and exact receipts.

The reviewed comments were contributor discussion, not demonstrated Google maintainer approval. In PR 48's author reply, the proposed cache key was revised to exclude timeout; older summaries still described the earlier proposal.

The Hacker News discussion asked for matched-model harness comparisons. An actual LocalLLaMA comment raised overfitting concerns; the same discussion considered context pressure for local models. These are useful evaluation questions, not measured defects or proof that one optimizer is better. Compare the same captured cases, model and budgets, keep controls private, and measure the actual supplied context.

AutoSaddler's V2 architecture and scenario contract explain the durable candidates, attempt journal and split evaluation retained here. Current upstream documentation may include additions beyond our pinned AutoSaddler revision; installed behavior and the recorded pin remain authoritative.