GEPA Omni Integration Experiment¶

This page records the controlled release experiment for SuperQode's GEPA Omni integration. It is reproducible evidence for the integration and a source for future release notes and blog posts. It is not a statistical model benchmark.

Outcome¶

The full Omni pipeline completed successfully on July 26, 2026:

• all three exploration engines ran with six real evaluations each
• GEPA continuation ran with six more evaluations
• MetaHarness found the winning exploration candidate
• the continuation retained a perfect optimizer-visible score
• the sealed held-out score improved from 0.0 to 1.0
• both held-out cases passed
• candidate policy audit reported no violations or warnings
• the live HarnessSpec was not modified
• Claude usage was authenticated through a Pro subscription, not an API key

Experiment design¶

The experiment optimized examples/harnesses/omni-tiny-local.yaml against examples/evals/omni-release.yaml.

The candidate was a complete HarnessSpec. Its optimization policy allowed changes to agents, model_policy, and workflow, while protecting execution policy, checks, approvals, and sandbox configuration.

The evaluation contract contained:

• three optimizer-visible release-decision cases
• two distinct sealed held-out cases
• both positive and negative release decisions
• exact output-format requirements
• unresolved-check, security-review, migration, and rollback conditions

Models and authentication¶

Before the run, claude auth status reported authMethod: claude.ai and a Pro subscription. ANTHROPIC_API_KEY and ANTHROPIC_AUTH_TOKEN were absent, and the runner also removed both variables from the child environment.

Reproducibility¶

The experiment ran against the Omni integration working tree based on the listed SuperQode commit. The release should record the final integration commit after these changes are committed.

Run the same bounded experiment:

scripts/run_tiny_omni_experiment.sh \
  --spec examples/harnesses/omni-tiny-local.yaml \
  --mode omni

The runner pins GEPA, selects the release task file, uses subscription authentication, and applies these bounds:

Total optimizer evaluations: 24
Per exploration engine:       6
Continuation:                 6
Total proposer ceiling:       $2.00 API-price-equivalent
Candidate workers per engine: 1

Explorer results¶

The proposer cost is the API-price-equivalent accounting returned by Claude Code and GEPA. It represents subscription allowance consumption for this run, not API-key billing.

The 24 optimizer evaluations used 10,820 local-model tokens. The final baseline-versus-candidate held-out comparison used another 2,013 local tokens, for 12,833 recorded local tokens in total. Wall-clock time was approximately 11 minutes 22 seconds.

Candidate and held-out result¶

The baseline prompt asked for a concise release decision but did not define a stable output contract or an explicit blocker policy. The winning candidate changed one field: the release-decider agent's system prompt.

The candidate added:

DECISION: yes
REASON: [brief explanation]

OR

DECISION: no
REASON: [brief explanation]

It also made the decision policy explicit: answer yes only when all requirements are met and no blocker remains; answer no when required checks fail or concerns are unresolved.

The candidate generalized to both sealed cases: a ready dependency update with a tested rollback plan, and a database change blocked by missing migration and rollback plans.

Safety result¶

The final candidate audit recorded:

Changed fields:    1
Changed surfaces:  agents
Policy violations: 0
Warnings:          0
Regressions:       0
Held-out gate:     passed

The optimizer did not modify execution permissions, checks, approval policy, sandbox configuration, model routing, or workflow. SuperQode staged the candidate separately and left the source HarnessSpec unchanged.

What this proves¶

This controlled experiment demonstrates that:

• subscription-authenticated Claude Code can drive GEPA's agent engines without an Anthropic API key
• Ollama can provide both harness rollouts and GEPA reflection
• Omni can compare three optimization strategies and continue from the winner
• SuperQode can audit complete HarnessSpec candidates before evaluation
• a separate held-out gate can reject regressions before human adoption
• proposer and evaluation budgets remain bounded and observable

Limitations¶

• This was one controlled run over five small binary-scored tasks.
• The intentionally under-specified baseline made improvement straightforward.
• The result demonstrates integration behavior, not general superiority over manual prompt engineering or another optimizer.
• Dollar values are API-price-equivalent usage, not an Anthropic API invoice.
• GEPA's budget-exhaustion stop currently emits verbose tracebacks even when a phase ends normally at its configured limit.
• LiteLLM emitted optional Bedrock/SageMaker import and asynchronous logging cleanup warnings; neither affected the local Ollama path or final result.

Use representative project tasks, multiple seeds, and repeated held-out runs before making production-quality or comparative performance claims.