Rebeca MurilloProduct and software engineer · AI Driven · R&DBook a call
EN·FR·ES
← All writing

Comparing agentic orchestration with the Claude Code harness: performance and token consumption

Quick read

The test

The same feature, built eight times from the same spec. Six builds went through my orchestrator: three workflows, each on Opus with Sonnet, then on Sonnet with Haiku. Two builds were each a single Claude Code session, with no orchestration around it.

The result

The eight builds delivered the same game. A script played the same 6,000 games on each build, and all eight gave exactly the same result. I also played each version on the tablet and saw no difference.

The cost

$15.68 to $16.56 on Opus and Sonnet, $2.67 to $4.84 on Sonnet and Haiku, at API prices. The plain sessions are estimated at $0.25 to $2.33. The models multiplied the cost by three to six; the method moved it much less.

My take

The setup I needed six months ago is already covered by the harnesses. Next: a more complex feature, with the model adjusted to each story.

Introduction

When I started exploring and testing open-source models in May 2026, I found myself babysitting my agents a lot. I had an OpenClaw setup with multiple agent personas, each with its own role, and getting through from an issue to a PR was not straightforward. I had a clear process for development, but the agents needed guidance and a clearer orchestration. So I started building one.

Fast forward to October 2026: harnesses like Claude Code have integrated these features in their systems, and they are capable of orchestrating pretty well. But I was curious about the token consumption in this case. When the harness orchestrates, one agent knows the process and iterates through it in loops, for every feature and every story, in one session. So my assumption was that the token consumption increases when the harness orchestrates in one session, making the whole run more expensive. What about splitting it into one session per story? And finally, how does it compare with my custom orchestration, which splits the implementation process into three steps: planning, implementation and review?

This is the first evaluation announced in the previous post, Why you need to do AI evaluations on your projects and repositories in 2026: one task, eight configurations on Claude models with Claude Code as the harness, one run each, graded by my repository’s own tests. The same feature built on open-weight models is the subject of a following post.

1. The setup and workflows

The detailed specs, the intent and the architecture are produced by a previous step, done with a human in the loop and validated by a human before the actual implementation starts. The results below come from that scenario. The specs were produced with the BMAD method for product specifications, and every evaluation run started from the same code baseline and the same specs.

The feature

The feature is a Snake game for Familio, a personal app I’m building with my children to track chores with rewards, with a game space. The stack is a full-stack Node.js application in TypeScript, with Astro, React, Vitest and Playwright. In the game, the player’s snake hunts computer-driven bad snakes, grows as it eats them and speeds up as it goes. The specification resulted in one epic with three stories:

  1. the hunt engine;
  2. the hunt in the record and on the parent’s card;
  3. the hunt’s tile, grid and screens.

A game of the hunt on a tablet screen: the player's purple snake eats the bad snakes on the grid, the counter goes up to 8, then the end card shows the new record

The same scripted game was recorded on every build. This one is one of the eight builds (y7d in the tables below).

The spec went through the full BMAD chain: requirements, UX design, architecture, story breakdown, then the spec and its stories. Every build started from the same commit. Each of the eight builds added 726 to 787 lines of production code and 1,450 to 2,142 lines of unit tests. The writing of the spec and its reviews are not counted in any of the token figures below.

The configurations

Each configuration ran through my orchestrator, with Claude Code as the harness.

BuildWorkflowOrchestrating sessionPlan and reviewImplement
y3fstagednone, the orchestrator starts each stepOpus 5.5Sonnet 5.5
t4estory sessionSonnet 5.5Opus 5.5Sonnet 5.5
y7depic sessionSonnet 5.5Opus 5.5Sonnet 5.5
h5kstagednoneSonnet 5.5Haiku 5.5
j5cstory sessionSonnet 5.5Sonnet 5.5Haiku 5.5
v2mepic sessionSonnet 5.5Sonnet 5.5Haiku 5.5
k3xplain Claude Codeone Sonnet 5.5 session——
r8nplain Claude Codeone Haiku 5.5 session——

Two limits of this grid: no build had Opus orchestrating, and the plain template takes a single model, so k3x and r8n are not the Sonnet-plans, Haiku-implements pair. j5c and v2m ran on a later orchestrator release and template version, with the prompts unchanged.

The four workflows

Every agent step writes a marker file with its status. A step that leaves no readable marker is set to be retried once. A blocked agent consults a senior model when the run names one, and parks for a person when it does not. The first three workflows share everything around the build: the spec gate before, then the unit-test verify, the end-to-end (e2e) test generation, the e2e quality gate and the retrospective.

Staged. My orchestrator starts every step as its own agent session: three sessions per story, each on its own model. The diagram shows where the work goes back after a failure.

flowchart TB
  gate[Spec gate, plan model]:::agent
  subgraph story [Each story, agents retried once]
    plan[Plan, plan model]:::agent --> impl[Implement, implement model]:::agent
    impl --> verify(Verify: unit tests):::check
    verify --> review[Review, review model]:::agent
  end
  gate --> plan
  verify -. fails: redo, max 2 .-> impl
  review --> more{More stories?}:::check
  more -->|yes| plan
  more -->|no| gen[E2E generation, plan model]:::agent
  gen --> e2e(E2E quality gate):::check
  e2e -. fails: redo, max 2 .-> gen
  e2e --> retro[Retrospective, plan model]:::agent
  retro --> pr([Pushed, link for the PR]):::check
  story -. blocked .-> senior[Senior consult, if set]:::agent
  verify -. still fails .-> senior
  senior -->|decision| story
  senior -. not set .-> park([Park for a person]):::gate
  gate -. blocked .-> park
  e2e -. still fails .-> park
  retro -. rejected .-> park
  classDef agent fill:#ffffff,stroke:#3d4654,color:#111827
  classDef check fill:#f2f5f4,stroke:#6b7280,color:#111827
  classDef gate fill:#e8f0ee,stroke:#006b56,stroke-width:2px,color:#111827

Story session. One Claude Code session per story spawns the same three stages as subagents, on the same models: the harness starts the stages instead of my orchestrator. The verify runs after the session, and a failure is set to send the whole session back. The next story starts a new session.

flowchart TB
  gate[Spec gate, plan model]:::agent
  subgraph session [One session per story, Sonnet, retried once]
    plan[Plan subagent]:::agent --> impl[Implement subagent]:::agent --> review[Review subagent]:::agent
  end
  gate --> session
  review --> verify(Verify: unit tests):::check
  verify -. fails: redo, max 2 .-> session
  verify --> more{More stories?}:::check
  more -->|yes, new session| session
  more -->|no| gen[E2E generation, plan model]:::agent
  gen --> e2e(E2E quality gate):::check
  e2e -. fails: redo, max 2 .-> gen
  e2e --> retro[Retrospective, plan model]:::agent
  retro --> pr([Pushed, link for the PR]):::check
  session -. blocked .-> senior[Senior consult, if set]:::agent
  verify -. still fails .-> senior
  senior -->|decision| session
  senior -. not set .-> park([Park for a person]):::gate
  gate -. blocked .-> park
  e2e -. still fails .-> park
  retro -. rejected .-> park
  classDef agent fill:#ffffff,stroke:#3d4654,color:#111827
  classDef check fill:#f2f5f4,stroke:#6b7280,color:#111827
  classDef gate fill:#e8f0ee,stroke:#006b56,stroke-width:2px,color:#111827

Epic session. One session for the whole epic runs the three stages for every story, in order, then the verify runs once for the epic.

flowchart TB
  gate[Spec gate, plan model]:::agent
  subgraph session [One session for the epic, Sonnet, retried once]
    plan[Plan subagent]:::agent --> impl[Implement subagent]:::agent --> review[Review subagent]:::agent
    review --> more{Next story?}:::check
    more -->|yes| plan
  end
  gate --> session
  more -->|no| verify(Verify: unit tests):::check
  verify -. fails: redo, max 2 .-> session
  verify --> gen[E2E generation, plan model]:::agent
  gen --> e2e(E2E quality gate):::check
  e2e -. fails: redo, max 2 .-> gen
  e2e --> retro[Retrospective, plan model]:::agent
  retro --> pr([Pushed, link for the PR]):::check
  session -. blocked .-> senior[Senior consult, if set]:::agent
  verify -. still fails .-> senior
  senior -->|decision| session
  senior -. not set .-> park([Park for a person]):::gate
  gate -. blocked .-> park
  e2e -. still fails .-> park
  retro -. rejected .-> park
  classDef agent fill:#ffffff,stroke:#3d4654,color:#111827
  classDef check fill:#f2f5f4,stroke:#6b7280,color:#111827
  classDef gate fill:#e8f0ee,stroke:#006b56,stroke-width:2px,color:#111827

Plain Claude Code. One session receives the spec folder and one prompt, which asks it to make the tests and the full quality gate pass. No spec gate, no stages, no verify, no e2e steps, no retrospective and no senior consult. The orchestrator only checks out, commits and pushes; it never runs the tests, so the grading after the run is what tells whether it worked.

flowchart TB
  pin(Pin commit, install):::check --> build[One session builds the epic]:::agent
  build -. no marker: retry once .-> build
  build -->|done| push(Commit and push):::check
  push --> pr([Link for the PR]):::check
  build -. blocked .-> park([Park for a person]):::gate
  classDef agent fill:#ffffff,stroke:#3d4654,color:#111827
  classDef check fill:#f2f5f4,stroke:#6b7280,color:#111827
  classDef gate fill:#e8f0ee,stroke:#006b56,stroke-width:2px,color:#111827

Sources.

2. The findings

Before the runs, I was expecting the Sonnet and Haiku pair to be the cheapest, but also the one to fail, or at least to fail a few times.

It was surprising to me that every version of the workflow returned the same result from the user experience perspective: the game was working correctly, the screens were developed correctly, the interactions were working, all equivalent to one another. I tested all the versions on the tablet: I played each one a few times, lost and won a couple of times on each, and observed no difference. Honestly, I was not expecting the same result on all versions.

The equivalence is measured on the eight builds:

  • A script played the same 6,000 games on the eight game engines, with the same random draws and the same key presses: the eight returned the same state at every step. One scripted game of 124 ticks also shows the same board at every tick on each build’s real screen.
  • Each build’s unit tests pass on the seven others’ code in 8,377 of 8,470 case runs, and its e2e specs on the seven others’ apps in 3,231 of 3,234. Every failure is a test tied to a choice the spec left open.
  • The eight are green on the repository’s full quality gate (the whole app’s suite, not only the feature’s tests), with 93.3 to 93.6% line coverage.
  • My rating on the tablet, done for seven of the eight without knowing which build I was playing, gave the same result every time.

One caution before the numbers: every configuration ran once. A 2026 study of coding agents on a single repository found that a measured gain in pass rate, at the number of tasks one repository supplies, “cannot be separated from the agent’s run-to-run variance”. A difference below is a difference between two runs.

BuildWorkflowModelsCostPrompt tokensOutput tokensMinutes running
y3fstagedOpus / Sonnet$16.5022.3M219k122
t4estory sessionOpus / Sonnet$15.6820.2M234k85
y7depic sessionOpus / Sonnet$16.5621.9M230k103
h5kstagedSonnet / Haiku$2.77 to $3.5017.8M209k90
j5cstory sessionSonnet / Haiku$3.33 to $4.8427.2M277k61
v2mepic sessionSonnet / Haiku$2.67 to $4.0119.8M255k54
k3xplainSonnet$2.33 (estimate)6.57M70k19
r8nplainHaiku$0.25 to $1.11 (estimate)12.25M132k40

The cost is a price, not a bill: my LiteLLM gateway priced each request at API rates (November 2026), while the builds ran on my subscription. The gateway had no price for Haiku above 100k tokens of context, so its figure is a lower bound; the upper figure is Claude Code’s own estimate. The two plain builds never went through the gateway: their dollars are estimates and their tokens are the orchestrator’s count, so I compare them on tokens, and approximately. Prompt tokens include cache reads.

Prompt tokens of each run, cache reads included: 22.34M, 20.21M and 21.95M for the three Opus and Sonnet builds, 17.84M, 27.22M and 19.81M for the three Sonnet and Haiku builds, 6.57M and 12.25M for the two plain Claude Code sessions.

Three things stand out:

  • The model pair multiplied the cost by three to six. The token counts of the two pairs are close; the price per token is what differs.
  • The method moved it little on Opus and Sonnet (20.2M to 22.3M prompt tokens) and more on Sonnet and Haiku (17.8M to 27.2M).
  • The plain Claude Code sessions used the fewest tokens and the least time, without any of the steps around the build.

The third point is the one that stayed with me: the setup I needed six months ago is already covered by the harnesses.

Sources.

3. Interesting insights

#QuestionShort answer
1Are my orchestration steps before and after development a guardrail that helps low-capacity agents reach the goal?Not on this feature. The verify and the e2e gate stopped no defect in the six orchestrated runs, and the plain Haiku session reached the same game with no gate at all.
2Are those steps useless when using high-capacity models?They found little to repair: 7 review patches on the three Opus and Sonnet builds, none for something a user would have met. “Useless” is more than one run per configuration can say.
3Is the token consumption equivalent for the same models but different methods?On Opus and Sonnet, yes (20.2M to 22.3M prompt tokens). On Sonnet and Haiku, no (17.8M to 27.2M), although the highest run repeated a step twice.
4Which method and model mix is the least efficient, or the most costly but not worth it?The Opus and Sonnet pair, whatever the method: three to six times the cost for the same game and slightly stronger e2e tests.
5Which step of my orchestrator is useless or redundant?On this spec, the e2e generation step: 0 to 5 tests added after the stories had written their own. I am keeping it for now as a safety net, and the retrospective for self-learning.
6Are the tests equivalent: e2e generated, unit tests, coverage?The coverage is: 93.3 to 93.6% of lines. What the tests catch is not: 8 to 13 of 13 defects injected into the game screen.
7How many retries, failures and senior unlocks?Nine failed attempts in three runs, none caused by the code, and no senior consult in any run. The runs waited about 133 minutes for me, 131 of them on infrastructure.

More details on each of these questions below.

1. Are my orchestration steps a guardrail that helps low-capacity agents reach the goal?

BuildUnit-test verify failed on the codeE2E gate failed on a testTest runs the agents started themselves (unit / e2e)Full quality gate after the run
y3f0 of 30 of 430 / 20green
t4e0 of 30 of 122 / 8green
y7d0 of 10 of 129 / 14green
h5k0 of 30 of 142 / 17green
j5c0 of 30 of 432 / 14green
v2m0 of 10 of 130 / 11green
k3xno gateno gate8 / 5green
r8nno gateno gate13 / 5green

The gates had nothing to catch: the agents ran the same suites themselves before handing over (30 unit and 20 e2e runs in y3f alone) and fixed what failed inside the step. So the gates confirmed that the build was green. The plain Haiku session, with no gate at all, is graded green too; its tests are another matter (question 6).

I injected 16 defects into each build’s engine to see what its tests catch (question 6): the eight miss the same three, the ones the acceptance criteria do not describe. The eight builds also wrote the same 18 production files. They follow the spec closely; no run varied the spec, so how much of the result it explains is not measured.

2. Are those steps useless when using high-capacity models?

BuildReviewerTime of one reviewReview patchesCode defects or test gaps named by the retrospective
y3fOpus69 to 739 s39
t4eOpus104 to 804 s211
y7dOpus78 to 774 s26
h5kSonnet28 to 39 s01
j5cSonnet21 to 40 s00
v2mSonnet19 to 29 s01
k3xno review——no retrospective
r8nno review——no retrospective

With Opus the steps read the code, reran the suites and added tests and findings; none of the 7 patches fixed something a user would have met. With Sonnet they mostly confirmed what the implementer had already run: five of the nine reviews ran no command. The retrospectives named 26 code defects or test gaps on the three Opus and Sonnet builds and 2 on the three others; a retrospective reports them and repairs nothing.

“Useless” is more than the data can say, though: the reviewer and the implementer changed together (Opus reviewed Sonnet’s code, Sonnet reviewed Haiku’s), and no plain Claude Code session ran on Opus.

3. Is the token consumption equivalent for the same models but different methods?

BuildWorkflowPrompt tokensAgainst the staged build of the same modelsOutput tokensTool calls
y3fstaged, Opus / Sonnet22.3M—219k323
t4estory session, Opus / Sonnet20.2M−10%234k284
y7depic session, Opus / Sonnet21.9M−2%230k291
h5kstaged, Sonnet / Haiku17.8M—209k214
j5cstory session, Sonnet / Haiku27.2M+53%277k325
v2mepic session, Sonnet / Haiku19.8M+11%255k245
k3xplain, Sonnet6.57Mother counter, not comparable70k47
r8nplain, Haiku12.25Mother counter, not comparable132k75

On Opus and Sonnet, the three methods cost $15.68 to $16.56, 6% apart. That is close to the 3 to 4% or so by which the gateway and Claude Code’s own estimate disagree on the same runs. On Sonnet and Haiku the story session used 53% more tokens than the staged run, but j5c repeated its e2e generation twice (question 7) and ran on a later orchestrator release, so one run cannot say the method is the cause.

The numbers do not settle my assumption. On Opus and Sonnet, the epic session, where one agent loops through the whole process, used slightly fewer tokens than the staged run; on Sonnet and Haiku, 11% more. The plain Sonnet session used about a third of the tokens of the Opus and Sonnet builds, and the plain Haiku session half to two-thirds of the Sonnet and Haiku builds, but they skipped the spec gate, the e2e generation and the retrospective, so this is not a like-for-like comparison. Haiku alone took about twice the tokens and twice the time of Sonnet alone for the same green result, with about half of that time lost on a killed command (question 7).

4. Which method and model mix is the least efficient?

BuildModelsCostScreen defects caught by its e2e specs (of 13)By its component testsBy either
y3fOpus / Sonnet$16.50111012
t4eOpus / Sonnet$15.68101011
y7dOpus / Sonnet$16.56111213
h5kSonnet / Haiku$2.77 to $3.508910
j5cSonnet / Haiku$3.33 to $4.8491012
v2mSonnet / Haiku$2.67 to $4.0171112
k3xSonnet, plain$2.33 (estimate)101011
r8nHaiku, plain$0.25 to $1.11 (estimate)868

The Opus and Sonnet pair is the costly one that does not gain much. What the extra cost bought is in the tests: I injected the same 13 defects into each build’s game screen and ran that build’s own tests. The e2e specs of the Opus and Sonnet builds catch 10 to 11, against 7 to 9 for the Sonnet and Haiku builds; with the component tests the gap almost closes. The plain Sonnet session is in the same range, and the plain Haiku session is last.

Defects injected into the game screen and which builds' tests catch them, by component tests and by e2e specs. Six defects are caught by every build at both levels; the plain Haiku build misses five of the thirteen.

5. Which step of my orchestrator is useless or redundant?

BuildSpec gateBuildE2E generationRetrospectiveTests added by the e2e generation
y3f staged, Opus / Sonnet$0.60$8.74$2.43$4.733
t4e story session, Opus / Sonnet$0.49$10.75$1.08$3.375
y7d epic session, Opus / Sonnet$0.55$11.81$1.01$3.181
h5k staged, Sonnet / Haiku (lower bound)$0.18$2.12$0.15$0.320
j5c story session, Sonnet / Haiku (lower bound)$0.17$2.39$0.46$0.300
v2m epic session, Sonnet / Haiku (lower bound)$0.18$2.02$0.19$0.280
k3x plain, Sonnet (estimate)no step$2.33no stepno step—
r8n plain, Haiku (estimate)no step$0.25 to $1.11no stepno step—
Where the money of each run went, per group of steps: the build takes $8.74 to $11.81 on Opus and Sonnet and $2.02 to $2.39 on Sonnet and Haiku; the retrospective takes up to $4.73.

On this spec, the e2e generation step is redundant. The spec asks stories 2 and 3 to write their own e2e spec, so the step came after the coverage was already there. Every model wrote that the stories had covered the feature, then added 0 to 5 tests, all of which passed on code nobody changed.

After the build, the retrospective is the most expensive step in five of the six runs: 9 to 29% of each run’s cost. All six ended on the same verdict, accepted with open items. The verify and the e2e gate never stopped a defect, but they are deterministic, taking about 10 and 630 seconds, and they are the proof the build is green. The plan steps and the spec gate were not analyzed.

I think the e2e generation is a safety net for potential bugs, and the retrospective is used for self-learning. So I am keeping both for now. I would keep them during the onboarding of a project; once every run is stable (no bugs, no regressions), they could be removed to save tokens.

6. Are the tests equivalent: e2e generated, unit tests, coverage?

BuildUnit-test cases writtenE2E cases writtenLine coverage of the repositoryLine coverage of the game screens it wroteEngine defects caught (of 16)
y3f1002493.62%100%13
t4e952793.63%100%13
y7d1022493.64%100%13
h5k931793.38%95.28%13
j5c1112293.61%99.56%13
v2m1171893.64%100%12
k3x791993.63%100%13
r8n741893.32%94.78%13

The repository’s coverage does not tell the builds apart. The coverage of the game screens does a little: h5k and r8n, at 95%, are also the two builds whose tests catch the fewest screen defects (question 4). The Sonnet and Haiku builds wrote shorter e2e suites (17 to 22 cases against 24 to 27), and the plain sessions wrote the fewest unit tests, with most of the gap on the child’s screens. v2m wrote the most unit tests of the eight and is the only build whose engine tests miss a defect the others catch. No e2e suite of the eight notices that the game stops speeding up.

7. How many retries, failures and senior unlocks?

BuildFailed attemptsSteps sent backAutomatic retriesParksSenior consultsMinutes waiting for me
y3f32010100
t4e000000
y7d000000
h5k3003032
j5c320101
v2m000000
k3x0000no senior0
r8n0000no senior0

None of the nine failed attempts was an agent stuck on the code. In y3f and j5c, the e2e gate could not start three times because an earlier agent’s test server still held its port, and the failures sent the e2e generation step back twice in each run ($1.48 and $0.31 of repeated work). In h5k, a submodule update timed out twice, and one implement step blocked on a sentence of the spec, which I answered in under two minutes. No run consulted a senior model: the three Opus and Sonnet builds ran without one, and j5c and v2m had Opus 5.5 as senior and never called it.

The largest loss of time was not a step: the full e2e suite takes about ten minutes, the length of the harness’s command window, so an agent running it in the foreground gets it killed and leaves its test server on the port. That cost y3f its 100-minute park, and the plain Haiku session about 20 of its 40 minutes.

I have a Claude session on the host watching the runs: when a run is parked, I receive a message from Claude. So the time parked is the time I was away from my phone or my computer (wow, a long time, it seems).

Conclusion

After seeing that the results were the same on the different AI model setups, I think the feature was too simple and easy: it was not challenging enough, even for very low-capacity models (the app already had a first Snake game, and 69 to 87% of each new game screen component is that game’s screen, adapted). The builds’ metrics are different from one another, but the result was still the same, which got me thinking about the actual complexity of the task.

The setup I needed six months ago is already obsolete, with Claude Code at least. Tech is moving fast and we need to keep up! This gives room for more explorations on different setups and methods.

After these observations, I have one more question: is there proof in these results that the spec definition is what guarantees the final result, no matter the model and the method? A lot of planning was already done ahead of time, so there were very few decisions left to the agent. The eight builds wrote the same 18 production files, but no run varied the spec, so this evaluation cannot answer it.

For my next features, I would run either the staged workflow with a mix of models, for more control, or one session per story. In both cases, the model should be adjusted to each story’s complexity, since the story complexity might change the results. So I will build into my orchestrator the ability to switch models per story.

As a next step, I will be evaluating a more complex feature with different setups, to observe the differences.

Working together

I keep room for project based or part-time consulting collaborations.

Book a call →