Rebeca MurilloProduct and software engineer · AI Driven · R&DBook a call
EN·FR·ES
← All writing

Why you need to do AI evaluations on your projects and repositories in 2026

Quick read

#1 Avoid vendor lock-in

Plans change and services go down. When your only provider is disrupted, your whole development activity stops.

#2 Know your features costs

In my first evaluation, the same feature cost from about $3 to $21 at API prices. A number per feature is a baseline for the next pricing adjustment.

#3 Identify which models to use, and when

Which step of the process deserves Opus-level reasoning, and for which one would Haiku be enough? Evaluation runs on your own repository will tell you.

Two limits

One run is not a result, and results go stale: evaluations have to be repeated when models or harnesses change.

Introduction

The evolution of AI in 2026 has been overwhelming, with AI models and coding harnesses evolving into fully autonomous and capable agents. By the end of 2026, in my projects, when enough context and guidelines are provided to an agent, the quality of the code is at senior engineer level, especially with the most capable models. The research is more divided: merged agent pull requests attract fixes at 1.62 times the odds of human PRs in the same repositories. Of course these studies are subjective, since the original prompt and planning with a human would change the result. In my experience, agents also catch more minor errors than the human eye.

The question today is: which model to use for each step of a process and for what budget?

We still get free tokens with frontier model subscriptions, when measuring token consumption and cost since a $100 or $200 subscription gives a lot more value in tokens. One independent measurement estimated in January 2026 that Claude plans deliver 8 to 37 times their price in API-equivalent usage. After tracking several of my developed features, I saw the cost of the token consumption go up to $500 to $1,000 (crazy!), while in reality I am only paying for a $200 subscription. This was an important feature, developing stories through 3 different repositories for the same project.

That got me wondering: how long will this pricing last? GitHub wrote in April 2026 that “the current premium request model is no longer sustainable” for agentic usage. During 2026, most providers changed how usage is counted:

  • Anthropic, from November 2025 to February 2026: Enterprise seats no longer come with subsidized tokens; every token is billed at API rates.
  • OpenAI, in April: a new $100 Pro tier, and Codex usage rebalanced in Plus.
  • GitHub, on June 1: premium requests replaced by AI Credits.
  • Cursor, in June: new Teams pricing.

Not every change went the same way: Anthropic doubled the Claude Code five-hour limits in May. Still, I think we should be expecting another adjustment in the upcoming year.

Sources.

#1 Avoid vendor lock-in

In a highly competitive market, with prices and plans changing constantly, it is highly risky to work with only one AI model provider.

  • Plans and services change: Google announced in May 2026 that Gemini CLI would stop serving requests for its AI Pro and Ultra subscribers on June 18, one month later, in favor of a new CLI.
  • Services go down: Anthropic reported errors across all its models, Claude Code included, for nearly five hours on July 30, 2026; OpenAI users were unable to access ChatGPT, Codex and the API Platform on April 20, 2026; Google’s Vertex AI Gemini API returned elevated errors for two hours on February 27, 2026. When your only provider is disrupted, your whole development activity just stops (for a few hours each time, so far).

This is one of the reasons I am building a software factory compatible with three harnesses for now: Claude Code, Codex and OpenCode, with a LiteLLM gateway behind them, a single API in front of many model providers, so I can also switch AI model providers.

Sources.

  • Google: “On June 18, 2026, Gemini CLI and Gemini Code Assist IDE extensions will stop serving requests for Google AI Pro and Ultra” (Transitioning Gemini CLI to Antigravity CLI, May 2026).
  • Anthropic, July 30, 2026: the incident affected “claude.ai, Claude API (api.anthropic.com), Claude Code, and Claude Cowork” (Anthropic status page).
  • OpenAI, April 20, 2026: “Impacted users are currently unable to access ChatGPT, Codex and API Platform” (OpenAI status page).
  • Google Cloud, February 27, 2026: Vertex AI Gemini API, “Incident began at 2026-02-27 04:37 and ended at 2026-02-27 06:35” (Google Cloud status page).

#2 Know your features costs

It is well documented that AI agents perform differently depending on the programming language and the environment used. A 2026 benchmark on multilingual refactoring found that “performance varies substantially across languages. No single model dominates all languages”; another 2026 study shows that the token cost can vary significantly by programming language; and with the model held fixed, the harness alone makes success rates and resource consumption “vary substantially”.

So it is key to know how much a feature is costing, in tokens and in monetary value. Putting a number on a feature also gives you perspective on the value received and the investment being made. In my first evaluation, the same feature, built from the same spec on the same repository, cost from about $3 to $21 at API prices, and the models moved the cost much more than the method (results in the next posts).

Even if you are running on a subscription that seems to be giving tokens away, you should at least know how much each developed feature is costing in tokens. This knowledge is also a baseline to prepare for a potential pricing adjustment in the market, so you can plan ahead for a potential rise in your AI budget. (How you count matters: in that evaluation, one open-model run cost $2.72 at the gateway and $97.95 in the harness’s own estimate, which applies Claude prices to other models.)

Sources.

#3 Identify which models to use, and when

For some development tasks, is it really necessary to use highly capable models all the time? When can we use a lower-capacity model, and why? Which step of the process deserves Opus-level reasoning, and for which one would Haiku be enough? What is the pricing difference between frontier models and open-source models? Can we train a low-capacity model to provide better quality results? So many questions that can be answered by comparing different runs with different models.

Research is starting to answer some of them. In one 2026 study, routing each task to the right model matched the best single model’s solve rate “at about a fifth of the total cost per solve”. Claude Code, for one, lets you assign a model per subagent, “routing tasks to faster, lower-cost models like Haiku”.

But the price list alone does not give the cost of a feature. In my first evaluation, the most expensive build was not one of the three on Opus and Sonnet ($15.68 to $16.56) but a staged build on open-weight models (about $21.03), because of what the models did with their turns after the code was written. Public benchmarks will not tell you which case applies to your code: Epoch AI notes that on SWE-bench Verified “all current and future models have been trained on the codebases used in its tasks”. Evaluation runs on your own repository will tell you which model performs better for your specific codebase.

Sources.

Conclusion

Experimenting with evaluations on a repository can provide a lot of value: in getting to know the codebase better, but also in knowing the actual implementation budgets, and which models are best suited for the tasks.

Two limits to keep in mind. One run is not a result: on a single repository, a 2026 study found that a 4.9-point gain, at the number of tasks one repository supplies, “cannot be separated from the agent’s run-to-run variance”. And results go stale: Anthropic’s own postmortem describes how a change of default reasoning effort and a new system prompt instruction changed Claude Code’s quality for weeks, with the same model. Evaluations have to be repeated when models or harnesses change.

I will be doing evaluations on some of my projects, and I will report here, starting with one feature built by different models and methods.

Sources.

Working together

I keep room for project based or part-time consulting collaborations.

Book a call →