Service

Agent eval and production readiness

We build an evaluation for your AI agent that runs in your own pipeline and shows how the agent performs on agreed tasks and what to fix before a production decision.

Scope
One agent
Price
From $35,000
Typical duration
2 weeks
Delivery
Remote, in your environment

What you get

You get an answer key for the tasks your agent has to handle and an evaluation that runs in your own pipeline whenever the agent changes. We run it against the models you choose and write up which tasks pass, which fail and why, and what each model costs per answer.

  • An evaluation running in your CI pipeline
  • An answer key for the agent’s tasks, agreed on with your business owner
  • Pass and fail results for each model you choose to test
  • The lowest-cost model, among those tested, that meets the agreed criteria
  • A readout that lists the gaps to close before a production decision

How the result is measured

Your business owner agrees on the tasks and on what a correct result is for each. The evaluation runs the agent against that answer key and records pass or fail for each task, along with the cost per answer for each model tested.

The model comparison covers only the models you choose to test. If a smaller model meets the agreed criteria, the readout shows it and the difference in cost.

A pass means the agent handled the tasks in the answer key. A production decision also depends on your security review, your operating plan and an owner’s approval, so the readout lists the gaps we saw alongside the results.

When it fits

  • Teams with an agent or prototype that someone has to approve for production
  • Teams whose agent did well in a demo and who can’t yet say how often it’s right
  • Teams that want a repeatable way to decide whether to switch when a new model is released or a price changes

This service needs an existing agent to test, so if you don’t have one yet, start with the governed-data agent pilot, which includes an answer key and an evaluation.

What we need from you

  • An existing agent or prototype
  • Access to its repository and CI pipeline
  • A business owner who can agree on the correct results

Working in your environment

The evaluation runs in your own CI pipeline and cloud account, under the access your team approves.

After the engagement

You receive the answer key, the evaluation and the readout. The evaluation stays in your pipeline, so your team can add tasks to it and run it again when the prompt, the data or the model changes.

Questions

What is an agent eval?

An agent eval measures an agent’s performance against a defined set of tasks and expected results. You can run it again to see how a change affects performance.

Can you test whether our AI agent is ready for production?

The evaluation shows how the agent performs. It runs in your own pipeline and reports pass and fail results for each model you choose, and the readout lists the gaps to close before a production decision.

What does agent eval and production readiness cost?

It starts at $35,000 for one agent. Our proposal gives a fixed price before work starts.

How long does an agent eval take?

An agent eval typically takes two weeks from kickoff, once we have access to the agent and its pipeline.

Can an eval lower our model bill?

It can, if a smaller model meets the agreed criteria. The evaluation runs the same tasks against each model you choose, and the readout shows the difference in cost per answer.

What happens when a new model is released?

You run the same evaluation against the new model and compare the results with the last run.

Tell us about the agent you have and the decision you need to make about it.

Discuss a project