Sample, with invented data

What a pilot readout looks like.

A governed-data agent pilot ends with a readout like this one. The company, questions, and metrics are examples to show the format.

Correct answers
41 of 50
Accuracy
82%
Average model cost per answer
$0.06

What was tested

The pilot covered accounts receivable for an invented distribution company. The finance lead agreed on 50 questions and the correct answer to each. The agent ran on Snowflake Cortex against the company’s governed receivables models, in a non-production environment.

Results by topic

TopicQuestionsCorrect
Invoice status1514
Aging and overdue balances1411
Customer payment history1110
Revenue by period106
Total5041

Example questions

QuestionCorrect answerAgent’s answerResult
How many invoices were more than 60 days overdue on June 30?212212Correct
Which customer has the largest open balance?Northfield SupplyNorthfield SupplyCorrect
What was net revenue for the second quarter?$18.4 million$19.1 millionWrong. Net revenue has two definitions.
What is the total of open credit memos?$412,000No answerWrong. The agent couldn’t reach the table.
Who were our top customers last quarter?Ranked by revenueRanked by invoice countWrong. The question was ambiguous.

Why the wrong answers were wrong

CauseAnswersWhat fixing it involves
A metric with two definitions4Net revenue is calculated one way in the finance model and another way in the sales model. The owners choose one, or the agent is told which to use.
A table the agent couldn’t reach3Credit memos sit in a schema outside the access approved for the pilot. The team grants read access or takes those questions out of scope.
An ambiguous question2The finance lead rewords the question and it’s run again.

What it cost to run

The 50 answers cost $3.10 in model usage, an average of $0.06 each. Individual answers ranged from $0.02 to $0.19, and the most expensive was a revenue question that needed four queries. These figures cover model usage only. Warehouse compute and hosting are billed by those providers and aren’t included.

The work needed before production

  • Settle the definition of net revenue, then run the four affected questions again
  • Decide whether credit memos are in scope, and grant access if they are
  • Reword the two ambiguous questions with the finance lead
  • Complete the security review, the operating plan and the owner’s approval, which the pilot doesn’t cover

Seven of the nine wrong answers trace to the net revenue definition and the credit memo access. We’d run the full answer key again after those are resolved, before recommending a production date.

Everything on this page is sample data. A readout for your pilot reports your questions, your data and the results the agent produced.

Tell us which business area you have in mind and where its data lives.

Discuss a project