Sample, with invented data
What a pilot readout looks like.
A governed-data agent pilot ends with a readout like this one. The company, questions, and metrics are examples to show the format.
- Correct answers
- 41 of 50
- Accuracy
- 82%
- Average model cost per answer
- $0.06
What was tested
The pilot covered accounts receivable for an invented distribution company. The finance lead agreed on 50 questions and the correct answer to each. The agent ran on Snowflake Cortex against the company’s governed receivables models, in a non-production environment.
Results by topic
| Topic | Questions | Correct |
|---|---|---|
| Invoice status | 15 | 14 |
| Aging and overdue balances | 14 | 11 |
| Customer payment history | 11 | 10 |
| Revenue by period | 10 | 6 |
| Total | 50 | 41 |
Example questions
| Question | Correct answer | Agent’s answer | Result |
|---|---|---|---|
| How many invoices were more than 60 days overdue on June 30? | 212 | 212 | Correct |
| Which customer has the largest open balance? | Northfield Supply | Northfield Supply | Correct |
| What was net revenue for the second quarter? | $18.4 million | $19.1 million | Wrong. Net revenue has two definitions. |
| What is the total of open credit memos? | $412,000 | No answer | Wrong. The agent couldn’t reach the table. |
| Who were our top customers last quarter? | Ranked by revenue | Ranked by invoice count | Wrong. The question was ambiguous. |
Why the wrong answers were wrong
| Cause | Answers | What fixing it involves |
|---|---|---|
| A metric with two definitions | 4 | Net revenue is calculated one way in the finance model and another way in the sales model. The owners choose one, or the agent is told which to use. |
| A table the agent couldn’t reach | 3 | Credit memos sit in a schema outside the access approved for the pilot. The team grants read access or takes those questions out of scope. |
| An ambiguous question | 2 | The finance lead rewords the question and it’s run again. |
What it cost to run
The 50 answers cost $3.10 in model usage, an average of $0.06 each. Individual answers ranged from $0.02 to $0.19, and the most expensive was a revenue question that needed four queries. These figures cover model usage only. Warehouse compute and hosting are billed by those providers and aren’t included.
The work needed before production
- Settle the definition of net revenue, then run the four affected questions again
- Decide whether credit memos are in scope, and grant access if they are
- Reword the two ambiguous questions with the finance lead
- Complete the security review, the operating plan and the owner’s approval, which the pilot doesn’t cover
Seven of the nine wrong answers trace to the net revenue definition and the credit memo access. We’d run the full answer key again after those are resolved, before recommending a production date.
Everything on this page is sample data. A readout for your pilot reports your questions, your data and the results the agent produced.
Tell us which business area you have in mind and where its data lives.
Discuss a project