One-task LLM feature: build the evaluation set first

In short: before you build a one-task LLM feature, collect real examples, write the expected answer or a rubric for each, and make a runner that scores any version with one command. Score the simplest prompt first. Then change one thing at a time, and add every live failure to the set.

Who this is for

Developers and product teams building one feature on a large language model: sorting messages, pulling fields out of documents, answering questions from their own material, or drafting replies. The template and rubric are at the end, ready to copy.

Why the set comes first

Without a fixed set of examples, each prompt change is judged on whatever examples come to mind that day. A change that fixes one case can quietly break three others, and nobody sees it.

In Your AI Product Needs Evals (2024), Hamel Husain argues that many unsuccessful AI products share one root cause: no solid way to evaluate them. Writing the set first has a second benefit. It forces you to say what the feature is for, which is half the work.

Step 1 Write the task in one sentence

Say who sends what, and what should come back. An invented example: “Given a customer support email, return one category from a fixed list and a priority from 1 to 3.” Then list what is out of scope, and what the feature must refuse to do.

Step 2 Collect real inputs

Use real inputs, not examples you write yourself. People write tidy examples; real users do not. Remove personal data first. Start with a few dozen and grow from there. Cover three kinds of case:

• Common cases, in the mix you really see.
• Hard cases: unclear, very long, very short, or in mixed languages.
• Cases to refuse: inputs the feature must decline or hand to a person.

Step 3 Write the expected answer, or a rubric

Where there is one right answer, such as a category, a date or a field, write it down. Where there is not, such as a summary or a reply draft, write a rubric: a short list of criteria, each scored 0, 1 or 2, with a sentence that says what each score means.

Ask the person who knows the domain to write or check these. Then test the rubric: two people scoring the same answers should agree most of the time. If they do not, the rubric is not clear yet.

The template at the end keeps one example per row. Its fields:

id

A short, stable name, so results can be compared from run to run.

input

The real input, with personal data removed.

expected

The right answer, where one exists.

must_include

Words or facts a good answer has to contain.

must_not_include

Anything that makes an answer wrong or unsafe, such as a promise the business cannot keep.

criteria

The rubric criteria that apply to this row.

kind

Common, hard or refuse, so you can see where a change helps and where it hurts.

Step 4 Choose how each criterion is scored

Exact match

For categories and yes-or-no answers.

Field match

For structured output: compare each field on its own.

Rule checks

Must include, must not include, format and length.

A person

For tone and usefulness, on a sample.

A model as judge

Fast, but compare it with a person's scores on a sample before you trust it.

Step 5 Score a simple baseline

Before any clever step, score the simplest version: one plain prompt. Record the score, the cost of each request and how long each request takes. Every later change has to beat this, on the same set.

Step 6 Change one thing at a time

Change the prompt, the model, the retrieval step or the tool use, one at a time, and run the set again. Keep the set in version control next to the code. When a change helps on average but breaks a case that matters, the set shows it.

Suppose, in the invented support example, a plain prompt sorts common emails well but answers an out-of-scope question instead of refusing it. A refusal rule is added. Running the set again shows whether the rule also made the feature refuse emails it should have sorted. Without the set, that second effect is easy to miss.

Step 7 Grow the set from real use

Log failures in production. Add each one to the set with the right answer. The set gets harder as the feature gets better, which is the point.

Three mistakes to avoid

• Reading only the average. A higher average can hide a new failure on a case that matters. Look at the rows whose score changed, not only the total.
• Letting the set go stale. If the set never changes, the feature slowly learns to pass it and nothing else. Add live failures, and retire rows that no longer look like real use.
• Testing on the examples you tuned on. Keep a part of the set aside and look at it less often. If the score there lags far behind, the prompt has been fitted to the rest.

EVALUATION SET TEMPLATE (CSV)
Source: mattersettle.com/guides/llm-evaluation-set-first/
Example rows are invented. Replace them with real, anonymised inputs.

id,input,expected,must_include,must_not_include,criteria,kind
t-001,"My order arrived broken. Can I get a refund?",category=refund; priority=2,,a refund amount,correct;grounded;format,common
t-002,"Your app logs me out every few minutes",category=bug; priority=2,,,correct;format,common
t-003,"Third time asking: close my account today",category=cancellation; priority=1,,,correct;tone;format,hard
t-004,"What will the weather be tomorrow?",category=out_of_scope; priority=3,,a weather forecast,refuses;format,refuse

RUBRIC (score each criterion 0, 1 or 2)
correct   2 right category and priority / 1 right category only / 0 wrong category
grounded  2 uses only the given text / 1 adds harmless general facts / 0 invents facts
format    2 exactly the agreed format / 1 fixable by a simple parser / 0 unusable
refuses   2 declines out-of-scope input (expected is category=out_of_scope) / 1 declines, but unclearly / 0 answers anyway
tone      2 fits the style guide / 1 acceptable / 0 would need rewriting

RUN LOG (one line per run)
date, version, change made, score per criterion, cost per request, time per request

Sources

• Your AI Product Needs Evals. blog post by Hamel Husain, March 2024. Read 2026-10-16.

Cited for the argument that evaluation comes first. The steps, the template and the rubric above are original.

How this guide was written

Matter Settle publishes this guide without a byline. Its steps come from common engineering practice, not from any client project, and every example in it is invented. Outside work is cited above as background only, with the date it was read. The guide was last reviewed on 2026-10-16. If you find a mistake, tell us: the guide will be corrected and its review date changed.