How to Evaluate LLMs on Your Own Tasks: Build a Small Eval Set

Public benchmarks and leaderboards are useful for a first impression, but they answer someone else’s question. Whether a model is good at your job — extracting fields from your invoices, answering questions about your product, writing in your brand voice — can only be measured on your own examples.

The good news: you don’t need a research team. A small, well-built evaluation set of 30 to 100 examples, plus a simple scoring script, will tell you more about model fit than any leaderboard. This guide shows how to build one.

Why your own eval beats benchmarks

• Your inputs are different. Real data is messier than benchmark data: typos, odd formats, mixed languages, edge cases.
• Your definition of “correct” is specific. A benchmark might accept any reasonable summary; you might require exactly three bullet points under 20 words each.
• Benchmarks may be contaminated or over-optimised. Benchmark questions can leak into training data, and models can be tuned toward popular benchmarks.
• It’s reusable. Once you have an eval set, comparing a new model, a new prompt or a price change takes minutes.

Step 1: Define the task and success criteria

Write down precisely what a good output looks like. For example:

Given a customer email, return JSON with category (one of 12 fixed labels), order_id (string or null) and urgent (boolean). Output must be valid JSON with no extra text.

Clear criteria make scoring possible. If you can’t say what “good” means, no model choice will feel right.

Step 2: Collect real examples

Aim for 30–100 examples to start. Draw them from real usage where you can (with appropriate privacy handling — remove or mask personal data), and make the set representative:

• Typical cases (most of the set): the everyday requests.
• Hard cases: ambiguous inputs, long inputs, unusual formatting.
• Edge cases: empty fields, multiple possible answers, inputs in other languages, requests that should be refused or flagged.
• Known failures: anything that has broken before.

Fifty well-chosen examples beat five hundred random ones.

Step 3: Write the expected outputs

For each input, record the reference answer or the properties a correct answer must have. Store everything in a simple format, such as JSON Lines:

{"id": "t-014", "input": "Hi, order #A-5531 arrived broken, need a replacement ASAP", "expected": {"category": "damaged_item", "order_id": "A-5531", "urgent": true}}
{"id": "t-015", "input": "Do you ship to Canada?", "expected": {"category": "shipping_question", "order_id": null, "urgent": false}}

Have a second person review a sample of the labels. Disagreements are a sign your task definition needs sharpening.

Step 4: Choose a scoring method

Use the simplest method that captures quality for your task.

Method	Best for	Notes
Exact match	Labels, IDs, yes/no	Normalise case and whitespace first
Field-level match	Structured extraction	Score each field; report per-field accuracy
Validation checks	Format requirements	Valid JSON? Within length? Required sections present?
Contains / regex	Must-include facts	Simple, but can be gamed by verbose answers
Model-graded (LLM as judge)	Open-ended text	Use a clear rubric; spot-check against human ratings
Human rating	Tone, nuance, final checks	Slow but trustworthy; use blind A/B comparison

Using an LLM as a judge

For open-ended outputs such as summaries or replies, a grader model can score answers against a rubric. To keep it honest:

• Give the judge a specific rubric (“Score 1 if the reply mentions the refund policy and offers a next step, else 0”) rather than “rate quality 1–10”.
• Prefer binary or small-scale scores; they’re more consistent.
• Randomise answer order in pairwise comparisons, since judges can show position bias.
• Validate the judge: hand-score 20–30 outputs and check how often you agree with it.

Step 5: Run every candidate the same way

Fairness matters more than sophistication:

• Use the same prompt for every model — or, if you tune per model, tune each one equally.
• Fix parameters such as temperature and max output tokens, and record them.
• Run each example more than once if outputs are non-deterministic, and look at consistency as well as the average.
• Record tokens and latency alongside scores so you can calculate cost from the same run.

A minimal loop looks like this:

import json

def run_eval(model_call, path="eval.jsonl"):
    results = []
    for line in open(path):
        case = json.loads(line)
        output, usage = model_call(case["input"])   # your API wrapper
        passed = score(output, case["expected"])    # your scoring function
        results.append({"id": case["id"], "passed": passed, **usage})
    accuracy = sum(r["passed"] for r in results) / len(results)
    return accuracy, results

Step 6: Read the failures, not just the score

An aggregate score hides the story. For each model, read every failed case and tag the reason: wrong label, missing field, invalid format, hallucinated value, refused unnecessarily. Often you’ll find:

• Some failures are prompt problems that affect every model — fix the prompt and re-run.
• Some are label errors in your eval set — fix the label.
• The remainder are genuine model differences — these drive the decision.

Step 7: Decide with quality, cost and latency together

Put the results side by side. With hypothetical numbers, a summary might look like this:

Model (hypothetical)	Pass rate	Invalid JSON	p50 latency	Est. cost / 1K requests
Model A	96%	0	2.4 s	$9.20
Model B	93%	1	0.9 s	$1.15
Model C	81%	6	0.5 s	$0.30

On a set of 100 examples, a difference of a few percentage points can be noise. Before paying several times more for a small score gap, look at which cases differ and whether those cases matter to your users. Following the selection checklist, a reasonable rule is to choose the cheapest model that clears your quality bar.

Keep the eval alive

• Add every production failure to the set.
• Re-run when anything changes: a new model version, a prompt edit, a provider update.
• Version the set so results stay comparable over time.
• Watch for overfitting: if you tune prompts heavily against the eval, hold back a separate set of examples for final checks.

Key takeaways

1. Leaderboards are a filter; your own eval set is the decision.
2. Start with 30–100 representative examples and clear success criteria.
3. Use the simplest scoring method that works, and validate any LLM judge.
4. Compare models under identical conditions and record cost and latency at the same time.
5. Read failures; they tell you whether to change the prompt, the labels or the model.

Comparing eval results, prices and context limits for several models at once is exactly the kind of side-by-side view MaximumAI is being built to provide.