How Will You Know? — Designing the Study You'll Run

90 min

Overview

You have a committed agent, a question about it, and — as of lunch — a hunch backed by rough trials. Now comes the question that turns a hunch into research: how would you know? This session is where you design the study you would run to answer your question properly — defined measures, fair baselines, a small benchmark of real cases, a judging protocol. To be clear about what this is: the study you design today is the one you run after the seminar, as the heart of your finished paper. This week you will only road-test pieces of it, roughly, in tomorrow's proof-of-concept session. Design it anyway, and design it well — a paper with rough early evidence and a well-designed proposed study is an honest, strong paper. The deliverable is the Proposed Study: the Methodology section of Friday's skeleton.

The standard

Every measure must survive the question chain you saw this morning: What does it measure? Better for WHOM, for WHAT task? Would the number still move if the actual thing got worse?

A measure you cannot defend under those questions is not a measure. It is a decoration.

The session at a glance

1
Turn your rough tallies into defined measures — success AND cost.
2
Choose the baselines: what would your agent fairly be compared against?
3
Build the mini-benchmark: 5–10 representative cases from your domain.
4
Design the judging protocol: who or what decides, and how it's validated.
5
Swap with another team and hunt for gaps in each other's plans.
6
Assemble the Proposed Study.

Part 1 · Define the measures

At lunch you tallied roughly: worked or not, turns, tokens. Those are the right instincts — now ask which of them could grow into defined measures, and what your question actually needs measured:

A defined measure has four parts

what is countedthe number, comparison, or verdict actually collected — not "quality improves" but "fraction of the ten cases where the expert reviewer prefers the agent's output, blinded"

on whatthe inputs it is computed over — which cases, how many, chosen how

against whatthe reference that makes the number meaningful — a baseline, a threshold, a human's score on the same cases

what it missessay it yourself before a reviewer does — the aspect of "better" this measure does not capture, and what other evidence covers that gap

Let your question type steer what gets measured: a capability question measures task success and where it breaks; a design question measures the difference between builds; a safety question measures what the agent does when confused or misdirected — refusals, escalations, the worst action taken; an assessment question makes the measures themselves the contribution, and adds checks that they mean what they claim.

Cost is half the answer

Measure cost with the same rigor as success: tokens, dollars, wall time, human attention consumed. "It works" at unbounded cost answers nothing — the interesting result is almost always the trade.

Part 2 · Choose the baselines

An unfair baseline is how good teams fool themselves. Remember the frontier model that "lost" until someone gave it the prompt the specialized model had been trained on.

Pick at least two

1
Today's practice: the human process from your Positioning Memo — what does it score on your measures, or what would it plausibly score?
2
The generic cousin: your own agent stripped of what makes it yours — a frontier model with generic tools, no domain action language, no training document. The gap between it and your build is your design's contribution, measured.
3
The strongest fair variant: the best prompt and setup you can give the baseline. Write down the prompts — prompt selection is part of the design, and a reviewer will ask.

Part 3 · Build the mini-benchmark

Your measures need something to run on. Assemble five to ten representative cases from your domain — real ones where you can get them, realistic constructions where you can't.

Each case gets three lines

the inputthe actual task instance — concrete enough that two different agents could both attempt it tonight

what better looks likefor THIS case, in one checkable sentence — the fact recovered, the error caught, the decision supported

the trapwhy this case is in the set — the ambiguity, edge, or failure mode it exercises. A benchmark of only easy cases measures nothing

Coverage over volume

Ten cases that span the hard parts beat a hundred that repeat the easy one. Deliberately include: one common case, one rare-but-costly case, one ambiguous case, and one case where the RIGHT answer is "refuse and escalate to a human."

Part 4 · The judging protocol

Who or what turns an agent's output into a score?

The options — and what each must confess

ground truththe answer is checkable mechanically. Cleanest — when it exists. Say why your check really captures success

rubric + humanan expert scores against written criteria. Say who the expert is, and whether they are blinded to which system produced the output

LLM-as-judgecheap and scalable — but the judge has its own prompt, biases, and failure modes. If you use one, your plan must include validating the judge against human judgment on a sample

the user themselvesif your claim is about non-experts or practitioners, they belong in the loop — a proxy metric cannot stand in for the person the work is for

And because agents are probabilistic: state your repetition plan. How many runs per case? A single trajectory proves almost nothing — one response changes the next action, and the whole conversation diverges. Say how many trials, and how you will report the spread, not just the average.

Part 5 · The exchange — hunt for gaps

Swap plans with another team. Fifteen minutes each way — read theirs the way a helpful reviewer would, looking for the holes they can still fix:

The gap hunter's checklist

1
Could an agent score well on these measures without actually being good? Describe how.
2
Could a genuinely good agent score badly? Describe the honest failure.
3
Is any baseline weaker than it should be — would a better prompt or setup change the story?
4
Which case is missing — what would you add to the benchmark to cover the weakness the set doesn't touch?
5
Name the one measure a reviewer would question first — and suggest how to shore it up.

Take the gaps seriously and gratefully: every one found now is one a real reviewer doesn't find later, when it's too late to fix.

The deliverable: the Proposed Study

The Proposed Study

user prompt

Claims: the one or two claims the finished study could support — scoped to prompts, data, and conditions tested Measures: success measures and cost measures — each with its four parts Baselines: the comparisons, with the prompts/setups each will receive Mini-benchmark: the case list — input, "better looks like," and the trap, per case Judging protocol: who or what scores, how it's validated, and the repetition plan Known limits: what this study would not establish — written by you, before a reviewer writes it for you

Before we regroup

💾 Capture

Bring the Proposed Study.

Write down:

1The plan — all six blocks.
2The two most useful gaps the other team found, and what you changed because of them.
3One line: the measure you are still least sure of.

Be ready to discuss:

Tomorrow you road-test pieces of this — a rough proof of concept on two or three of your own cases. Tonight, make sure those two or three cases are concrete enough to actually run.