Research-informed AI evaluation and design.
I work where subjective quality meets measurement: designing scoring frameworks and rubrics, testing them for consistency across human and AI evaluators, and writing the known soft spots into the record instead of pretending they’re solved. The project below is the four-phase system I built to figure out when a prompt is actually good.
I was building an AI Copilot for educators and administrators to build AI literacy and generate their own prompts. The prompt was the product.
I talked with practicing educators to understand what they needed. The prompt was what they valued most.
There was no way to tell whether the prompts the Copilot generated were any good.
But before we could measure “good,” we had to define what “good” meant.
So I built two things: a way to measure prompt quality, and an evidence-based knowledge base the Copilot could draw from.
That turned into four phases, each one triggered by what the last revealed.
Four phases, each triggered by what the last revealed. Click any phase to open it.
Every phase feeds the knowledge base the Copilot draws from.
Design question: What makes a prompt a good prompt?
Prompt quality is easy to have opinions about and hard to measure. Phase 1 turned opinion into an instrument.
My starting draft had fifteen constructs, a parallel list of "observable features," and a separate summary table. It was a thorough brainstorm, not an instrument. Turning it into one meant grounding every construct in research on what actually makes a prompt work, not just my own judgement.
Fifteen constructs and a list of observable features, everything that might signal a good prompt.
Cut fifteen down to ten, decided what each one actually measures, chose a response format for each, wrote evidence-based indicators, and ran a formal QA pass before calling it stable.
Does it earn its place? If a construct always moves in step with another, the two are measuring the same thing, so one can go. Every construct has to catch something the others miss.
Eight constructs use a 1 to 5 scale; two are recorded as present/absent flags. A contradiction inside a prompt isn't "3 out of 5": it either exists or it doesn't. Forcing it onto a rating scale would invent precision that isn't there.
This choice pays off in Phase 2: the whole disagreement between the two AI evaluators turned on one of those flags.
The framework scores each prompt on ten constructs: eight on a shared 1 to 5 scale, and two recorded as evaluator flags. Two are shown here in full so the two response types are clear; the rest are part of unpublished work.
Is it clear what the AI should produce, and what a successful result looks like?
A clear task with no definition of success is usually a 3, not a 4.
Do the prompt's parts contradict each other, or fail to serve its stated goal?
Not a matter of degree. The problem is either there or it is not.
The remaining constructs are scored the same way and stay under wraps:
Three real products, not just a write-up. Each one also becomes part of the knowledge base the Copilot draws from, so the same standards that grade a prompt also guide how new ones get written.
Eight scored 1 to 5, two recorded as flags, under one shared scoring philosophy.
For every construct: a definition, one evaluation question, observable indicators, and common weaknesses.
Level-by-level descriptions for the eight rated constructs, with notes on the close calls.
What the versions mean: the guide is at v1.0, the first complete, stable version, ready to test; the rubrics are at v0.1, an earlier draft still being calibrated against pilot data. Both are working versions with the known soft spots written down, not finished instruments.
Here is one of those products up close, a single rubric from the set:
No objective can be identified. A reader cannot say what the AI is meant to produce.
An objective is present but undermined: competing goals with no priority, or wording loose enough that readers describe the output differently.
The primary task is identifiable and readers broadly agree, but a gap remains, often a definition of success left implicit.
The objective is clear and the result is easy to describe, with only minor ambiguity. Any competing goals are prioritized.
The objective is explicit, one primary task is unmistakable, and success is defined. There is no room for two readers to disagree.
The full framework, evaluator guide, and scoring rubrics are shared on request, not published. If you would like to review them, send a short note with who you are and what you would use them for.
Design question: Can the framework run as a consistent, evidence-based tool, fast enough to keep up with the Copilot?
A framework is only a claim until it's tested. Phase 2 tested it.
I built the evaluator as an AI agent on two platforms, Claude and ChatGPT, using the same framework and the same knowledge base. Checking every generated prompt by hand doesn't scale, so it had to run as a tool anyway; building it twice turned that tool into a test.
If the framework is sound, both agents should score a given prompt the same way. Agreement would confirm the tool is valid.
I ran the same prompts through both agents and compared their evaluations.
They didn't agree.
Identifiable primary task, under-defined success.
The declared "AI's role: analyze student work" conflicts with the requested output.
Primary objective is clear; "redesign the task" leaves success somewhat implicit.
Treated the workflow sentence as context, so it saw no contradiction.
Either:
1) my framework and instructions aren't clear enough for two AIs to interpret them the same way; or
2) the observable indicators for some constructs aren't as unambiguous as they need to be.
I analysed both evaluations against each other. A few things stood out:
Both hit the clause "AI's role: analyze student work." Claude read it as an instruction that clashed with the requested draft and raised an Internal Consistency flag; ChatGPT read it as background context and raised none. That single split is what drove Fair versus Good.
On ambiguous points one agent tended to score 3 and the other 4, consistently. That's a severity difference pointing at the rubric, not at either model.
One let the flag override a mid-average and returned "Fair"; the other let the average lead and returned "Good." Both defensible, because I'd never specified which wins.
Design question: Does the prompt actually achieve the educator's goal in practice?