Evaluation workbench
A sample project brief for comparing model outputs against a fixed set of cases.
This is an illustrative project brief. No application or source repository is available.
The question
How can a team compare changes to a prompt without relying on a few remembered examples?
Proposed scope
A small local tool would load a fixed set of inputs, run each candidate, and save the outputs beside the configuration used to produce them. A review view would place results next to each other.
Design decisions to test
- Start with local files so each test case can be reviewed in version control.
- Record provider, model, prompt, and run time for each result.
- Keep human review separate from automated checks.
- Avoid a single score when errors have different costs.
Open question
Which failures should stop a release? That requires real cases and an agreed review process before the tool can provide useful evidence.