five-on-it / apps/creator/src/evals
apps/creator/src/evals
The creator's evals (TECHNICAL_SPECS §9.8): twenty fixed briefs, each made by each candidate model the way a runner makes it (runJob, on the platform in this process, local-platform.ts), and measured: whether it built without an error with the missions the brief asked for, whether moderation let its texts through, its steps (the model's calls), what it cost and how long it took. A model meets the target (EVAL_TARGET) with about nine creations in ten built, at five cents a creation or less on average. The report is Markdown and JSON (reportMarkdown); eval.ts runs the evals by hand (npm run creator:eval), never in CI, on an eval key of their own.