five-on-it / apps/creator/src/evals
apps/creator/src/evals
The creator's evals (TECHNICAL_SPECS §9.8): twenty-three fixed briefs (three of them cities, two of them conversations: the owner says more after the model's first answer), each made by each candidate model the way a runner makes it (runJob, turn after turn, on the platform in this process, local-platform.ts), and measured: whether it built without an error with the missions the brief asked for, whether moderation let its texts through, its steps (the model's calls), what it cost and how long it took. A model meets the targets of each tier it was run on (EVAL_TARGET: nine creations in ten built, at ten cents a creation or less on average; EVAL_CITY_TARGET for cities). The report is Markdown and JSON (reportMarkdown); eval.ts runs the evals by hand (npm run creator:eval), never in CI, on an eval key of their own.