Five On It

Appearance

five-on-it / apps/creator/src/eval

apps/creator/src/eval ​

npm run creator:eval: the creator's evals (evals.ts) run by hand against each candidate model on OpenRouter or Anthropic's API, never in CI. It refuses to start without an eval key of its own for each (OPENROUTER_EVAL_KEY, never the live platform's OPENROUTER_API_KEY; ANTHROPIC_EVAL_KEY for the claude-… candidates, never the live ANTHROPIC_API_KEY), prints what it could spend at most first, and spends it only with --yes. The candidates are FOI_EVAL_MODELS (comma-separated, the bands' ids by default), each priced by OpenRouter's models endpoint or at Anthropic's list prices; the briefs are all twenty (--briefs N: the first N). The platform runs in this process on the bundled levels, regions generated here and towns where Blender is found, the word list moderating; the report goes to --out (reports/creator-eval, ignored by git, by default) as Markdown and JSON. --fake runs the scripted model instead, for a dry run at no cost.

console
OPENROUTER_EVAL_KEY=sk-or-… npm run creator:eval -- --briefs 4        # what it would spend
OPENROUTER_EVAL_KEY=sk-or-… npm run creator:eval -- --yes             # all of it
ANTHROPIC_EVAL_KEY=sk-ant-… FOI_EVAL_MODELS=claude-haiku-5-5,claude-sonnet-5-5 npm run creator:eval -- --yes
npm run creator:eval -- --fake --briefs 2                             # the scripted model