diff-bwm-1: A Benchmark for Agents That Plan With Models
We are releasing diff-bwm-1, a six-eval benchmark that tests whether an LLM planning agent that can run a numerical model of its domain is consistent and can test a draft before shipping it.
Does an AI planning agent behave more like a programmer with code? A programmer runs the code, sees it fail, and fixes it before shipping. Can a planning agent do the same with a numerical model of its business?
Today we are releasing diff-bwm-1, a six-eval suite that tests exactly that. Each eval puts a live coding agent in a business-planning sandbox and asks it to propose a plan, then say whether it succeeded. A scorer checks the plan, and checks whether the agent's claim of success was true.
The six evals escalate from one company to coupled portfolios of business units that share scarce resources, to agents building their own models, to a third-party due-diligence spreadsheet with planted contradictions, to prose consulting deliverables, and finally to cases where the causal information cannot be recovered from history. The hard part is staying consistent under coupling, contradiction, or missing information, and not claiming success when the scorer would fail you.
The package includes the problems, the answers, and the code that stages, runs, scores, audits and reports runs. Code is MIT licensed and the scenarios and datasets are CC BY 4.0.
Stay tuned for our forthcoming preprint paper on the subject.