Evaluate the change in its environment.
Repository context, acceptance tests and expected-change evidence test whether an agent can make a coherent change beyond a snippet.
Discuss a similar programmeIsolated snippets miss dependencies, file navigation and coherent changes.
The hard problem
An isolated code snippet could not test whether an agent could navigate files, understand dependencies and make a coherent repository change. The evaluation needed the environment and the expected behaviour.
Our advantage
We constructed context-rich tasks, identified the relevant artefacts, designed acceptance tests, reviewed patches and captured failure taxonomies. Expected-change evidence and expert critiques made the evaluation outcome reviewable at repository level.
Value delivered
Deliverables included repository task packs, environment instructions, test criteria, expert critiques and adjudicated outcomes.
Platform context.
Bring the next
requirement.