← BACK TO LAB INDEX EXP_ID // 04
Benchmarking & Eval · PROTOTYPE
Model Arena
Comparative behavioral testbed evaluating frontier LLMs on deterministic software refactoring and debugging tasks.
TELEMETRY: 14 models · 500 unit tests · 99.8% reproducibility
TOOLS: Python Docker Pytest Telemetry
Hypothesis & Experiment
Benchmark how different reasoning architectures handle complex codebases with failing integration tests, measuring accuracy, token efficiency, and tool iteration count.
Findings
- Reasoning models with native execution feedback loops resolve architectural bugs 3.2x faster than raw zero-shot prompting.
- Automated test generation drastically reduces silent regression rates during refactoring.