Description
Coding models keep scoring higher on old benchmarks they may have trained on. SWE-bench-Live from Microsoft is a continuously updated software engineering benchmark collecting fresh tasks from real repos monthly.
It builds reproducible environments automatically, reducing contamination for a truer measure of fixing real issues.
Live:Monthly tasks.
Real issues:From GitHub.
Fresh:Less contamination.
Reproducible:Auto environments.
It builds reproducible environments automatically, reducing contamination for a truer measure of fixing real issues.
Features
Live:Monthly tasks.
Real issues:From GitHub.
Fresh:Less contamination.
Reproducible:Auto environments.
