Agentic coding
Private
SWE Sweep
Find and fix the real bugs in an old release of an open-source project. No bug reports, no failing tests.
121repositories
1,584bugs
8median bugs per repo
12 htime limit
Task format
- Given
- A repository at an old release, with real bugs that upstream fixed later.
- Submit
- Edits to the source. The tests for each bug are hidden.
- Scoring
- Share of bugs fixed. Breaking existing tests can cost credit.
Composition
- 291–4
- 375–9
- 3210–19
- 1820–39
- 540+
bugs per repository
How it is built
- Bugs
- Real upstream fixes made after the pinned release. Each fix's tests become a hidden check.
- Offline
- The agent can only reach its model API. Newer copies of the project are removed from the image.
- Grading
- A fresh copy of the image applies only source edits. Edits to tests, build files and binaries are dropped.
- Regressions
- 71,814 existing tests are enforced. Breaking one withholds credit for bugs in the same package.
- Validation
- The gold fixes score 1.0 and an empty patch scores 0 on every repository.
Model results
Codex gpt-6-astraCodex CLI · 171 attempts0.40
Claude Opus 5.5Claude Code · 122 attempts0.09
- 0.28
- median best per repo
- 14
- repos never solved
- 12
- repos fully solved
Baseline runs were capped at 2 hours. The longest attempt took 83 minutes, so the cap never applied. 18 Opus attempts that failed on API rate limits are excluded.
- 0.610.181–4 bugs
- 0.400.075–9
- 0.290.1010–19
- 0.190.0320+
- 9%49%0
- 8%27%< 10%
- 19%14%10–24%
- 26%5%25–49%
- 22%3%50–74%
- 5%0%75–99%
- 12%2%100%
share of a repository's bugs fixed in one attempt
| Fixes at least one bug | 91% | 51% |
|---|---|---|
| Bugs fixed | 492 of 1,817 | 95 of 2,184 |
| Time | 4.6 min | 12.6 min |
| Cost | $2.01 | $5.55 |
| Longest attempt | 12 min | 83 min |
- 68
- Codex ahead
- 5
- Opus ahead
- 17
- tied
- 79% of Codex attempts earn partial credit, so the reward separates good runs from great ones.
- Both models score lower as repositories get bigger.
- Both models quit long before the 12-hour limit: Codex after a median 4.6 minutes, Opus after 12.6. The bugs they miss are ones they never found, not ones they ran out of time on.
- Opus tried to download newer releases in over half its attempts. The sandbox blocked every request.
- Sandbox
- 8 vCPU, 32 GB RAM, offline
- Image
- One prebuilt image per repository