Chrysalis Labs
Datasets
Agentic coding Private

SWE Sweep

Find and fix the real bugs in an old release of an open-source project. No bug reports, no failing tests.

121repositories
1,584bugs
8median bugs per repo
12 htime limit

Task format

Given
A repository at an old release, with real bugs that upstream fixed later.
Submit
Edits to the source. The tests for each bug are hidden.
Scoring
Share of bugs fixed. Breaking existing tests can cost credit.

Composition

Repositories by language
  • Go58
  • Python38
  • JS / TS21
  • Java4
Bugs per repository
  • 291–4
  • 375–9
  • 3210–19
  • 1820–39
  • 540+

bugs per repository

How it is built

Bugs
Real upstream fixes made after the pinned release. Each fix's tests become a hidden check.
Offline
The agent can only reach its model API. Newer copies of the project are removed from the image.
Grading
A fresh copy of the image applies only source edits. Edits to tests, build files and binaries are dropped.
Regressions
71,814 existing tests are enforced. Breaking one withholds credit for bugs in the same package.
Validation
The gold fixes score 1.0 and an empty patch scores 0 on every repository.

Model results

LeaderboardMean share of bugs fixed per attempt
  • Codex gpt-6-astraCodex CLI · 171 attempts0.40
  • Claude Opus 5.5Claude Code · 122 attempts0.09
0.28
median best per repo
14
repos never solved
12
repos fully solved

Baseline runs were capped at 2 hours. The longest attempt took 83 minutes, so the cap never applied. 18 Opus attempts that failed on API rate limits are excluded.

Score by repository sizeMean share of bugs fixed, by how many bugs the repository has
  • 0.610.181–4 bugs
  • 0.400.075–9
  • 0.290.1010–19
  • 0.190.0320+
Score distributionShare of attempts in each score band
  • 9%49%0
  • 8%27%< 10%
  • 19%14%10–24%
  • 26%5%25–49%
  • 22%3%50–74%
  • 5%0%75–99%
  • 12%2%100%

share of a repository's bugs fixed in one attempt

Per attemptTime and cost are medians
CodexOpus
Fixes at least one bug91%51%
Bugs fixed492 of 1,81795 of 2,184
Time4.6 min12.6 min
Cost$2.01$5.55
Longest attempt12 min83 min
Head to headOne dot per repository both models attempted
Codex aheadOpus ahead000.250.250.50.50.750.7511Claude Opus 5.5 →Codex gpt-6-astra →
68
Codex ahead
5
Opus ahead
17
tied
Notes
  • 79% of Codex attempts earn partial credit, so the reward separates good runs from great ones.
  • Both models score lower as repositories get bigger.
  • Both models quit long before the 12-hour limit: Codex after a median 4.6 minutes, Opus after 12.6. The bugs they miss are ones they never found, not ones they ran out of time on.
  • Opus tried to download newer releases in over half its attempts. The sandbox blocked every request.
Sandbox
8 vCPU, 32 GB RAM, offline
Image
One prebuilt image per repository