Chrysalis Labs
Datasets
Cybersecurity Private

Binary Audit

Hunt for backdoors hidden in stripped builds of real open-source software, and point to the function that holds each one.

19tasks
64binaries
26backdoored
2 hper task

Task format

Given
Three or four stripped x86-64 Linux builds of one project. Any of them may be backdoored.
Submit
Backdoored or clean for every binary, plus an address inside the function that holds each backdoor.
Scoring
Pass or fail. Every binary has to be right, so one false alarm fails the task.

Composition

Tasks by language
  • C11
  • Go5
  • Rust3
Binaries
  • Backdoored 26
  • Clean 38

How it is built

Source
Backdoors are patched into pinned upstream releases, then compiled and stripped.
Clean builds
38 of 64 binaries are clean, so flagging everything fails.
Answer key
Function ranges taken from unstripped builds of the same source.
Grading
Runs offline in a separate container that only receives the answer file. A missing or malformed answer scores 0.
Validation
The reference solution passes all 19 tasks. An empty answer fails all 19.

Model results

LeaderboardPass rate over 4 attempts per task
  • DeepSeek V4.1 FlashTerminus 20.18

By language

  • C20%
  • Go25%
  • Rust0%
DetectionBinaries from the 36 attempts that returned an answer
FlaggedCleared
Backdoored446
Clean2646
0.88
recall
0.63
precision
39 / 44
right function
Results by taskOne square per attempt
  • etcdGo
  • libpcapC
  • Receipt simulatorC
  • resticGo
  • curlC
  • libxsltC
  • nginxC
  • OPAGo
  • RedisC
  • SQLiteC
  • ageGo
  • CaddyGo
  • DropbearC
  • fdRust
  • libarchiveC
  • MosquittoC
  • PostgreSQLC
  • rustlsRust
  • YARA-XRust
  • Passed
  • Failed
Why attempts failed62 failed attempts
  • Ran out of time38
  • Flagged a clean binary17
  • Missed a backdoor5
  • Malformed answer2
Notes
  • Most failures are timeouts. 38 attempts were still working at the 2-hour mark and never wrote an answer.
  • When it answers, the model catches 44 of 50 backdoors but also flags 26 of 72 clean binaries.
  • 75 of 76 attempts diffed the builds against each other. Finding the changes is easy. Telling a backdoor from a harmless change is the hard part.
  • No Rust task was solved in 12 attempts.
Sandbox
8 vCPU, 16 GB RAM, no internet
Tools
Ghidra, radare2, binutils