Cybersecurity
Private
Binary Audit
Hunt for backdoors hidden in stripped builds of real open-source software, and point to the function that holds each one.
19tasks
64binaries
26backdoored
2 hper task
Task format
- Given
- Three or four stripped x86-64 Linux builds of one project. Any of them may be backdoored.
- Submit
- Backdoored or clean for every binary, plus an address inside the function that holds each backdoor.
- Scoring
- Pass or fail. Every binary has to be right, so one false alarm fails the task.
Composition
- Backdoored 26
- Clean 38
How it is built
- Source
- Backdoors are patched into pinned upstream releases, then compiled and stripped.
- Clean builds
- 38 of 64 binaries are clean, so flagging everything fails.
- Answer key
- Function ranges taken from unstripped builds of the same source.
- Grading
- Runs offline in a separate container that only receives the answer file. A missing or malformed answer scores 0.
- Validation
- The reference solution passes all 19 tasks. An empty answer fails all 19.
Model results
DeepSeek V4.1 FlashTerminus 20.18
By language
FlaggedCleared
Backdoored446
Clean2646
- 0.88
- recall
- 0.63
- precision
- 39 / 44
- right function
- etcdGo
- libpcapC
- Receipt simulatorC
- resticGo
- curlC
- libxsltC
- nginxC
- OPAGo
- RedisC
- SQLiteC
- ageGo
- CaddyGo
- DropbearC
- fdRust
- libarchiveC
- MosquittoC
- PostgreSQLC
- rustlsRust
- YARA-XRust
- Passed
- Failed
- Most failures are timeouts. 38 attempts were still working at the 2-hour mark and never wrote an answer.
- When it answers, the model catches 44 of 50 backdoors but also flags 26 of 72 clean binaries.
- 75 of 76 attempts diffed the builds against each other. Finding the changes is easy. Telling a backdoor from a harmless change is the hard part.
- No Rust task was solved in 12 attempts.
- Sandbox
- 8 vCPU, 16 GB RAM, no internet
- Tools
- Ghidra, radare2, binutils