Programs rebuilt without seeing the code

[ Front page ] [ The edge ] [ Finished ] [ About ] [ Links ]

77.4 per cent

Held by Claude Opus 5.5. Set 2026-09-22, and has stood for 35 days. Written down here 2026-10-02.

Two numbers on this page ask a machine to write software, and the distance between them is the point of having both. The other one hands a model a repository and a bug report: the code is all there, the change is small, and the project's own tests say when to stop. This one deletes the code. The model may run the original program as often as it likes and watch what comes out, and it may read the documentation, and that is all - no source, no internet, no borrowing the real binary, which is deleted before the marking starts. From that it has to build the whole program again, twice: once in a mainstream language and once in Ada, because a language with little training data behind it is how you tell construction from recall. The marking is the unforgiving part. There is no partial credit and no nearly. The rebuilt program must emit the same bytes as the original on every one of hundreds to thousands of tests, a third of which it has never seen, and a single differing character scores the attempt zero. The targets are real working software - Unix utilities, query tools, interpreters, compression, cryptography, bioinformatics - and the hard ones are given ten billion tokens and seven days. Six weeks before this page was written the best model in the world rebuilt fewer than one program in ten. It now rebuilds three in four. That is the steepest climb anything on this site has recorded, and it is also the likeliest to end here as a retirement rather than a record, because a test with no partial credit does not level off, it finishes. The exercise is older than any of this and has a name: clean-room reimplementation, the method Phoenix used on the IBM PC BIOS in 1984, where one team was allowed to watch the thing run and write down what it did, and a second team that was never permitted to see the original wrote the replacement from that description alone. What took two rooms of engineers and a year of lawyers now has a token budget.

That description was written when this frontier was opened, on 2026-10-02, and is never rewritten. Where it and the number above it disagree, the number is the reading and the description is history.

What counts

The best score any model has recorded on MirrorCode, the long-horizon coding benchmark Epoch AI built with METR, as published in Epoch AI's own benchmark file. The figure is the one the publisher calls the leaderboard configuration, MirrorCode (ML, +Private, 2L): fifteen target programs from the Medium and Large buckets, each reimplemented in two languages, one mainstream and generally Go and one low-resource and generally Ada, which is thirty tasks, each attempted three times with ten billion tokens and seven days per attempt. A task counts only on a hundred-per-cent solve - every end-to-end test passing, where passing means the rebuilt program writes exactly the same bytes to stdout and stderr as the original - and the score is the model's average rate of those solves across the fifteen programs. The configuration is named here instead of a version string for a reason that is the reader's business rather than ours: the publisher says in writing that these scores are not comparable with the ones in its own paper, which ran all twenty-five programs under a different budget, and the file's task-version column is filled on one row of nine. The configuration is the thing that is stable, so the configuration is what this counts.

What moves it

A model records a higher score on the same configuration and Epoch AI publishes it.

Where to check it: https://epoch.ai/benchmarks/mirrorcode

The program reads it from https://epoch.ai/data/benchmarks.csv, which is the same figure in a form a script can parse.

Last checked: 2026-10-24. The source itself last measured this: 2026-10-14.

Previous: Assembly mistakes spotted in a photograph   Next: The most ordinary work one computer can do

[ Front page ] [ The edge ] [ Finished ] [ About ] [ Links ]