Results and retractions
The first headline number was wrong, and we said so.
MMR-Bench's earliest result looked excellent. It was a data leak. Retracting it is the most load-bearing thing in this project's history, so it stays on the record.
v1.0 regression
Run on : 69 passed, 2 warnings. The warnings are pre-existing PyTorch Transformer nested-tensor warnings inside archived v0.2 model tests.
Reproduced on on a clean runner from a fresh checkout: 69 passed, 2 warnings, with certificate verification and replay both green.
The v1.0 experiment plan ran twelve steps in order — from replacing unsafe evaluation, through the semantic corrections, to the certificate trials.
- Replace unsafe general Python evaluation with an AST whitelist.
- Correct
^,ROUND, and 1900 / 1904 date semantics. - Re-run all v0.1–v0.9 tests.
- Run 30 randomized input mutations.
- Run 20 exponent trials.
- Run 20 rounding trials.
- Run 20 1904-date trials.
- Sign four representative XLSX audits.
- Verify signatures and exact replay.
- Mutate ten certificate payloads and require rejection.
- Replay ten certificates against incorrect source files and require rejection.
- Confirm that no certificate authorizes a formula write.
Fixture results
Eleven fixtures, each built to produce one specific relation between the three sources. The bridge column counts formulas the external engine could not read until structured references were expanded.
| Fixture | Loads directly | Bridged | Outcome |
|---|---|---|---|
| Correct structured XLSX | no | 12 | 51 three-way agreements |
| Names-only normalized XLSX | yes | 0 | 51 three-way agreements |
| Stale cache | no | 12 | 49 agreements, 2 stale caches |
| Formula drift | no | 12 | 45 agreements, 6 stale dependent caches |
| Stale safe copy | no | 12 | 49 agreements, 2 caches absent by policy |
| Drift safe copy | no | 12 | 45 agreements, 6 caches absent by policy |
2^3 conflict | yes | 0 | cache and external 8, MMR 1 |
ROUND | yes | 0 | external-only supported result |
| 1900 date | yes | 0 | external and cache agree |
| 1904 date | yes | 0 | external result differs by 1,462 days |
1/0 | yes | 0 | spreadsheet error agreement |
The 2^3 row is the defect that produced the v1.0 evaluator rewrite: Excel's ^ was reaching Python as XOR, so =2^3 returned 1 where the workbook and the external engine both said 8. It was found by the protocol rather than by reading the code, which is the point of holding three readings instead of one.
ROUND now follows Excel's half-away-from-zero rule, so ROUND(2.675, 2) is 2.68. The 1904 gap is constant at 1,462 days and belongs to the external engine, not to the workbook — the protocol records it as an engine limitation and forbids a write.
Across thirty input-mutation workbooks, MMR and the external engine agreed on 100% of formulas, producing zero review candidates from engine disagreement.
The v0.1 retraction
The first release reported 100% standard accuracy, collapsing to roughly 44.5% under a 90° rotation test. That is not a model result. The data generator had fixed right and down to increasing directions and left and up to decreasing ones, so the direction token alone answered the question — the model never had to read the matrix.
The fix was to randomize path polarity and build paired queries over the same matrix: (X, d, y) together with (X, opp(d), 1−y). Direction stops being sufficient; the marked path has to be read.
The corrected calibration run, on 8×8 matrices with a vocabulary of 16 and path length 5:
| Model | Params | Standard | Rotated 90° | Path hit rate |
|---|---|---|---|---|
| Row-major, no direction | 16,186 | 50.00% | 50.00% | 22.25% |
| Row-major + direction token | 16,186 | 59.00% | 54.00% | 22.25% |
| MMR, shared four directions | 15,459 | 83.50% | 79.25% | 100.00% |
| 2D CNN | 26,378 | 54.50% | 48.00% | 22.25% |
Against the strongest baseline, that is +24.5 points standard and +25.25 points rotated, at slightly fewer parameters. The path hit rate is the more interesting column: only MMR located the marked path at all.
On this synthetic directional-order task, a shared multi-directional traversal reads structure that a linearized sequence does not.
Real Excel workbooks, or general AI tasks. It is a calibration on generated data.
Safe repair policy
From v0.4 onward the repair thresholds have been explicit numbers rather than a judgement call.
- Candidate
- Emitted when the structural anomaly score is at least 0.31.
- Automatic repair
- Only when the anomaly score is at least 0.40 and structural repair confidence is at least 0.65.
- Everything else
- Goes to human review.
- Source worksheet
- Never overwritten. Repairs are written to a copy.
- Model score
- Supporting evidence, never sole authorization.
These thresholds were evaluated only on the v0.4 synthetic distribution. They are not certified thresholds for arbitrary production workbooks.
Falsification criteria
The v0.5 experiment plan fixed in advance what would count as failure. The method is not considered generally solved if:
- Clustered errors form a competing local majority.
- Style removal produces false repairs.
- Anonymization changes structural decisions.
- Clean workbooks are modified.
- Exact repair cannot be reconstructed from peer formulas.