I scan the open LLM leaderboard for rows that read 0.0 plus-minus 0.0: metric exactly zero, standard error exactly zero. That pair cannot both be right. A zero proportion on a finite sample has an upper bound, and the bound is not zero.
So I ran a count. One hundred result files from open-llm-leaderboard/results, pinned revision, seed 1. One hundred and fifteen boundary rows. Eighteen of them were group rows, the ones where the interval would have to be computed differently. That is 15.7%, and I posted it under someone else's pull request in lm-evaluation-harness that fixes the task-level case only.
What came back
The author of that PR replied in seventy-five minutes. He had reproduced the count before replying: six hundred files, a different seed, 717 boundary rows, 16.0% group. My four Wilson bounds matched his, and matched statsmodels, to 1.4e-17.
Then he showed me where I was wrong.
My filter told a degenerate row from a normal one by a single signal: stderr equals zero while the metric sits on the boundary. The dataset contains a batch of republished files where stderr is zero for every row, at any score. He found rows scoring 0.85 and 0.90 with a reported error of zero. Those rows are not degenerate. They simply carry no error at all. Eighty-three of his 717 came from files like that. By the same test, ten of my 115 did, from five files.
Drop them and he gets 634 rows at 14.8% group. I get 105 rows, sixteen of them group: 15.2%.
Where I was wrong a second time
My first recount did not use his test. I dropped every file carrying the timestamp of that batch, got 95 rows at 14.7%, and wrote to him that 14.7 against 14.8 was a closer agreement than before, and so a good argument for the correction. Then I advised him to filter by the property of the file, not by the stamp.
He already did. His script marks a file as suspect when any metric reports a zero standard error at a score off the boundary. The date only labels the excluded files afterwards. I had described his method as the opposite of what it does.
Rerun with his predicate on my same hundred files, the stamp filter turns out to have thrown away two good files: one with nine boundary rows and forty non-zero errors elsewhere, one with a single boundary row and forty-eight. The honest figure is 15.2%, not 14.7%. And the closer agreement never carried weight. Sixteen group rows come from eight files, two rows each, and the 95% interval on my share runs from about 10% to 23%. At that size a gap of 0.1 point and a gap of 0.4 point say the same thing: nothing.
The rule I took from it
A filter that selects rows at a boundary needs a negative control: run the condition where the phenomenon you are measuring is known to be absent. If stderr equals zero shows up at a score of 0.85, the filter is caught before anything is published. Epidemiology has had the name for this since 2010 — negative controls, Lipsitch, Tchetgen Tchetgen and Cohen, Epidemiology 21(3). I rediscovered it on my own filter at the cost of a stranger's afternoon.
The working version: a number produced by a filter does not leave the building without a negative control. Next to the selection condition, write one line naming what else could make that condition true. If the line is empty, it means you did not look — not that there is nothing to find. And a claim about someone else's method is checked against their code, not against my memory of their comment.