Kimi K3DIFFno longer counts
38 of 150 differed (fails from 36) · 75 of 300 differed over 2 sets (fails from 52) · the real model differs from itself on 8%
Where it missed
Chemistry (melting points)7/13
Programming (release years)9/21
Medical (drug half-lives)8/22
Biology (chromosome numbers)8/24
For example
Lycopodium clavatum diploid chromosome countreal 68 · provider 156
Josephson constant in gigahertz per voltreal 483,598 · provider 483.6
Diploid chromosome number (2n) of *Angiopteris evecta*real 80 · provider 156
🧠Kept reasoning when asked not to: about 72 reasoning tokens on each of 4 ways to switch it off.
GPT-6.1 SolDIFFno longer counts
20 of 100 differed (fails from 12) · 19 of 100 differed (fails from 12) · the real model differs from itself on 0%
Where it missed
Medical (drug half-lives)10/23
Mathematics (special values)4/10
Programming (release years)5/16
Chinese internet (platform dates)5/21
For example
FIPS 205 SLH DSA SHA2 128f FORS tree height edgesreal 6 · provider 12
Sp 1 central binary published orbital period (days)real 2.906 · provider 0.2658
brivaracetam terminal elimination half life hoursreal 9 · provider 1
✓Reasoning switched off with disable thinking, as the real model was asked.
GLM 5.3DIFFno longer counts
58 of 129 differed (fails from 33) · 40 of 130 differed (fails from 39) · the real model differs from itself on 8%
Where it missed
Chemistry (melting points)8/16
Chinese history (specific dates)6/12
Medical (drug half-lives)9/19
Mathematics (special values)4/9
For example
year ip rfc 791 header min bytesreal 20 · provider 1,981
safer sk 128 number of roundsreal 10 · provider 40
diameter of asteroid 4179 Toutatis in kilometersreal 2.5 · provider 5
✓Reasoning switched off with disable thinking, as the real model was asked.
Claude Fable 5.1DIFFno longer counts
53 of 110 differed (fails from 25) · 32 of 100 differed (fails from 14) · the real model differs from itself on 5%
Where it missed
Astronomy (periods and distances)5/8
Chinese literature (publication years)8/15
Chemistry (melting points)10/19
Programming (release years)15/30
For example
number of chapters in the first edition of dune by frank herbert book one count of booksreal 3 · provider 22
Number of sporadic (non-regular) irregular primes below 100real 3 · provider 17
Pegvaliase-pqpz (Palynziq) mean elimination half-life at steady state, hoursreal 47 · provider 200
✓Reasoning switched off with disable thinking, as the real model was asked.
Gemini 3.1 Pro PreviewDIFFno longer counts
48 of 110 differed (fails from 25) · 31 of 120 differed (fails from 28) · the real model differs from itself on 5%
Where it missed
Chemistry (boiling points)7/11
Programming (release years)11/24
Chemistry (melting points)5/12
Chinese geography (measurements)7/17
For example
IC 1101 Galaxy diameter in millions of light yearsreal 6 · provider 2,000
bis(ethylenediamine)copper(II) sulfate melting point (C)real 250 · provider 1,200
Total number of weeks the 1979 film 'Apocalypse Now' was in productionreal 68 · provider 238
✓Reasoning switched off with disable thinking, as the real model was asked.
Kimi K3.1 ProDIFFno longer counts
24 of 110 differed (fails from 19) · 44 of 220 differed over 2 sets (fails from 42) · the real model differs from itself on 3%
Where it missed
Biology (chromosome numbers)10/23
Chemistry (boiling points)3/7
Chemistry (melting points)5/14
Chinese history (specific dates)3/10
For example
Diploid chromosome number of Huperzia serrata (chromosomes per somatic cell, 2n)real 268 · provider 68
Diploid chromosome count of Cibotium barometz (2n; chromosomes per somatic cell)real 136 · provider 68
Ex Machina principal photography duration in weeksreal 4 · provider 6
✓Reasoning switched off with reasoning effort minimal, as the real model was asked.
Grok 4.7DIFFno longer counts
45 of 100 differed (fails from 12) · 18 of 100 differed (fails from 12) · the real model differs from itself on 0%
Where it missed
Chinese history (specific dates)6/10
Mathematics (special values)6/10
Chinese literature (publication years)5/10
Cryptography (parameters)5/12
For example
irregular prime count below 100real 3 · provider 11
count of irregular primes below 100real 3 · provider 11
Hymenophyllum tunbrigense sporophyte diploid chromosome number 2nreal 26 · provider 72
✓Reasoning switched off with disable thinking, as the real model was asked.
GPT-6 AstraDIFFno longer counts
44 of 110 differed (fails from 17) · 36 of 110 differed (fails from 25) · the real model differs from itself on 2%
Where it missed
Physics (material properties)8/11
Pop culture (box office and ratings)8/13
Programming (release years)16/36
Medical (drug half-lives)9/21
For example
SCHOTT N-BK7 Sellmeier C2 squared-resonance-wavelength coefficient, catalog dispersion formula (10^-12 m^2)real 2.002e-14 · provider 0.02002
SCHOTT N-SF11 catalog Sellmeier C1 (10^-12 m^2)real 1.319e-14 · provider 0.01319
Fused silica Sellmeier third resonance wavelength, Malitson 1965 fit at 293.15 K (10^-6 m)real 0.000009896 · provider 9.896
🧠Kept reasoning when asked not to: about 46 reasoning tokens on each of 4 ways to switch it off.
Gemini 4 ArgonDIFFno longer counts
19 of 100 differed (fails from 12) · 25 of 100 differed (fails from 14) · the real model differs from itself on 1%
Where it missed
Biology (chromosome numbers)18/30
Programming (release years)7/15
Chinese history (specific dates)5/13
Chinese literature (publication years)2/11
For example
diploid chromosome number of Hymenophyllum tunbrigensereal 26 · provider 72
Diploid chromosome number of Pilularia globulifera (chromosomes)real 26 · provider 52
diploid chromosome number of Tmesipteris oblanceolatareal 416 · provider 104
🧠Kept reasoning when asked not to: about 56 reasoning tokens on each of 4 ways to switch it off.
DeepSeek V4 ProDIFFno longer counts
33 of 140 differed (fails from 33) · 35 of 120 differed (fails from 28) · the real model differs from itself on 7%
Where it missed
Chemistry (boiling points)4/7
Chemistry (melting points)3/6
Mathematics (special values)5/11
Physics (material properties)4/9
For example
Number of known Pierpont primes with 10^6 digitsreal 0 · provider 1
Barium titanate coercive field at 300 Kreal 100,000 · provider 200,000
Number of hidden cuts in the film "Birdman" (2014) as stated by the editorreal 100 · provider 16
🧠Kept reasoning when asked not to: about 75 reasoning tokens on each of 4 ways to switch it off.
Claude Opus 5.5DIFFno longer counts
57 of 88 differed (fails from 8) · 57 of 100 differed (fails from 11) · the real model differs from itself on 1%
Where it missed
Physics (material properties)3/3
Cryptography (parameters)1/1
Biology (chromosome numbers)29/31
Chinese history (specific dates)6/8
For example
lasmiditan elimination half life hoursreal 5.7 · provider 17.7
yixing nangong yue meridian arc survey year cereal 724 · provider 1,715
Number of chapters (hui) in the Woxian caotang edition of Rulin waishireal 56 · provider 120
🧠Kept reasoning when asked not to: about 88 reasoning tokens on each of 4 ways to switch it off.
Claude Opus 5.5DIFFbefore points
37 of 110 differed (fails from 16) · the real model differs from itself on 5%
Where it missed
Chemistry (melting points)4/6
Physics (material properties)2/3
Medical (drug half-lives)10/17
Biology (chromosome numbers)15/28
For example
myrmecia croslandi female diploid chromosome countreal 2 · provider 24
neutron gyromagnetic ratio in units of 1e8 s^-1 T^-1 CODATA2018real 1.833 · provider -1.832
salvinia cucullata somatic chromosome number 2nreal 18 · provider 45