OpenBench Testing Framework

Active : 3 Machines / 6 Threads / 0.04 MNPS
Priority 0
CremagLeela-lc5lc5-implementationdiff8.0+0.08LLR: 1.56 (-2.94, 2.94) [-3.00, 0.00]
Games: 9728 W: 2454 L: 2392 D: 4882
Ptnml(0-2): 3, 925, 2948, 983, 5
lc5: VisitLimiter/MaterializationTable/VisitWorker/Evaluator cleanup and ensure root is expanded before stopping vs 08c99639
Finished
CremagLeela-lc5[Leela] lc5-implementationdiff8.0+0.08Elo: -13.90 +- 13.76 (95%) [N=500]
Games: 500 W: 112 L: 132 D: 256
Ptnml(0-2): 0, 61, 148, 41, 0
lc5 vs classic (500 games)
CremagLeela-lc5lc5-implementationdiff8.0+0.08LLR: 2.95 (-2.94, 2.94) [-3.00, 0.00]
Games: 28878 W: 7218 L: 7120 D: 14540
Ptnml(0-2): 20, 2834, 8624, 2950, 11
lc5: SearchSession refactor (13 commits: slot mutex removal, local wakeups, eviction/starvation fix, TicketTable/SessionGate/VisitWorker/Evaluator extraction) vs 1d0e209a
CremagLeela-lc5lc5-implementationdiff8.0+0.08LLR: -2.58 (-2.94, 2.94) [0.00, 3.00]
Games: 26750 W: 6640 L: 6724 D: 13386
Ptnml(0-2): 28, 2746, 7903, 2678, 20
lc5 step 14: StickyEndgames=true vs false
CremagLeela-lc5lc5-implementationdiff8.0+0.08LLR: -0.00 (-2.94, 2.94) [0.00, 3.00]
Games: 28104 W: 6924 L: 6869 D: 14311
Ptnml(0-2): 13, 2831, 8313, 2878, 17
lc5 step 15: TwoFoldDraws=contained vs root
CremagLeela-lc5lc5-implementationdiff8.0+0.08LLR: 0.13 (-2.94, 2.94) [-3.00, 0.00]
Games: 16488 W: 4153 L: 4178 D: 8157
Ptnml(0-2): 5, 1618, 5027, 1585, 9
lc5 NodeStore stores node statistics (63951bd4) vs 0a668749; default Lc5GraphNodeLimit=0, so no eviction: checks the default path is unchanged
CremagLeela-lc5lc5-implementationdiff8.0+0.08LLR: -2.96 (-2.94, 2.94) [0.00, 3.00]
Games: 21798 W: 5401 L: 5514 D: 10883
Ptnml(0-2): 13, 2255, 6485, 2124, 22
lc5 step 16: Lc5Policy=puct-dag vs puct
CremagLeela-lc5lc5-implementationdiff8.0+0.08LLR: 2.95 (-2.94, 2.94) [0.00, 3.00]
Games: 19584 W: 4918 L: 4721 D: 9945
Ptnml(0-2): 15, 1928, 5712, 2119, 18
lc5 step 16: puct-dag vs puct, HistoryKeyLength=1
CremagLeela-lc5[Leela] b80da3dd2a44156c7cf62686136e84b47d652c66diff8.0+0.08Elo: -25.58 +- 5.16 (95%) [N=4000]
Games: 4000 W: 868 L: 1162 D: 1970
Ptnml(0-2): 2, 614, 1061, 322, 1
lc5 checkpoint after step 16 (defaults: reselect, MovesLeft) vs classic, same commit
CremagLeela-lc5lc5-implementationdiff8.0+0.08LLR: 2.96 (-2.94, 2.94) [0.00, 3.00]
Games: 8662 W: 2264 L: 2090 D: 4308
Ptnml(0-2): 3, 806, 2544, 970, 8
lc5 step 12: WaiterResume=reselect vs select
CremagLeela-lc5lc5-implementationdiff8.0+0.08LLR: -0.47 (-2.94, 2.94) [0.00, 3.00]
Games: 9078 W: 2253 L: 2260 D: 4565
Ptnml(0-2): 7, 939, 2651, 938, 4
lc5 step 13: MovesLeft=true vs false
CremagLeela-lc5lc5-implementationdiff8.0+0.08LLR: -1.47 (-2.94, 2.94) [-3.00, 0.00]
Games: 18042 W: 4451 L: 4564 D: 9027
Ptnml(0-2): 20, 1864, 5352, 1779, 6
lc5 step 7 (541ad462, back up through one per-node update) vs step 6 (26084902). Pure refactor, identical golden output.
CremagLeela-lc5lc5-implementationdiff8.0+0.08LLR: -2.95 (-2.94, 2.94) [0.00, 3.00]
Games: 1220 W: 214 L: 401 D: 605
Ptnml(0-2): 1, 242, 310, 57, 0
lc5 step 11: PuctTotals=completed vs started
CremagLeela-lc5lc5-implementationdiff8.0+0.08LLR: 0.29 (-2.94, 2.94) [-3.00, 0.00]
Games: 396 W: 117 L: 103 D: 176
Ptnml(0-2): 1, 26, 129, 42, 0
lc5 step 10b (fp16 priors) vs step 10a; golden: small PV/depth diffs, node sweep within noise
CremagLeela-lc5lc5-implementationdiff8.0+0.08LLR: 0.32 (-2.94, 2.94) [-3.00, 0.00]
Games: 1504 W: 397 L: 383 D: 724
Ptnml(0-2): 1, 147, 443, 159, 2
lc5 step 10a (statistics only for tried edges) vs step 9b; pure refactor, golden identical, +25% NPS at 8 threads
CremagLeela-lc5lc5-implementationdiff8.0+0.08LLR: 0.52 (-2.94, 2.94) [-3.00, 0.00]
Games: 7500 W: 1844 L: 1831 D: 3825
Ptnml(0-2): 3, 765, 2203, 774, 5
lc5 step 9 (template the search on the policy, pure refactor) vs step 8 d5430502
CremagLeela-lc5lc5-implementationdiff8.0+0.08LLR: -0.01 (-2.94, 2.94) [-3.00, 0.00]
Games: 5810 W: 1465 L: 1477 D: 2868
Ptnml(0-2): 3, 598, 1714, 588, 2
lc5 step 8: VisitEngine interface (pure refactor) vs step 7
CremagLeela-lc5lc5-implementationdiff8.0+0.08LLR: 0.93 (-2.94, 2.94) [0.00, 3.00]
Games: 6090 W: 1570 L: 1510 D: 3010
Ptnml(0-2): 2, 572, 1838, 630, 3
lc5 step 6 (smaller edges: 24 B edges, float means, 16-bit in_flight) vs step 5
CremagLeela-lc5lc5-implementationdiff8.0+0.08LLR: 0.59 (-2.94, 2.94) [0.00, 3.00]
Games: 6810 W: 1722 L: 1678 D: 3410
Ptnml(0-2): 1, 661, 2042, 695, 6
lc5 step 5 (38477608): score only started edges plus the first unstarted one, vs step 4. Identical search output; CPU NPS +3.6%.
CremagLeela-lc5lc5-implementationdiff8.0+0.08LLR: 0.73 (-2.94, 2.94) [0.00, 3.00]
Games: 11406 W: 2901 L: 2840 D: 5665
Ptnml(0-2): 5, 1140, 3354, 1197, 7
lc5 roadmap step 2: finish ready and in-flight backups on stop (b57536c6) vs step 1 (8f890b1e)
CremagLeela-lc5lc5-implementationdiff8.0+0.08LLR: 0.68 (-2.94, 2.94) [0.00, 3.00]
Games: 8256 W: 2061 L: 2009 D: 4186
Ptnml(0-2): 5, 803, 2463, 849, 8
lc5: count network value on node at install (8f890b1e) vs parent 035c5b78
CremagLeela-lc5lc5-implementationdiff8.0+0.08LLR: 0.28 (-2.94, 2.94) [-3.00, 0.00]
Games: 11060 W: 2679 L: 2686 D: 5695
Ptnml(0-2): 2, 1109, 3319, 1094, 6
lc5: one shard lookup per selection step (refactor; same output at fixed nodes) vs its parent (revert of step 3 = step 2 code)
CremagLeela-lc5lc5-implementationdiff8.0+0.08LLR: -0.18 (-2.94, 2.94) [0.00, 3.00]
Games: 54 W: 6 L: 22 D: 26
Ptnml(0-2): 0, 17, 9, 1, 0
lc5 step 3 (classic time managers, legacy) vs step 2 (b57536c6)
CremagLeela-lc5lc5-implementationdiff8.0+0.08Elo: 52.51 +- 107.00 (95%) [N=20]
Games: 20 W: 7 L: 4 D: 9
Ptnml(0-2): 0, 2, 3, 5, 0
lc5: count network value on node at install (8f890b1e) vs parent 035c5b78
CremagLeela-lc5[Leela] lc5-implementationdiff8.0+0.08Elo: -34.86 +- 78.28 (95%) [N=20]
Games: 20 W: 2 L: 4 D: 14
Ptnml(0-2): 0, 3, 6, 1, 0
Engine-selection check: same commit 035c5b78, dev engine Leela-lc5 (lc5 search) vs base engine Leela (classic search).
CremagLeelalc5-implementationdiff8.0+0.08Elo: -52.51 +- 86.09 (95%) [N=20]
Games: 20 W: 2 L: 5 D: 13
Ptnml(0-2): 0, 4, 5, 1, 0
superminimal: cremagent skill smoke test