[Bug] The lexicon performance test asserts on an average, not a best-of-N #638

New issue

Closed

opened 2026-07-22 16:23:36 +00:00 by brenno · 0 comments

brenno commented

2026-07-22 16:23:36 +00:00

Owner

Found in the pre-publication test review.

Evidence: test/privacy_bulk_lexicon_test.dart:246-257 runs 20 scans back to back and asserts the average against lessThan(5) ms. test/large_deck_performance_test.dart:112-121 solves the same problem correctly with fastestOf(3) — "returns the fastest run". There is no Timeout( in any test file, and dart_test.yaml defines no retry, tag or isolation for either of these.

Why this matters: an average over 20 runs is more sensitive to a single outlier than a minimum over 3, and under a full suite (516 test files, parallel isolates) that is the classic wall-clock trip. The first contributor running make check on a loaded laptop gets a red privacy performance test for a change to, say, a dialog — a failure with nothing to do with their work, which is the fastest way to teach someone that the gate is noise.

Proposal: move fastestOf from large_deck_performance_test.dart into test/support/ and reuse it here: final perScan = fastestOf(5, () => lexicon.findIn(text));. Same assertion, same threshold, no noise sensitivity.

Found in the pre-publication test review. **Evidence:** `test/privacy_bulk_lexicon_test.dart:246-257` runs 20 scans back to back and asserts the **average** against `lessThan(5)` ms. `test/large_deck_performance_test.dart:112-121` solves the same problem correctly with `fastestOf(3)` — "returns the fastest run". There is no `Timeout(` in any test file, and `dart_test.yaml` defines no retry, tag or isolation for either of these. **Why this matters:** an average over 20 runs is more sensitive to a single outlier than a minimum over 3, and under a full suite (516 test files, parallel isolates) that is the classic wall-clock trip. The first contributor running `make check` on a loaded laptop gets a red privacy performance test for a change to, say, a dialog — a failure with nothing to do with their work, which is the fastest way to teach someone that the gate is noise. **Proposal:** move `fastestOf` from `large_deck_performance_test.dart` into `test/support/` and reuse it here: `final perScan = fastestOf(5, () => lexicon.findIn(text));`. Same assertion, same threshold, no noise sensitivity.

brenno added the

bug

label

2026-07-22 16:23:36 +00:00

brenno referenced this issue from a commit

2026-07-22 18:07:01 +00:00

fix: zet de versie op 0.1.0, en meet de prestatietest op de snelste

brenno referenced this issue from a pull request that will close it,

2026-07-22 18:07:02 +00:00

fix: versie naar 0.1.0, en de prestatietest op de snelste meten (#629, #638) #664

brenno closed this issue

2026-07-22 18:07:16 +00:00