[Feature] Show whether the ratchets and coverage are actually improving #538

New issue

Closed

opened 2026-07-22 11:21:23 +00:00 by brenno · 1 comment

brenno commented

2026-07-22 11:21:23 +00:00

Owner

Problem / motivation

The gates measure whether something works. They do not measure whether they are getting better.

There are now eight or so baselines and ratchets — catchUnderscoreBaseline, rawColorBaseline, debugPrintBaseline, controlByteBaseline, serviceUiImportBaseline, fileSizeBaseline, filePickerPathBaseline, nosemgrepBaseline, plus the coverage floor. Each is allowed to shrink and forbidden to grow. Nothing shows whether any of them has actually shrunk, and a ratchet that stands still for a year is a debt that has been made comfortable.

Coverage has the same shape: 86% is green, but green says nothing about whether the covered lines are the ones that matter. A file can sit at 90% with every branch of its error handling untested.

From the CRA position decision (#519): the borrowed principle is "apply effective regular tests" — effective being the load-bearing word.

Proposed solution

A periodic, advisory report rather than a gate:

each ratchet's current value against its value N commits ago, so movement is visible;
which baselined entries have been there longest;
coverage by directory rather than in one number, so a weak area cannot hide behind a strong one;
optionally, a mutation-testing run over one subsystem per report (make mutate exists) — the only measure that says whether a test would catch anything.

Advisory on purpose. Turning "the ratchet has not moved" into a build failure punishes a quiet month.

Marp / file-format impact

None.

Alternatives considered

Tightening each baseline by hand when someone remembers — that is the current state, and it is why this issue exists.

**Problem / motivation** The gates measure whether something *works*. They do not measure whether they are getting better. There are now eight or so baselines and ratchets — `catchUnderscoreBaseline`, `rawColorBaseline`, `debugPrintBaseline`, `controlByteBaseline`, `serviceUiImportBaseline`, `fileSizeBaseline`, `filePickerPathBaseline`, `nosemgrepBaseline`, plus the coverage floor. Each is allowed to shrink and forbidden to grow. Nothing shows whether any of them has *actually* shrunk, and a ratchet that stands still for a year is a debt that has been made comfortable. Coverage has the same shape: 86% is green, but green says nothing about whether the covered lines are the ones that matter. A file can sit at 90% with every branch of its error handling untested. From the CRA position decision (#519): the borrowed principle is "apply effective regular tests" — *effective* being the load-bearing word. **Proposed solution** A periodic, advisory report rather than a gate: - each ratchet's current value against its value N commits ago, so movement is visible; - which baselined entries have been there longest; - coverage by directory rather than in one number, so a weak area cannot hide behind a strong one; - optionally, a mutation-testing run over one subsystem per report (`make mutate` exists) — the only measure that says whether a test would catch anything. Advisory on purpose. Turning "the ratchet has not moved" into a build failure punishes a quiet month. **Marp / file-format impact** None. **Alternatives considered** Tightening each baseline by hand when someone remembers — that is the current state, and it is why this issue exists.

brenno added the

accepted

enhancement

labels

2026-07-22 11:21:23 +00:00

brenno commented

2026-07-22 11:24:06 +00:00

Author

Owner

Part of the solution should also be that this information is available through the project Wiki and regularly is updated (for instance once per month). Also with information how the project evolves (functionality and new code).

brenno referenced this issue from a pull request that will close it,

2026-07-22 13:53:11 +00:00

feat: meet doorlooptijd, ratchetbeweging en of de documentatie klopt #558

brenno referenced this issue from a commit

2026-07-22 13:53:13 +00:00

chore(meting): geef de ratchetmeting een doel en noteer wat open blijft

brenno closed this issue

2026-07-22 13:53:13 +00:00