[Docs] The CRA table credits mutation testing that is one operator over 7 of 96 files #616

New issue

Closed

opened 2026-07-22 16:21:48 +00:00 by brenno · 1 comment

brenno commented

2026-07-22 16:21:48 +00:00

Owner

Found in the pre-publication test review.

Evidence: assurance/CRA-2024-2847-positie.md:41 cites "mutatietesten" as evidence for Annex I §3 with a dash in the Open column — while line 72 of that same file says the question of "whether a test actually proves anything (make mutate)" is precisely not answered. CONTRIBUTING.md:157 calls it a "gate".

Measured: make mutate kills 16/16 on markdown_validator.dart — a genuinely good result — but tool/mutation_check.dart:36 mutates only String.startsWith and String.endsWith. There are 286 such predicates across 96 files in lib/; make mutate-parsers targets seven of them. There is no condition, boundary-value or return-value mutant, and no recorded score.

Why this matters now: an auditor or CRA reader takes that row to mean mutation coverage is handled, which upgrades the 86.2% line coverage into evidence of assertion strength that does not exist. It is the sort of claim a critical outsider checks first — and the refutation is in the same file.

Proposal: one sentence in the CRA table and in CONTRIBUTING: "mutation testing, manual, one operator (startsWith/endsWith) over the Markdown parsers — 7 of 96 files; last sweep , 0 survivors." No new tooling; just state the scope.

Found in the pre-publication test review. **Evidence:** `assurance/CRA-2024-2847-positie.md:41` cites "mutatietesten" as evidence for Annex I §3 with a dash in the *Open* column — while **line 72 of that same file** says the question of "whether a test actually proves anything (`make mutate`)" is precisely *not* answered. `CONTRIBUTING.md:157` calls it a "gate". Measured: `make mutate` kills 16/16 on `markdown_validator.dart` — a genuinely good result — but `tool/mutation_check.dart:36` mutates only `String.startsWith` and `String.endsWith`. There are **286 such predicates across 96 files** in `lib/`; `make mutate-parsers` targets seven of them. There is no condition, boundary-value or return-value mutant, and no recorded score. **Why this matters now:** an auditor or CRA reader takes that row to mean mutation coverage is handled, which upgrades the 86.2% line coverage into evidence of assertion strength that does not exist. It is the sort of claim a critical outsider checks first — and the refutation is in the same file. **Proposal:** one sentence in the CRA table and in CONTRIBUTING: "mutation testing, manual, one operator (`startsWith`/`endsWith`) over the Markdown parsers — 7 of 96 files; last sweep <date>, 0 survivors." No new tooling; just state the scope.

brenno added the

docs

label

2026-07-22 16:21:48 +00:00

brenno referenced this issue from a commit

2026-07-22 17:07:10 +00:00

docs(code): geef de vijf dragende klassen een contract, en meet de mutatietest (#624 #616)

brenno referenced this issue from a commit

2026-07-22 17:10:07 +00:00

docs: haal negen achterhaalde beweringen uit de publieke teksten

brenno referenced this issue from a commit

2026-07-22 17:12:52 +00:00

docs(code): geef de vijf dragende klassen een contract, en meet de mutatietest (#624 #616)

brenno referenced this issue

2026-07-22 17:13:34 +00:00

docs: haal negen achterhaalde beweringen uit de publieke teksten #655

brenno referenced this issue

2026-07-22 17:14:10 +00:00

docs: de publicatieklaar-tros — 26 docs-issues in zeven commits #656

brenno commented

2026-07-22 17:19:16 +00:00

Author

Owner

Opgelost in `cc00070` (PR #655).

De rij bij Annex I §3 noemt make sast en make check-secrets, en de kolom Open zegt nu wat mutatietesten werkelijk zijn: handwerk, één operator (startsWith/endsWith), zeven vaste doelen.

Twee cijfers zelf nageteld in plaats van overgenomen: het zijn 7 vaste doelen in de Makefile (ik schreef eerst 5), van de 96 bestanden in lib/ waar zo`n predicaat in staat — 309 voorkomens in totaal.

Opgelost in \`cc00070\` (PR #655). De rij bij Annex I §3 noemt `make sast` en `make check-secrets`, en de kolom Open zegt nu wat mutatietesten werkelijk zijn: handwerk, één operator (`startsWith`/`endsWith`), zeven vaste doelen. Twee cijfers zelf nageteld in plaats van overgenomen: het zijn **7** vaste doelen in de Makefile (ik schreef eerst 5), van de **96** bestanden in `lib/` waar zo`n predicaat in staat — 309 voorkomens in totaal.

brenno closed this issue

2026-07-22 17:19:16 +00:00

brenno referenced this issue

2026-07-22 17:24:19 +00:00

docs: de publicatieklaar-tros — 26 docs-issues in zeven commits #656

brenno referenced this issue from a commit

2026-07-22 17:29:21 +00:00

docs(code): geef de vijf dragende klassen een contract, en meet de mutatietest (#624 #616)

brenno referenced this issue

2026-07-22 17:30:46 +00:00

[Bug] The mutation sweep has a survivor: the checklist-scope predicate is untested or dead #660

brenno referenced this issue