Why count combinations
Test suites are usually reported by size. Size is easy to produce and easy to read, but it mostly measures how much compute was spent. What matters is how many genuinely different situations the system was put in. Research on combinatorial testing finds that bugs usually need a few conditions at once, so a bug that appears in one rare combination is never found by repeating the common ones.
Seventeen shapes of access rule
Real access rules are rarely just "column equals value". We built 147 access roles across 17 shapes, and ran each on live data:
- ranges and exclusions;
- null and not-null checks;
- pattern matches;
- rules that span two tables;
- rules that overlap, and rules that conflict and must resolve to the most restrictive one.
No shape was tested only where failure was impossible. The shapes with the most at stake were the adversarial ones: several rules applying at once, and rules that contradict each other. With enforcement on, no shape returned an unauthorized row.
Ten structures of query
A rule added to the outer query is worthless if the data also arrives through a subquery, a common table expression or the second half of a UNION. These aren't exotic: a language model writing analytical SQL produces them all the time. We stacked ten such structures onto the test queries, including an injection-style OR condition built to defeat a naive filter. With enforcement on, no structure returned an unauthorized row.
Three defects we found in our own rewriter
Each produced SQL that parsed cleanly, ran without error, and returned the wrong rows. That is the hardest kind of defect to catch, because nothing looks broken.
| Defect | What went wrong |
|---|---|
| First branch only | The rule reached only the first half of a UNION. The second half ran unfiltered. |
| One rule for two aliases | A table used twice in one query under different names got a single, ambiguous rule, leaving one side of the self-join unfiltered. |
| Rule in the wrong scope | A rule meant for the outer query could land inside a nested subquery instead. |
All three are fixed, and the deliberate-fault suite in the companion analysis includes faults that reproduce each one.
Two systems can report the same perfect score. One was never tested on hard cases. The other was tested, failed three times, and was fixed. They are not the same system, even though the number looks identical.
Measuring coverage
Across 76,706 test cases we measured coverage over four factors: the engine, the kind of rule, the shape of the query, and the database.
| Combination | Covered | Possible | Coverage |
|---|---|---|---|
| Single factor values | 39 | 39 | 100% |
| Pairs | 465 | 465 | 100% |
| Triples | 1,896 | 1,897 | 99.95% |
Combinations that cannot exist are left out. Some databases have no column that could support a range rule, so that pairing is impossible rather than untested: of 187 possible rule-and-database pairings, 147 can actually be built. Counting the impossible ones against ourselves would understate coverage just as ignoring real gaps would overstate it.
Limits
- Four factors aren't everything. Query length, join depth, how the data is distributed and heavy simultaneous load are not part of this analysis.
- One real gap. The one uncovered triple is a genuine gap, not an impossible combination, and we haven't yet established whether it can occur in practice.
- Generated rules, public data. That keeps the analysis checkable, and limits what it proves about a real deployment.
Method and data
The benchmark is BIRD Mini-Dev, PostgreSQL dev split. All figures were recomputed from saved results and verified against the live database in October 2026; the coverage analysis was rerun with the same result. The policy generator, test harness and per-query verdicts are available on request.