The problem with grading yourself
The strongest objection to a vendor's benchmark is that the vendor also writes the answer key. There is a quieter problem too: a test where nothing could ever go wrong will give a perfect score to a system that does nothing. A zero only means something if the number could have been higher.
1. An answer key we don't build
For each access policy, the database makes its own copy of the protected table, keeping only the rows the person may see. It takes one plain statement:
CREATE TABLE rbac.customers AS SELECT * FROM public.customers WHERE "currency" = 'CZK';
The benchmark's own reference SQL then runs against that copy, and whatever it returns is the right answer for that person. Our engine never touches it, so it is graded against a result it didn't compute. In October 2026 we rebuilt all eleven of these tables from scratch and compared them, row by row in both directions, with the ones the published results were scored on. Not one row differed.
2. Proof the test can fail
We ran every query a second time with enforcement switched off, only to record whether a leak was possible at all.
- Could have leaked 16,101 · 59.3%
- Nothing to leak 11,046 · 40.7%
So in 16,101 cases a broken engine would have shown up. The other 11,046 never touched restricted rows, and on their own they prove nothing. That split is the first number a skeptical reader should check.
3. Three checks on the measurement itself
The answer key can be right and the test harness still blind. We checked that three ways, each rerun live in October 2026.
| Check | Result | What it rules out |
|---|---|---|
| Nine deliberate faults | 9,507 of 9,507 | That the harness would miss a broken engine. We broke it nine ways: skipping the rule, applying it only to the outer or the first scope, skipping it inside common table expressions or UNION branches, reversing it, replacing it with an always-true condition, OR-ing it with one, and comparing against the wrong value. Every break that changed an answer was caught. |
| PostgreSQL's own access control | 394 of 394 | That our answer key is itself wrong. Under an account that can't bypass it, PostgreSQL's built-in row-level security agreed with our restricted copies on every query the two could be compared on. Six couldn't be compared. |
| Coverage of combinations | 100% · 99.95% | That a big test count hides shallow testing. Every valid pair of conditions, and all but one three-way combination, across 76,706 test cases. |
A note on the 9,507. Many deliberate breaks change nothing on a given query; reversing a rule does nothing if the query already filters on that column. No test can catch a break that didn't change the answer, so counting those as misses would be unfair and counting them as catches would flatter us. We set those 3,777 aside.
4. How we state the result
No finite test proves a failure rate of zero, so we state a bound instead. Across 27,147 executed queries on live PostgreSQL, spanning 11 databases, 147 access roles and 17 rule shapes, we saw no case where an unauthorized row came back. That bounds the leak rate at about 0.011 percent per query, at 95 percent confidence.
Counting each of the 147 roles as the unit instead of each query, the bound is 2.02 percent. Both numbers are right; they answer different questions, so we always say which unit we mean.
Limits
- Generated rules, public data. The policies are generated, not taken from a real company, and the data is a public benchmark. That keeps the work checkable, and it limits what it proves about a production deployment.
- Structured queries only. This says nothing about retrieval from unstructured documents, which the benchmark doesn't exercise.
Method and data
The benchmark is BIRD Mini-Dev, PostgreSQL dev split. All figures were recomputed from saved results and verified against the live database in October 2026, and the answer key tables were rebuilt from scratch with no row differing. The fault tests, the PostgreSQL cross-check and the coverage analysis were each rerun live. The policy generator, test harness and per-query verdicts are available on request.