34 leaks with the rule in the prompt. None with the rule on the query.

Why measure this

When a company connects AI to its databases, different people may see different data. The usual approach is to write the access rule into the model's prompt and trust the model to follow it.

Text-to-SQL benchmarks don't test that. They score whether the answer is right, but they don't know who is asking, so an answer drawn from rows the person shouldn't see still counts as correct. We wanted to know how often the prompt approach actually holds.

How we tested it

• The questions. BIRD Mini-Dev, a widely used public benchmark: 500 questions across 11 real databases, each with its own reference SQL.
• The rules. One access policy per database, generated mechanically rather than picked by hand. Together they cover 1,714,143 rows, of which 1,194,691 are visible to the person each policy describes.
• Two runs per model. In the first, the rule was written into the prompt. In the second, the prompt said nothing about access, and the rule was added to the generated query before it ran.
• The scoring. Only the 178 to 180 questions per model that touch the protected table were scored. A query counted as correct only if it returned exactly the rows that person may see.

What we found

Two of the five models never returned an unauthorized row. Three did. Pooled across all five, that is 34 unauthorized results in 895 scored queries, against none in the same 895 when the rule was enforced on the query. A Fisher exact test puts the chance of a gap this large arising by luck at 8.5 × 10⁻¹¹.

No failures in 895 tries doesn't prove the failure rate is zero. It bounds it: below 0.334 percent at 95 percent confidence.

The models that leaked were the cheaper and open-weight ones in this set. We noticed that pattern, but we never varied price as part of the study, so it is an observation, not a measured relationship.

Accuracy went up, not down

You might expect a model told less about the request to answer less accurately. The opposite happened. Taking the rule out of the prompt raised the share of exactly right answers on every model, from 1.8 points on Claude Opus 5.5 to 12.0 points on Gemini 3.8 Flash, with an average gain of 7.4 points. We measured that the gain happened. We did not measure why.

Limits

• Wording matters. The prompt condition used one appended sentence stating the rule. In separate testing, rewording the same instruction moved one model's compliance from 77.3 percent to 100 percent. These results describe this phrasing, not the best any vendor could do.
• Fixed settings. Temperature 0, a 2,048-token output limit, one attempt per question. Models were called through OpenRouter in October 2026 at fixed versions; later versions may behave differently.
• One model often wrote no SQL. DeepSeek V4.1 Flash produced no usable SQL for 127 of 500 replies, which lowers its accuracy in both runs. The raw replies weren't kept, so we can't say why.
• Generated rules, public data. The policies are generated, not taken from a real company, and the data is a public benchmark. That makes the result checkable, and it also limits what it proves about a production deployment.

Method and data

The benchmark is BIRD Mini-Dev, PostgreSQL dev split, published on the BIRD benchmark website. All figures were recomputed from saved results and verified against the live database in October 2026. The policy generator, test harness, per-query verdicts and the SQL each model generated are available on request, so any figure here can be recomputed independently.