When Missing RLS FORCE, Write-Only Config Properties, and Cached Module Versions Fail Without Failing

The ET Ducky cloud API and its agents are verified by a mixture of Roslyn analyzers, integration tests against a live Postgres, and the ordinary build. Over a fortnight of work on it, six defects went undetected by that machinery. None of them failed in the way an ordinary bug does. In each case a check reported success without the thing it checked being true.

A migration applied row-level security to a table it had never created, recorded itself as applied, and protected nothing. A build reported a warning count sixty percent lower than the week before while carrying sixty-three compile errors underneath the improvement. A defensive guard around a misspelled global turned a broken list refresh into a feature that appeared to work. In each case the signal that should have surfaced the defect was the same signal that concealed it.

What follows is the pattern, the six instances of it we hit, and the rule we now apply when adding verification of any kind.

A protection that appeared in every catalog view and enforced nothing

PostgreSQL row-level security has two switches rather than one, and the second is easy to miss because the first appears to be sufficient. ENABLE ROW LEVEL SECURITY applies policies to ordinary roles but deliberately exempts the role that owns the table, and applying them to the owner as well requires FORCE ROW LEVEL SECURITY as a separate statement. An application that connects as its own schema owner therefore bypasses every policy on a table that was enabled without being forced, and connecting that way is common because the same role usually runs the migrations.

The policy still exists in that state. It is listed in pg_policy, the table reports rowsecurity = true, and it appears in every catalog view an engineer would think to check when asking whether the table is protected. It simply does not run for the one role that matters.

The shape that produces this is a migration that hand-writes its row-level security rather than calling a helper that emits all three statements together, which leaves something like the following for each table it creates:

ALTER TABLE public."SomeTable" ENABLE ROW LEVEL SECURITY;
DROP POLICY IF EXISTS tenant_isolation ON public."SomeTable";
CREATE POLICY tenant_isolation ON public."SomeTable" ...

An enable and a policy, with no force. If the application also has an ORM-level query filter then reads are still being scoped and this is a loss of defence in depth rather than a leak, which is the usual case and the reason it can persist for a long time without anybody noticing. The entire reason to maintain two independent tenancy boundaries is that a single mistake should not be sufficient to cross either of them, and on a table in that state there is only one.

The missing keyword matters less than the fact that every signal available says the protection is present, since the migration succeeds, the policy exists, the catalog agrees, and nothing in the system is in a position to notice the gap because nothing is asking the question in a form that could come back false.

Six shapes the same failure takes

Once you start looking for checks that can succeed vacuously it becomes clear this is a family rather than a curiosity, and the following six are all real, all encountered in the same few weeks, and all fail in a way that is indistinguishable from working.

A function that skips instead of raising. A row-level security helper of this kind is usually deliberately tolerant, so when the table or the column it was given does not exist it emits a notice and returns rather than throwing. That is defensible in isolation, since it lets one migration apply policies across a list of tables without dying on an environment where one is absent. The consequence is that a migration can call it on a table it forgot to create, complete successfully, record itself as applied, and leave nothing behind. A migration in that state leaves its only evidence as a notice in a log that nobody reads. A function that degrades to a no-op is one whose caller cannot distinguish having worked from having declined, so if it will not raise then the caller has to assert.

A metric that improves because the work stopped. A build reported 392 warnings against 1,670 the week before, which reads as a seventy-seven percent improvement and was the precise opposite, because it also carried sixty-three errors that nobody looked at. A formatting tool had written unmerged-change conflict markers into nine files, a core project consequently failed to compile, and everything downstream of it emitted no warnings at all. Any check that reads a quality metric without reading the error count first can be fooled the same way, and a person skimming a six-thousand-line log will be fooled every time, since errors are not a more severe grade of warning, they are the thing that invalidates the warning count entirely.

A guard that converts a typo into silence. The idiomatic defensive check in JavaScript is to confirm an object and its method exist before calling them, which is what we did around a call to refresh a list after a bulk delete, and the global we guarded does not exist anywhere in the application. Deletions succeeded on the server, the rows stayed on screen, and nothing was logged or thrown, because a guard written for an expected absent case had been handed a misspelling instead. A defensive guard is a claim that the absent case is anticipated and handled, so when the absent case is actually a mistake the guard is the thing preventing you from finding out, and the fix is not to remove it but to make it say something on the way past.

A scope that had already ended. SET LOCAL in PostgreSQL lasts for the enclosing transaction only, so running it, allowing the transaction to end, and then running the statement it was meant to enable produces a statement that executes without it. In our case that was an update that matched zero rows because row-level security was hiding the row it targeted, reported as UPDATE 0, which is also exactly what you see when the row genuinely does not exist. A zero row count is the correct answer to a great many different questions and carries no information about which one was asked.

A binder that skips what it cannot use. Renaming a configuration key, we kept the old name as a compatibility alias for one release, and the natural way to write that alias is a setter forwarding to the new property on the grounds that the value only needs to travel inwards. The .NET configuration binder skips write-only properties without an exception, a warning or a log line, so every host that had been configured by hand under the old key would have silently reverted to the default on upgrade, which is the single worst outcome the rename could have produced and would have been reached by writing the shim the obvious way. It works only because somebody kept a getter for reasons they wrote down, and a fact recorded only in one person's memory is not a safeguard.

A fix that ships and never arrives. A front end can end up with two independent cache-busting registries, one being version tags in the HTML and the other a set hardcoded inside a bundle that lazily loads page modules, and a correction to a module in the second group was deployed, served, and never reached a browser because the version string it was requested under had not moved. Correcting the second registry then required moving the bundle's own version as well, since otherwise the browser serves the cached bundle, that cached bundle requests the old module version, and the correction cannot propagate no matter how many times it is deployed.

Why review does not catch this class

Review is effective at asking whether code is correct and ineffective at asking whether code is reached, and every example above is locally correct: the migration is valid SQL, the guard is idiomatic JavaScript, the setter-only property compiles and does exactly what it says. The defect in each case is a relationship between the change and something outside the diff, whether an owner role, an error count, a global that does not exist, a transaction boundary, a binder's documented behaviour or a second version registry. A reviewer would have to hold all of that in mind while reading a change that looks entirely fine.

There is a more structural reason as well, which is that in each case the check and the thing being checked are the same object. Asking a migration whether it applied row-level security returns a yes, asking a build whether its warning count improved returns a yes, and asking a deploy whether it deployed returns a yes, none of which tell you anything, because you cannot verify something by asking it about itself and a surprising amount of tooling is built as though you can.

Preferring checks that cannot pass unless the property holds

The rule that covers all six is that a useful check is one that cannot pass unless the property being tested actually holds. The corollary is that a check that can pass for a reason unrelated to the property is not merely a weak check, it is an actively misleading one, because it converts an open question into a confident wrong answer and removes the discomfort that would otherwise have prompted somebody to look.

Assert the outcome rather than the call

Calling a function that applies a protection establishes only that the call happened, so the useful move is to query for the protection afterwards and fail on its absence, which for the row-level security case means going to the catalog directly:

SELECT c.relname
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname = 'public'
  AND c.relkind = 'r'
  AND EXISTS (SELECT 1 FROM pg_attribute a
              WHERE a.attrelid = c.oid AND a.attname = 'OrganizationId')
  AND NOT (c.relrowsecurity AND c.relforcerowsecurity);

Any row this returns is a table that believes it is protected and is not, which is a different question from whether the migration succeeded and is the one that actually matters. It is also worth deriving the list of tables from the schema rather than from a checked-in list, since a table added next month is then covered on the day it is created, which is precisely when it is most likely to be forgotten.

Run the check under the same constraints as the thing it verifies. The assertion above is worthless when run as a superuser, because a superuser bypasses row-level security entirely and every assertion in the suite would pass without proving anything, which is the exact failure the test exists to prevent reproduced inside the test itself. Our integration harness therefore creates a non-superuser role that owns the schema, mirroring how the application connects in production, and the bootstrap script says so in a comment because somebody will otherwise resolve a permissions error by granting superuser and quietly disable the entire suite while making it pass.

The general form is that a test with more privilege than production is testing a system you do not ship, and the same reasoning applies well beyond databases to anything that behaves differently for a privileged caller.

Prefer the compiler and the database to anything that reads text. Given a choice between a rule that runs inside a compiler, a query that runs against a real database, and a script that reads source text looking for a pattern, the first two are worth substantially more effort than the third. We have direct evidence for this, because a text-based scan we wrote to check an invariant across the codebase reported twenty-three violations of which every single one was false, since it matched if() and foreach() as method names and had no way to see that a base method carried the attribute its overrides inherited. The compiler's own output, which was zero warnings, turned out to be the truth. A syntax tree understands inheritance, scope and symbols, a regular expression understands characters, and most invariants worth enforcing are stated in the first vocabulary rather than the second.

When adding a check makes things worse

The rule also rules things out, including work that looks like an improvement. We wrote a test asserting that every HTTP endpoint declares an authorization posture, implemented by reflecting over the assembly and reading IL for authorization calls, and its first run reported more than a hundred offenders of which every one was a false positive.

A compile-time analyzer had been asking that exact question for over a year and was configured as a build error in release, so a passing release build already proved the property. It also did the job better, because a syntax tree can see class-level attributes inherited by every action, marker attributes used as posture declarations, and the comment convention the codebase uses for endpoints authenticated by middleware, all three of which reflection is blind to. The test had re-derived a settled answer from strictly less information and got it wrong.

We withdrew it the same day. The lesson is narrower than a warning against duplicating an analyzer. A check that asks a question already answered at compile time can only be equal to or worse than the existing answer, and a worse one is not free, because a hundred false positives is a test that gets disabled within a week and takes with it whatever genuine finding it might have produced later. The bar for adding verification is not whether it could catch something, it is whether it can be satisfied only by the property holding and whether it fails for no other reason.

What the loop looks like when it closes

A check of that kind costs about an hour to write and tends to be worth it on the first run, because the tables it names are the ones that have been in that state longest and are least likely to be under suspicion. Re-applying the protection is then a small migration, and the useful discipline is to have that migration assert the result afterwards rather than trust that the call did anything. The same test then passes against a live database connected as a non-superuser owner, which is the only configuration in which passing means something.

That is the whole loop, and the interval before it exists is not usually the product of carelessness. It is the product of nothing having asked.

The rules in order

Assert the outcome rather than the call, since a function that returns without raising has told you only that it was reached.

Derive the set being checked from the system rather than from a checked-in list, so that something added next month is covered on the day it is created.

Run the check under the same constraints as the thing it verifies, because a test with more privilege than production is testing a system you do not ship.

Prefer a rule that runs in a compiler or a query that runs against a real database to a script that reads source text, since the first two understand symbols and scope and the third understands characters.

Before adding a check, establish that it can be satisfied only by the property holding and that it fails for no other reason, because a check that passes for unrelated reasons converts an open question into a confident wrong answer.

Then ask of every check already in place what it would report if the thing it checks were entirely absent, and treat any that answer success as something other than a check.