Turning Production Bug Post-Mortems into Roslyn Analyzers

The ET Ducky cloud API ships with eleven custom Roslyn analyzers, and every one of them exists because a specific bug reached production or a specific audit found a class of latent ones. This post documents the method that emerged: how a post-mortem becomes a compile-time rule, why the rules are deliberately conservative, how structured justification comments serve as the escape hatch, and the pattern we now consider the ceiling of the approach — an analyzer whose annotations are the documentation, verified against runtime behavior by an end-to-end check.

The premise: fix the class, not the instance

A bug fix repairs one site. The post-mortem question that produces an analyzer is different: what decision did the author fail to make, and can the compiler force the next author to make it? Three production examples from this codebase:

• The cookie jar nobody configured. A shared HttpClient kept its default cookie container. The first response from an integrated service (Harbor) set a session cookie; every later mutating call replayed it and tripped the service's CSRF middleware with a 403 — despite valid credentials. The integration worked exactly once per process lifetime. The fix was one line (UseCookies = false on a dedicated named client). The class of bug is "nobody decided what the cookie jar does for this integration."
• The background service that silently did nothing. EF Core global query filters scope every query to the current tenant. Background services run under a root context with no tenant — so the moment a tenant-filter fix shipped, five hosted services began returning zero rows for every tenant, silently, because none of their queries called IgnoreQueryFilters(). No exception, no log, no data. The class: "code that runs outside a tenant context must explicitly declare how it crosses the tenant boundary."
• The entitlement check that answered the wrong question. A checkout endpoint gated repurchase on "does a non-revoked license exist?" But churn deliberately leaves licenses non-revoked and lets them expire via the renewal gate — so a churned workspace was told it was already subscribed and could not repurchase. The class: "a license's revocation state is not an entitlement signal, and any boolean built on !Revoked drifts from the truth exactly on churned customers."

Each fix shipped, and then each produced a rule: ETD0011 requires every AddHttpClient registration to make its cookie policy explicit; ETD0002 requires tenant-filter handling in background-service query chains; ETD0013 bans !Revoked-shaped entitlement checks outside the license-domain services.

Conservative by design

Every analyzer in the set is a syntax-level heuristic with intentionally narrow scope, and the limitations are documented in the analyzer's own header. ETD0002 walks only direct same-class helper calls, not cross-method flow. ETD0010 covers controllers only, because controllers are where the property it checks is syntactically decidable — extending it to services is ticketed, not hand-waved. The reasoning: a Roslyn analyzer competes with code review, not with a theorem prover. A rule that fires false positives gets suppressed project-wide within a month and then catches nothing. A rule that catches eighty percent of a bug class with zero false positives keeps its credibility and keeps running. When a rule cannot decide, it asks the author to decide — which is the next mechanism.

Justification comments as structured escape hatches

Most of the rules do not ban a construct; they require a decision about it. The mechanism is a structured comment with a rule-specific marker and a mandatory reason:

// cookie-policy: bearer-token API, server sets no session cookies
// tenant-opt-out: deliberate cross-tenant sweep, admin metrics endpoint
// cloud-egress: license renewal POST to api.etducky.com
// sql-reviewed: executes vetted migration files only
// local-safe: fails loudly via LocalClerkService; SPA hides this surface in local mode

The analyzer accepts the marker on the statement, the enclosing member, or (for boundary classes) the type. This does three things a plain suppression attribute does not. The reason is mandatory and adjacent to the code, so review sees it. The markers are uniform and greppable, so every exception to a rule is enumerable with one command. And writing one is mildly effortful — enough that the path of least resistance is fixing the code rather than annotating it.

The ceiling: annotations as verified documentation

The pattern we consider most worth copying is ETD0014, the egress rule. The product's self-hosted tier documents a small, enumerable set of outbound network destinations. Historically that list lived in a whitepaper and rotted freely — any service could grow a cloud call and nothing would notice. The rule: every outbound HTTP dispatch in code reachable in local mode must carry a // cloud-egress: <reason> marker. Which means grep -rn "cloud-egress:" --include=*.cs now produces the documented egress list.

The other half closes the loop at runtime: a delegating handler on every factory-built HTTP client logs each dispatch host in local composition, and an end-to-end check asserts the observed host set matches the annotated set. The analyzer pins the authoring-time contract; the handler observes the runtime truth; drift between them is a build failure. Documentation that is generated from enforced annotations and checked against observed behavior cannot rot — which is the property the whitepaper claim actually needed.

One implementation detail worth stealing: the rule detects HTTP dispatches semantically, by checking that the receiver derives from HttpMessageInvoker, not by matching method names. SendAsync and GetAsync appear on SignalR clients, distributed caches, and third-party SDKs; name matching would drown the rule in false positives on day one.

Smaller rules that earn their keep

Not every analyzer needs a production incident. Three in the set are source-hygiene rules with codebase-specific rationale: config keys for the deployment mode may only be read inside the one profile class that owns the composition decision (a stray Configuration["Deployment:Mode"] would fork it); non-constant SQL passed to raw-SQL EF APIs requires a review marker, because row-level security constrains what a query can see, not whether it is injectable; and invisible or bidirectional Unicode characters in source are flagged outright — the Trojan-source class, with a single allow-listed NUL that a composite-key separator legitimately uses. Alongside the custom rules, the standard BannedApiAnalyzers package with a BannedSymbols.txt covers the cases where the fix is simply "never call this API here."

What this costs and what it returns

The eleven analyzers are a single netstandard2.0 project referenced by the API project as an analyzer; build-time cost is negligible and there is no runtime component except the egress-observation handler. Writing one is a day the first time and hours after that — the statement-scope walking and comment-detection helpers are shared across rules. The return is asymmetric: the tenant-filter rule alone guards the exact regression that once silently broke every background service in production, and it re-verifies that guarantee on every build. A test would have caught the instance; the analyzer refuses the class.

The method, compressed: when a bug's post-mortem names a decision that was never made, write a conservative syntax-level rule that forces the decision, provide a greppable justification marker as the escape hatch, document the rule's blind spots in its own header, and — where the rule protects an external claim — close the loop with a runtime check that the annotations still match reality.