Behavioral Security Monitoring with Kernel Events: Signature Rules vs Learned Fleet Baselines

Kernel telemetry sees the actions a piece of malware has to take: a process is created, a file is written, a socket connects. Turning that stream of events into a detection can be done two ways. One matches known-bad shapes with rules. The other learns what is normal and flags deviation from it. This post explains how each works over kernel events, where each fails, and why a per-application baseline built across a fleet is a strong complement to rules rather than a replacement for them. It also covers the constraint that governs any such baseline in a multi-tenant product: one organization's data must never influence another organization's detections.

The rule layer described here ships in the ET Ducky agent today; the two existing posts on the cross-platform rule engine and ransomware kill-chain detection cover it in detail. The fleet-baseline layer now ships as a cloud-side analytics rule, fleet-baseline-deviation, which is default-off and runs in shadow mode first; this post describes the reasoning behind it and how the two layers fit together.

The event stream both approaches share

Both detection styles feed on the same source: kernel-level events, captured with Event Tracing for Windows on Windows and with eBPF programs on Linux, and normalized into a common shape of process, file, and network events. On Windows this means providers for process creation and exit, file operations, and network connections; on Linux it means eBPF programs on the equivalent tracepoints, such as process execution and the connect and accept syscalls. The mechanics of that capture, and how process attribution differs between the two operating systems, are covered in ETW vs eBPF: attributing a network connection to a process.

Kernel-level telemetry matters here for a specific reason: it is harder to evade than instrumentation that lives in user space. A userland API hook can be unhooked or simply bypassed by calling the underlying syscall directly; an event emitted by the kernel as the action actually happens is closer to ground truth. This is a matter of degree, not an absolute. Kernel telemetry is harder to tamper with, not impossible, and techniques exist to evade even kernel instrumentation. The point is that the kernel event stream is a stronger foundation to build detection on than user-space hooks, whichever detection style sits on top.

Approach one: signature and threshold rules

A rule encodes a known-bad shape. When the shape appears in the event stream, the rule fires. The shapes that make good rules are the ones with little or no legitimate use:

• A shell process that spawns a network downloader, which then spawns another shell, within a short window. This is the structure of a curl http://... | bash attack.
• A burst of file renames to extensions associated with ransomware, above a count within a time window.
• A privilege-escalation binary invoked by a parent that is not an interactive session, which is unusual for a human-driven sudo.
• An outbound connection from a system daemon on a port outside the small set that daemon normally uses.

Rules have real strengths. They are precise, they produce few false positives when the shape is genuinely rare, they need no training period and work on a brand-new host immediately, and they are self-explaining, because the rule that fired states exactly what it matched. This is why they are the right tool for well-understood malicious behavior, and why they form the base layer of most endpoint detection.

Their limitation is structural: a rule can only catch a shape that someone has already described. Behavior that is malicious but novel, or that deliberately stays under a threshold, does not match and does not fire. A stealthy ransomware process that encrypts one file per second stays under a burst threshold. A living-off-the-land binary, a signed and legitimate system tool abused for a malicious end, may do nothing that any single rule considers bad. These are exactly the cases a baseline is meant to cover.

Approach two: behavioral baselines

A baseline inverts the question. Instead of asking whether behavior matches a known-bad shape, it asks whether behavior fits what is normal, and treats a poor fit as the signal. The appeal is coverage of the unknown: a baseline does not need anyone to have described the attack in advance, only for the attack to look different from normal.

The difficulty is entirely in the definition of normal. A baseline is only as trustworthy as the population it is learned from, and a naive choice of population produces either noise or blind spots. Two decisions shape a useful baseline: what entity it is scoped to, and where the notion of normal is drawn from.

Scope it to the application

An application has a characteristic behavior. A browser spawns a known set of helper processes and connects to a wide but structured range of destinations. A backup agent touches many files but talks to a narrow set of hosts. A line-of-business application connects to one server on one port. Scoping the baseline to the application, keyed on a stable identity such as its signing publisher and product rather than its file path, gives each application its own profile of what is ordinary for it. A deviation is then judged against the right reference: not "is this unusual for the machine" but "is this unusual for this application."

Draw normal from the fleet, not from one host

The stronger design choice is to learn normal for an application across the whole fleet of agents that run it, rather than from a single host's own history. Per-host history has three weaknesses. It is unavailable on a new host until that host has run long enough to learn itself. It bakes in whatever the host was already doing, including a compromise that predates the baseline. And its notion of unusual is weak, because one machine's limited experience is a small sample.

A fleet baseline addresses all three. It is available for a new host on day one, because the application's normal is already known from everywhere else it runs. It is hard for one or a few compromised hosts to move, especially with robust statistics. And it produces a much stronger signal, because the denominator is large: a child process seen on a fraction of a percent of the fleet agents that run an application is clearly unusual in a way that "this one host has not seen it before" never is. The observation that a browser spawns a command shell on a tiny share of agents is a far better lead than the same fact measured against a single machine.

There is a precedent for this already in the rule layer. The rule that flags a system daemon connecting on an unexpected port carries a hand-written table of expected ports per daemon. That table is a static, manually maintained baseline for a handful of processes. A learned fleet baseline is the generalization of that idea to every application, kept current automatically instead of by hand.

What to baseline, and what not to

The temptation is to baseline everything the kernel emits. That produces noise, because most kernel activity on a busy host is high-volume and uninteresting, and it produces false positives on every legitimate change. A useful baseline is selective, covering a small set of dimensions where "new for this application" genuinely tends to mean "worth a look":

• Process lineage. The set of child images an application spawns, and the parents that spawn it. A never-before-seen parent-to-child edge is the general form of the exec-chain rule, without the rule's hardcoded list of shells and downloaders.
• Network profile. The destinations and ports an application connects to. This is the generalization of the per-daemon port table.
• File activity shape. The rate and breadth of file writes, which distinguishes an application's normal I/O from a mass-encryption or mass-exfiltration burst.
• Persistence surface. New autoruns, services, or scheduled tasks, drawn from periodic inventory snapshots rather than a live event stream.

Raw file paths, raw registry values, and full URLs are deliberately not baselined. They are high-cardinality and often sensitive, and they add noise rather than signal. Sensitive categorical values that are useful in aggregate can be hashed before they are stored, so the baseline records that a value is unusual without recording the value.

Scoring a deviation

Two kinds of features call for two kinds of comparison. For categorical features, such as which child processes an application spawns, the measure is rarity: a value seen on a large share of the fleet is normal, and a value seen on a small share, or never seen at all, is the signal. For numeric features, such as file-write rate, the measure is distance from the fleet's typical value, expressed with robust statistics so a few outliers do not distort the reference. Only the suspicious direction counts: a new destination or an elevated write rate is worth flagging, while the absence of an expected behavior generally is not.

Whatever the measure, a deviation has to be explainable. A useful alert says that an application on an agent did something specific, and states the fleet norm it departed from, for example that a given child process appears on a small percentage of fleet agents running that application, or that an observed rate sits far above the fleet's typical range. An unexplained anomaly score is not actionable; a stated deviation from a stated norm is.

Baselines complement rules, they do not replace them

A deviation from normal is a weaker signal than a rule match, because normal legitimately changes. An application is updated and starts spawning a new helper. A new admin tool is rolled out. A patch changes a service's network behavior. Each of these looks like a deviation and none is malicious. Treated as a standalone alarm over raw events, a baseline generates false positives on ordinary change.

The resolution is to treat a deviation as a lead whose weight rises sharply when it agrees with other evidence. A novel child process on its own is a low-severity note. The same novel child process on a host that also just triggered a shadow-copy-deletion rule is a strong signal. This is the same correlation logic the ransomware kill-chain meta-rule already uses, where two independent stages occurring together on one process mean far more than either alone. A baseline is most valuable as a layer that adds coverage for novel behavior and escalates when it coincides with a rule, not as a replacement for the rules underneath it.

The constraint: one tenant's behavior must never influence another's

A fleet baseline in a multi-tenant product raises a question that a single-organization tool does not have to answer: whose fleet. The constraint is firm. An application's behavior in one organization must never influence a baseline, a score, a detection, or a response in another organization. "Fleet" means the organization's own agents, and nothing else.

Enforcing this is a matter of scoping every baseline, every score, and every alert to a single organization, under the same row-level isolation that separates all other tenant data. It rules out an appealing shortcut: a shared, cross-organization model that would let a small customer benefit from behavior observed across the whole customer base. That shortcut is off the table, because it would let one tenant's behavior shape another tenant's security outcome. The cost is that a small organization, with few agents running a given application, may not have enough of its own data to form a baseline for it, and simply gets no baseline there rather than borrowing one. That is the correct trade. Coverage is not worth crossing the isolation boundary.

Why baselines are hard, and the controls that make them usable

The honest summary is that baselines are harder to operate than rules, and most of the engineering is in keeping them from crying wolf. The main hazards and the controls for them:

• Normal is not stationary. Legitimate change looks like deviation. Controls: weight recent behavior so the baseline tracks drift, and suppress deviations that coincide with a recorded change such as a patch, an install, or a maintenance window.
• Cold start. A baseline is meaningless until it has enough support. Control: do not score a feature until a minimum number of agents and observations back it, and collect quietly until then.
• Poisoning. An attacker who is patient can try to train malicious behavior into the baseline as normal. Controls: robust estimators, a diversity requirement across agents, and excluding behavior that already tripped a rule from the learning set, so an attack cannot teach the baseline to ignore it.
• Tuning before alerting. A new baseline should run in an observe-only mode first, producing candidate deviations that are logged and reviewed but not alerted, so thresholds can be set against real data before anyone is paged.

Frequently asked questions

What is the difference between signature-based and behavioral detection?

Signature and threshold detection matches known-bad shapes, such as a specific command line or a file-rename burst above a count. It is precise and low-noise but only catches patterns someone has already described. Behavioral baselining learns what is normal for an entity and flags deviation, which can catch novel behavior that matches no known pattern, at the cost of more tuning and more false positives. Practical tooling uses both.

What is a behavioral baseline in endpoint security?

A model of normal behavior for some entity, such as an application, built from observed telemetry, against which new behavior is compared. When an application does something outside its baseline, for example connecting to a destination it never uses, the deviation is a candidate signal. The baseline is only as good as the definition of normal it is built from.

Why baseline application behavior across a fleet instead of per host?

A per-host baseline is unavailable until the host has learned itself, bakes in any pre-existing compromise, and draws on a small sample. A baseline across many agents running the same application is available for a new host immediately, is hard for a few compromised hosts to move, and gives a stronger signal, because a behavior seen on a fraction of a percent of the fleet is clearly unusual.

Can behavioral baselining replace detection rules?

No. A deviation on its own is a lead rather than a verdict, because normal legitimately changes when software is patched or installed. Rules stay precise for known-bad behavior. The strongest signal is a deviation that coincides with a rule hit on the same process, so baselining is best used as a complementary layer.

Where ET Ducky stands

ET Ducky ships the rule layer today: a cross-platform behavioral engine over ETW and eBPF, with a set of rules for exec chains, mass file activity, reverse shells, privilege escalation, unusual outbound traffic, and the ransomware kill chain, enabled per organization. The per-application fleet baseline described here now ships as the cloud-side fleet-baseline-deviation rule, default-off and run in shadow mode first so an organization can review what would have fired before enabling it: learned strictly within each organization, scored against what is normal for that organization's own fleet, and most valuable where a deviation lines up with a rule the engine already trusts.