MSc THESIS · UvA SNE × KPMG CYBER

The clean-up step that
broke the classifier

An obvious fix that quietly erases the evidence.

Stable-Edge Filtering for Passive OT Device Classification under Operational Change
Jonathan van den Heuvel · jvdhthesis.tech

THE PROBLEM

We classify what we can't scan

Passive OT asset discovery is a routine KPMG deliverable.
You can't active-scan a live plant — so we classify devices from captured traffic.
But real segments are messy: laptops connect, scanners sweep, maintenance happens.

Does a simple clean-up step keep the classifier robust under that change?

THE IDEA + THE LAB

Keep only the connections that last

Build the graph of who-talks-to-whom, then drop edges that don't persist over time.
Intuition: a one-off engineering session or a scan sweep is noise — filter it out.

THE TESTBED

20: HOSTS
5: CLASSES · ctrl · sup · eng · hist · it
4: CHANGE SCENARIOS
10×10: LAB × MODEL SEEDS
∴: INDUCTIVE (held-out hosts) · GraphSAGE

builds on Heo & Shin (2025) · artifacts released

The idea is almost too obvious to argue with. We build a graph of who talks to whom, and the model classifies devices from that pattern of connections. Before classifying, we throw away edges that don't persist across time windows — call it stable-edge filtering. The intuition: a transient connection, like a one-off engineering session or a scan sweep, is just noise, so clean it out and the picture should get sharper. To test it credibly I built a real OT lab: twenty hosts across five device classes, one passive capture point on a shared segment, four scripted operational-change scenarios with ground truth. Crucially I evaluated on held-out hosts the model never trained on, averaged over ten lab seeds times ten model seeds, and released the code and scenarios. Builds on Heo and Shin, who did this only on clean traces.

RQ2 · THE RESULT (held-out macro-F1)

Neutral on four.
It breaks the fifth.

Neutral on 4 of 5 scenarios: steady, onboarding, config drift, benign scanning.
No robustness benefit anywhere — the upside we hoped for simply isn't there.
Under maintenance it significantly hurts.

0.45 → 0.36 −0.089 · p = 0.027 · 8/10 runs worse

THE MECHANISM · MAINTENANCE

A paused controller looks like an idle laptop

1A controller (PLC) is paused for 40 minutes.
2Its polls stop — the “keep persistent edges” filter deletes all 20. in-degree 20→0 · in-bytes 2.1M→0
3Stripped bare, it has zero traffic — identical to an idle IT laptop → misread as “IT endpoint.”

A plain random forest breaks the same way — the filter destroys the evidence, not the model.

Let me tell you exactly how it breaks, because the mechanism is the lesson. During maintenance, one controller — a PLC — is paused for forty minutes. While it's paused, all the polls coming into it stop. The filter sees those connections vanish, decides they aren't persistent, and deletes them — all twenty. In-degree goes from twenty to zero, incoming bytes from 2.1 million to zero. But here's the thing: those polls are exactly what make a controller look like a controller. Strip them away and the paused PLC has no connections and no traffic — indistinguishable from an idle IT laptop. So the classifier confidently calls it an IT endpoint, and the controller-to-IT leak jumps from 0.08 to 0.30 — watch the cell. And this isn't a quirk of the fancy graph model: a plain random forest with no graph at all breaks the identical way. The filter didn't confuse the model — it destroyed the evidence, for any model.

THE TAKEAWAY

Persistence is the wrong thing to filter on.

A cheap clean-up step can quietly erase the evidence your classifier depends on.

Don't pre-filter passive OT graphs by recurrence.
Edge meaning — protocol, direction, endpoint roles — beats persistence.

Reproducible lab · 4 scenarios · code released | jvdhthesis.tech · Jonathan van den Heuvel · jonathan50@live.nl

So the deployable lesson — the part to take back to client work: don't blindly pre-filter passive OT graphs by how often a connection recurs. The edges that disappear during an outage aren't noise — they're precisely the ones that define what a device is. Persistence feels like a safe, free clean-up step, and it isn't: it can silently erase the evidence your classifier relies on, and you won't even notice because the pipeline still runs and still outputs a label. If you're going to filter, filter on what an edge means — the protocol, the direction, the roles of the two endpoints — not on how stable it looks over time. The whole lab, the four scenarios, and the code are released, so you can reproduce this on your own captures. Thank you — everything's at jvdhthesis.tech, happy to take questions.