MSc THESIS · UvA SNE × KPMG CYBER

The clean-up step that
broke the classifier

An obvious fix that quietly erases the evidence.

Stable-Edge Filtering for Passive OT Device Classification under Operational Change
Jonathan van den Heuvel · jvdh.tech

THE PROBLEM

We classify what we can't scan

Passive OT asset discovery is a routine KPMG deliverable.
You can't active-scan a live plant, so we classify devices from captured traffic.
But real segments are messy: laptops connect, scanners sweep, maintenance happens.

Does a simple clean-up step keep the classifier robust under that change?

THE IDEA + THE LAB

Keep only the connections that last

Build the graph of who-talks-to-whom, then drop edges that don't persist over time.
Intuition: a one-off engineering session or a scan sweep is noise, filter it out.

THE TESTBED

20: HOSTS
5: CLASSES · ctrl · sup · eng · hist · it
4: CHANGE SCENARIOS
10×10: LAB × MODEL SEEDS
∴: INDUCTIVE (held-out hosts) · GraphSAGE

builds on Heo & Shin (2025) · artifacts released

The idea is almost too obvious to argue with. We build a graph of who talks to whom, and the model classifies devices from that pattern of connections. Before classifying, we throw away edges that don't persist across time windows, call it stable-edge filtering. The intuition: a transient connection, like a one-off engineering session or a scan sweep, is just noise, so clean it out and the picture should get sharper. To test it credibly I built a real OT lab: twenty hosts across five device classes, one passive capture point on a shared segment, four scripted operational-change scenarios with ground truth. Crucially I evaluated on held-out hosts the model never trained on, averaged over ten lab seeds times ten model seeds, and released the code and scenarios. Builds on Heo and Shin, who did this only on clean traces.

BUILDING THE LAB · THE FACTORY

What the devices actually do

real roles → real traffic → the who-talks-to-whom graph the classifier reads

BUILDING THE LAB · THE TESTBED

20 containers, one passive tap

A small plant in Docker: 5 device classes, 4 each, on one shared segment.
Every container runs the same image, and its class decides whether it serves or polls.

# one image; class set in lab.yaml
dev = DEVICE_CLASSES[host.class]
dev.setup()        # PLC: serves :502
while running:
    phase = read("phase.json")
    for plc in dev.targets:  # client
        poll(plc)   # HMI · hist · eng
    sleep(dev.poll_interval)

RUNS 24/7 on a server · 78 scenario runs · 24 GB captured

A quick word on the testbed, because the result only means something if the lab is honest. This is twenty Docker containers standing in for a small plant: five device classes, four of each, the controllers, plus the HMIs, historians and engineering stations that poll them, and a few IT endpoints. They all share one network segment, watched by a single passive tap, exactly the position we’re in on a client site. Every container runs the same image; what makes one behave like a PLC and another like an HMI is just its class in a config file. The controller serves Modbus and S7; everyone else polls it on its own schedule. And this isn’t a laptop demo, it runs around the clock on a real server, and it has already produced seventy-odd scenario runs and about twenty-four gigabytes of captured traffic.

BUILDING THE LAB · OPERATIONAL CHANGE

Four scripted changes,
known ground truth

A phase signal tells every container when the world changes, live, mid-capture.
So I can perturb the network and still know the truth.

for phase in scenario.phases:
    write("phase.json", phase.effects)
    sleep(phase.minutes) # live
labels = label_edges(config)
# truth from CONFIG, not the traffic
# "stable" iff seen in EVERY phase

FIVE STATES · FOUR CHANGES

steady: nothing changes · the baseline
maintenance: plc-1 paused 40 min, its edges vanish→ this is the one that breaks
onboarding: a new HMI joins mid-run
drift: an engineer repoints to another PLC
noise: a scanner sweeps the segment

edge labels come from config, not traffic, no circularity

RQ2 · THE RESULT (held-out macro-F1)

Neutral on four.
It breaks the fifth.

Neutral on 4 of 5 scenarios: steady, onboarding, config drift, benign scanning.
No robustness benefit anywhere, the upside we hoped for simply isn't there.
Under maintenance it significantly hurts.

0.45 → 0.36 −0.089 · p = 0.027 · 8/10 runs worse

THE MECHANISM · MAINTENANCE

A paused controller looks like an idle laptop

1A controller (PLC) is paused for 40 minutes.
2Its polls stop, the “keep persistent edges” filter deletes all 20. in-degree 20→0 · in-bytes 2.1M→0
3Stripped bare, it has zero traffic, identical to an idle IT laptop → misread as “IT endpoint.”

A plain random forest breaks the same way, the filter destroys the evidence, not the model.

Let me tell you exactly how it breaks, because the mechanism is the lesson. During maintenance, one controller, a PLC, is paused for forty minutes. While it's paused, all the polls coming into it stop. The filter sees those connections vanish, decides they aren't persistent, and deletes them, all twenty. In-degree goes from twenty to zero, incoming bytes from 2.1 million to zero. But here's the thing: those polls are exactly what make a controller look like a controller. Strip them away and the paused PLC has no connections and no traffic, indistinguishable from an idle IT laptop. So the classifier confidently calls it an IT endpoint, and the controller-to-IT leak jumps from 0.08 to 0.30, watch the cell. And this isn't a quirk of the fancy graph model: a plain random forest with no graph at all breaks the identical way. The filter didn't confuse the model, it destroyed the evidence, for any model.

THE TAKEAWAY

Persistence is the wrong thing to filter on.

A cheap clean-up step can quietly erase the evidence your classifier depends on.

Don't pre-filter passive OT graphs by recurrence.
Edge meaning beats persistence, think protocol, direction, endpoint roles.

Reproducible lab · 4 scenarios · code released | jvdh.tech · Jonathan van den Heuvel · github.com/jonathanvdheuvel