ShortcutCatcher: Making Traffic Classification Reliable

Machine learning has given encrypted traffic classification a new momentum. Yet, once deployed, models often fail due to hidden shortcut features, i.e., spurious correlations learned from training data that do not hold in new environments. Prior work has shown their negative impact through costly manual intervention. Here, we present ShortcutCatcher, an automated, model-agnostic framework that detects and mitigates shortcuts with the help of explainable AI. The key idea is to contrast model behaviour on two datasets: a large training dataset and a separate verification dataset that differs in scenario but shares the same feature schema. ShortcutCatcher integrates feature explanation with cross-scenario evaluation in a closed loop, iteratively removing those critical features that would not be valid in deployment. Across multiple encrypted traffic classification tasks and model architectures, ShortcutCatcher uncovers shortcut dependencies and improves cross-scenario generalisation, up to three times over standard training. In addition, ShortcutCatcher exposes dataset limitations where collection artefacts act as silent shortcuts that have gone so far unnoticed, allowing us to finally expose realistic performance without assuming that the underlying task is intrinsically easy