Depth-Based Robust Principal Component Analysis for Anomaly Detection in Streaming Data

Abstract

The challenge of detecting anomalies from streaming data is posed by several issues including concept drifts, strict processing requirements in real time, and robustness against outlier and evolving data distribution. This study, therefore, proposes a Depth-based Robust PCA (DHRPCA) which integrates robust PCA with statistical depth function in high dimensional anomaly detection in streams. Unlike regular RPCA algorithms which require a whole matrix of data, DHRPCA allows updating of the anomaly detection model in an incremental manner through the receipt of new observations. In this regard, the proposed method utilizes MBD and PD in order to compute anomaly scores that are geometrically interpretable and robust, as well as a forgetting factor to deal with concept drift without retraining. F1-scores generated from experiments conducted on both synthetic and Twitter streaming datasets indicate values between 0.84 to 0.91 as opposed to traditional RPCA models whose values range from 0.45 to 0.82. The incremental method ensures low latency values below 25ms per batch of stream, hence suitability in real-time applications such as fraud detection, network monitoring, and industrial sensing.