Hybrid fraud detection model for financial institutions.

Abstract

Digitization has significantly expanded access to financial services while simultaneously increasing the exposure of financial platforms to fraud. Institutions globally are adopting machine learning (ML) to detect emerging fraud patterns, yet many detection systems remain evaluated almost exclusively through predictive metrics, limiting their reliability in real-world adversarial environments. This study develops and evaluates a hybrid fraud detection framework that integrates supervised ensemble learning with unsupervised anomaly detection, addressing both predictive performance and operational security gaps. The research uses the publicly available IEEE-CIS Fraud Detection Dataset from Kaggle, comprising over 590,000 transactions with both transaction and identity attributes. Data preprocessing followed a leakage-safe protocol that included temporal splitting based on TransactionDT (80% training, 20% validation), GroupKFold cross-validation using customer-identity features, and feature-engineering techniques restricted to the training folds to prevent leakage. The supervised layer consists of a stacked ensemble combining Random Forest, LightGBM and XGBoost as base learners with an XGBoost meta- model trained on out-of-fold predictions. To complement the supervised layer, an Isolation Forest anomaly-gating mechanism was incorporated to detect out-of-distribution and potentially adversarial transactions. SHAP explainability was integrated to generate local and global feature attributions, improving operational transparency for Security Operations Center (SOC) analysts. Model performance was evaluated using standard fraud-detection metrics including AUC-ROC, precision, recall and F1-score. The hybrid model achieved an AUC-ROC of 0.904, outperforming the baseline single-model learners implemented during experimentation. It also achieved PR-AUC of 0.5192 on the temporal validation set and at the F1-optimised threshold of 0.2661 precision was 0.6360, the recall was at 0.4446 while the F1-Score was at 0.5234. SHAP explanations revealed the dominant influence of identity-linked features and amount-related attributes which enabled clearer interpretation. STRIDE threat modeling was applied to assess the cybersecurity posture of the full pipeline. The analysis identified vulnerabilities related to spoofing, tampering, information disclosure, and denial-of-service, highlighting risks typically overlooked in predictive-only evaluation frameworks. Adversarial-resilience tests showed that anomaly gating improved robustness by filtering suspicious inputs, although this introduced a measurable reduction in recall for borderline fraud cases. Despite this trade-off, the integrated framework demonstrated a balanced blend of predictive performance, interpretability, and security-oriented evaluation. The study concludes that hybrid architectures enriched with threat modeling and explainability offer a more realistic assessment of fraud detection systems operating in adversarial environments, making the proposed framework suitable for further adaptation within financial institutions.