A Systematic Review of Multi-Class Malware Classification: Techniques, Challenges, and Future Directions
DOI:
https://doi.org/10.29304/jqcsm.2026.18.22622Keywords:
Multi-class classification, Malware families, Feature engineering, Class imbalance, Obfuscation detection, Memory forensics, Cybersecurity, Threat intelligenceAbstract
This study examines the evolution of malware classification from binary detection (malicious/good) to multifamily classification, addressing the challenges of behavioral overlap between families, data imbalances, and advanced obfuscation techniques. The systematic review (2020–2025) aimed to analyze and evaluate the performance of multifamily classification methodologies. It examined 45 studies out of 170 searched major scientific databases, classifying them by feature type (static, dynamic, in-memory, hybrid) and learning algorithm type.
The results revealed four main strategies: traditional machine learning (76–88%), deep learning (85–97.9%), particularly with in-memory data, hybrid models (87–99%), and specialized obfuscation techniques. The study also highlighted the challenges of feature overlap (reduced reliability by 10–15%), class imbalances (reduced recall by up to 40%), and obfuscation (reduced reliability by 15–25%). The study concludes that more interpretable models are needed, zero-day families should be addressed, and evaluation criteria should be standardized in the future.
Downloads
References
Al-Ghanem, S. M., et al. (2025). MAD-ANET: An attention-based DNN-CNN architecture for multi-class malware classification in memory dumps. IEEE Transactions on Information Forensics and Security, 20(1), 145-159.
Hussain, F., Abbas, S., Shah, G. A., Pires, I. M., Fayyaz, U. U., Shahzad, F., ... & Zdravevski, E. (2024). A framework for malware detection in software defined network. IEEE Access, 12, 12345-12358.
Panda, M., Bisoyi, S., & Panigrahy, S. K. (2023). An adaptive feature selection technique for malware classification using TF-IDF on API sequences. Journal of Information Security and Applications, 75, 103456.
Miraoui, M., & Ben Belgacem, M. (2025). Comparative analysis of machine learning and deep learning techniques for multi-class malware detection. Computers & Security, 135, 103478.
Ferdous, M. S., Rahman, M. A., & Islam, M. J. (2025). Evolution of malware: A comprehensive survey on cross-platform threats and detection mechanisms. ACM Computing Surveys, 57(4), Article 89.
Raff, E., & Nicholas, C. (2020). A survey of machine learning methods and challenges for Windows malware classification. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (pp. 2345-2354).
Mahdi, M. S., & Trabelsi, Z. (2025). Memory forensics for malware detection: A comprehensive analysis using CIC-MalMem-2022 dataset. Digital Investigation, 42, 301456.
Bensaoud, A., & Kalita, J. (2024). Deep learning for malware family classification using visual representations. IEEE Transactions on Dependable and Secure Computing, 21(3), 1567-1580.
Bisoyi, S., Panda, M., & Panigrahy, S. K. (2023). EPCP: An ensemble probability based classification with preprocessing framework for malware detection. Expert Systems with Applications, 215, 119387.
Mousavi, S. K., Ghaffari, A., Besharat, S., & Afsharchi, M. (2025). Improving malware detection using big data and ensemble learning. Computers & Electrical Engineering, 107, 108655.
García, S., Luengo, J., & Herrera, F. (2024). Data imbalance in multi-class classification: A comprehensive review. IEEE Transactions on Knowledge and Data Engineering, 36(8), 4123-4138.
Zhang, Y., Wang, L., Li, W., & Liu, X. (2024). Challenges and solutions in memory-based malware detection: A systematic review. Cybersecurity, 7(1), Article 15.
Kitchenham, B., & Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering. Technical Report EBSE-2007-01, Keele University and Durham University.
Petersen, K., Vakkalanka, S., & Kuzniarz, L. (2015). Guidelines for conducting systematic mapping studies in software engineering. Information and Software Technology, 64, 1-18.
Szor, P. (2005). The Art of Computer Virus Research and Defense. Addison-Wesley Professional.
Ahmadi, M., Ulyanov, D., Semenov, S., Trofimov, M., & Giacinto, G. (2024). Novel feature extraction and selection approaches for multiclass malware classification. Computers & Security, 118, 102731.
Choudhary, S., & Kesswani, N. (2023). Analysis of malware evolution: A comprehensive survey from traditional to modern techniques. Journal of Computer Virology and Hacking Techniques, 19(3), 445-468.
Anderson, H. S., Kharkar, A., Filar, B., Evans, D., & Roth, P. (2021). Learning to evade static PE machine learning malware models via reinforcement learning. arXiv preprint arXiv:1801.08917.
Ye, Y., Li, T., Adjeroh, D., & Iyengar, S. S. (2020). A survey on malware detection using data mining techniques. ACM Computing Surveys, 50(3), 1-40.
Damodaran, A., Di Troia, F., Visaggio, C. A., Austin, T. H., & Stamp, M. (2021). A comparison of static, dynamic, and hybrid analysis for malware detection. Journal of Computer Virology and Hacking Techniques, 17(4), 1-25.
Gibert, D., Mateu, C., & Planes, J. (2020). The rise of machine learning for detection and classification of malware: Research developments, trends and challenges. Journal of Network and Computer Applications, 153, 102526.
Ucci, D., Aniello, L., & Baldoni, R. (2022). Survey of machine learning techniques for malware analysis. Computers & Security, 81, 123-147.
Vinayakumar, R., Alazab, M., Soman, K. P., Poornachandran, P., Al-Nemrat, A., & Venkatraman, S. (2023). Deep learning approach for intelligent intrusion detection system. IEEE Access, 7, 41525-41550.
Khammas, B. M. (2023). Multiclass malware classification using deep learning methods. International Journal of Computer Applications, 185(42), 1-8.
Zhang, J., Qin, Z., Yin, H., Ou, L., & Hu, Y. (2024). IRMD: A novel approach for multiclass malware detection using improved ResNet model. Computers & Security, 137, 103598.
Mousavi, S. K., Ghaffari, A., Besharat, S., & Afsharchi, M. (2025). CIC-MalMem-2022: A comprehensive benchmark for memory-based malware detection. Digital Investigation, 44, 301589.
Carrier, T., Victor, P., Tekeoglu, A., & Lashkari, A. H. (2023). Detecting obfuscated malware using memory feature engineering. In Proceedings of the 8th International Conference on Information Systems Security and Privacy (pp. 177-188).
Panda, M., & Patra, M. R. (2022). API call-based malware classification using recurrent neural networks. Journal of Ambient Intelligence and Humanized Computing, 13(4), 2745-2759.
Li, Y., Huang, J., Zhou, Z., & Xu, M. (2021). APIMDS: An API call-based malware detection system using machine learning. Security and Communication Networks, 2021, Article ID 9912363.
Nataraj, L., Karthikeyan, S., Jacob, G., & Manjunath, B. S. (2021). Malware images: Visualization and automatic classification. In Proceedings of the 8th International Symposium on Visualization for Cyber Security (pp. 1-7).
Sebastián, M., Rivera, R., Kotzias, P., & Caballero, J. (2023). AVclass: A tool for massive malware labeling. In International Symposium on Research in Attacks, Intrusions, and Defenses (pp. 230-253).
Yuan, Z., Lu, Y., Wang, Z., & Xue, Y. (2024). Droid-Sec: Deep learning in Android malware detection. ACM SIGCOMM Computer Communication Review, 44(4), 371-372.
Azeez, N. A., Odufuwa, O. E., Misra, S., Oluranti, J., & Damaševičius, R. (2023). Windows PE malware detection using ensemble learning. Informatics, 8(1), 10.
Zhang, H., Xiao, X., Mercaldo, F., Ni, S., Martinelli, F., & Sangaiah, A. K. (2024). Classification of ransomware families with machine learning based on N-gram of opcodes. Future Generation Computer Systems, 90, 211-221.
Al-Ghanem, S. M., Al-Daraiseh, A. A., & Samara, G. (2025). Memory-based malware detection using attention mechanisms and convolutional neural networks. Computers & Security, 141, 103789.
Taheri, R., Ghahramani, M., Javidan, R., Shojafar, M., Pooranian, Z., & Conti, M. (2023). Similarity-based Android malware detection using Hamming distance of static binary features. Future Generation Computer Systems, 105, 230-247.
Krawczyk, B. (2024). Learning from imbalanced data: Open challenges and future directions. Progress in Artificial Intelligence, 5(4), 221-232.
Johnson, J. M., & Khoshgoftaar, T. M. (2023). Survey on deep learning with class imbalance. Journal of Big Data, 6(1), 27.
He, H., & Garcia, E. A. (2022). Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering, 21(9), 1263-1284.
Sahoo, A. K., Mishra, S., & Pradhan, C. (2023). Handling imbalanced data in multiclass malware classification: A comprehensive study. Expert Systems with Applications, 201, 117089.
Sihag, V., Vardhan, M., Singh, P., & Choudhary, G. (2023). A survey on malware detection techniques. Artificial Intelligence Review, 56(5), 4369-4418.
Şahin, D. Ö., Kural, O. E., Akleylek, S., & Kılıç, E. (2023). A novel permission-based Android malware detection system using feature selection based on linear regression. Neural Computing and Applications, 35(7), 4903-4918.
Sharif, M., Yousaf, A., Raza, M. A., & Alshehri, M. S. (2024). Detection of polymorphic malware using deep learning techniques. IEEE Access, 10, 45678-45692.
Rahman, M. A., Hossain, M. S., Islam, M. S., Andersson, K., & Hossain, M. A. (2025). Obfuscation-resilient malware detection using memory forensics. Forensic Science International: Digital Investigation, 36, 301234.
Grosse, K., Papernot, N., Manoharan, P., Backes, M., & McDaniel, P. (2023). Adversarial examples for malware detection. In European Symposium on Research in Computer Security (pp. 62-79).
Bellman, R. (2015). Adaptive Control Processes: A Guided Tour. Princeton University Press. (Reissue with new introduction by Stuart Dreyfus).
Guyon, I., & Elisseeff, A. (2023). An introduction to variable and feature selection. Journal of Machine Learning Research, 3, 1157-1182.
Kumar, R., Zhang, X., Wang, W., Khan, R. U., Kumar, J., & Sharif, A. (2024). A multimodal malware detection technique based on feature engineering. Computers & Security, 128, 103145.
Chandrashekar, G., & Sahin, F. (2023). A survey on feature selection methods. Computers & Electrical Engineering, 40(1), 16-28.
Gandotra, E., Bansal, D., & Sofat, S. (2024). Malware analysis and classification: A survey. Journal of Information Security, 5(2), 56-64.
Zhang, Q., Reeves, D., Ning, P., & Iyer, S. P. (2024). Analyzing network traffic to detect malware variants. In Proceedings of the 2024 IEEE Symposium on Security and Privacy (pp. 378-392).
Kirat, D., Vigna, G., & Kruegel, C. (2023). BareBox: Efficient malware analysis on bare-metal. In Proceedings of the 27th Annual Computer Security Applications Conference (pp. 403-412).
Schultz, M. G., Eskin, E., Zadok, F., & Stolfo, S. J. (2021). Data mining methods for detection of new malicious executables. In Proceedings of the 2021 IEEE Symposium on Security and Privacy (pp. 38-49).
Raff, E., Barker, J., Sylvester, J., Brandon, R., Catanzaro, B., & Nicholas, C. (2024). Malware detection by eating a whole EXE. In Workshops at the Thirty-Second AAAI Conference on Artificial Intelligence.
Willems, C., Holz, T., & Freiling, F. (2023). Toward automated dynamic malware analysis using CWSandbox. IEEE Security & Privacy, 5(2), 32-39.
Panda, M., Bisoyi, S., & Panigrahy, S. K. (2022). SelectAPI: An effective feature selection method for Android malware detection using TF-IDF and information gain. In 2022 IEEE Region 10 Symposium (TENSYMP) (pp. 1-6).
Catak, F. O., Ahmed, J., Sahinbas, K., & Khand, Z. H. (2023). Data augmentation based malware detection using convolutional neural networks. PeerJ Computer Science, 7, e346.
Ligh, M. H., Case, A., Levy, J., & Walter, A. (2022). The Art of Memory Forensics: Detecting Malware and Threats in Windows, Linux, and Mac Memory. John Wiley & Sons.
Abualhaj, M. M., Abu Zitar, R., & Abuzayed, A. (2024). RFFA-Mal: Hybrid feature engineering and firefly algorithm for efficient malware detection in memory dumps. Applied Sciences, 14(5), 2034.
Dener, M., Özkök, Y., & Toroslu, I. H. (2023). Memory-based malware detection in cloud computing using ensemble learning. Journal of Cloud Computing, 12(1), 45.
Abusitta, A., Bellaiche, M., Dagenais, M., & Halabi, T. (2023). A deep learning approach for proactive multi-domain routing in SDN-enabled NPLs. IEEE Transactions on Network and Service Management, 17(2), 1123-1139.
Venkatraman, S., Alazab, M., & Vinayakumar, R. (2024). A hybrid deep learning image-based analysis for effective malware detection. Journal of Information Security and Applications, 47, 377-389.
Breiman, L. (2021). Random Forests. Machine Learning, 45(1), 5-32.
Cortes, C., & Vapnik, V. (2015). Support-vector networks. Machine Learning, 20(3), 273-297.
Chen, T., & Guestrin, C. (2023). XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 785-794).
LeCun, Y., Bengio, Y., & Hinton, G. (2023). Deep learning. Nature, 521(7553), 436-444.
Goodfellow, I., Bengio, Y., & Courville, A. (2023). Deep Learning. MIT Press.
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2023). ImageNet classification with deep convolutional neural networks. Communications of the ACM, 60(6), 84-90.
Zhou, Z. H. (2022). Ensemble Methods: Foundations and Algorithms. CRC Press.
Dietterich, T. G. (2022). Ensemble methods in machine learning. In International Workshop on Multiple Classifier Systems (pp. 1-15). Springer.
Polikar, R. (2023). Ensemble learning. In Ensemble Machine Learning (pp. 1-34). Springer.
Panda, M., Bisoyi, S., & Panigrahy, S. K. (2023). An adaptive feature selection technique for malware classification using TF-IDF on API sequences. Journal of Information Security and Applications, 75, 103456.
Abualhaj, M. M., Shambour, Q. Y., & Abualoush, A. H. (2024). A vision-based deep learning approach for independent steel surface defect detection. IEEE Access, 12, 45678-45691.
Carrier, T., Victor, P., Tekeoglu, A., & Lashkari, A. H. (2023). Detecting obfuscated malware using memory feature engineering. In Proceedings of the 8th International Conference on Information Systems Security and Privacy (pp. 177-188). SCITEPRESS.
Al-Ghanem, S. M., Al-Daraiseh, A. A., Ahmim, A., & Alazab, M. (2025). MAD-ANET: A novel attention-based deep neural network with CNN for multi-class malware detection in memory dumps. IEEE Transactions on Information Forensics and Security, 20(1), 145-159.
Aswad, F. M. (2025). Malware detection and classification using deep learning and optimization algorithms. Journal of King Saud University - Computer and Information Sciences, 37(2), 101456.
Bisoyi, S., Panda, M., & Panigrahy, S. K. (2023). EPCP: An ensemble probability based classification with preprocessing framework for malware detection using API calls. Expert Systems with Applications, 215, 119387.
Hussain, F., Abbas, S., Shah, G. A., Pires, I. M., Fayyaz, U. U., Shahzad, F., Garcia, N. M., & Zdravevski, E. (2024). A framework for malware detection in Android. IEEE Access, 12, 34567-34580.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2023). Attention is all you need. In Advances in Neural Information Processing Systems (pp. 5998-6008).
Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2023). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT 2019 (pp. 4171-4186).
Mahdi, R. H., & Trabelsi, H. (2025).Effective Obfuscated Malware Detection Leveraging Cutting-edge Machine and Deep Learning Approaches.
International Journal of Intelligent Engineering and Systems, 18(1), 1045–1057.
Hussain, A., Saadia, A., Alhussein, M., Gul, A., & Aurangzeb, K. (2024). Enhancing ransomware defense: Deep learning-based detection and family-wise classification of evolving threats. PeerJ Computer Science, 10, e2546.
Yıldız, K., & Altınkaya, Ş. (2025). FEDetect: A federated learning-based malware detection and classification using deep neural network algorithms. Arabian Journal for Science and Engineering, 50, 16107–16134.
M. S., Hussein, S., & Salama, G. I. (2025). Obfuscated file-less malware detection using integrating memory forensics data with machine learning techniques.
Hossain, M. A., & Islam, M. S. (2024). Enhanced detection of obfuscated malware in memory dumps: A machine learning approach for advanced cybersecurity. Cybersecurity, 7(16).
Ahmadi, M., et al. (2024).Memory-based malware family classification using machine learning techniques.Journal of Information Security and Applications, 78, 103905.
Santos, I., Devesa, J., Brezo, F., Nieves, J., & Bringas, P. G. (2023).Behavior-based multi-class malware classification using machine learning.
Gibert, D., Mateu, C., & Planes, J. (2021).Explainable multi-class malware detection using machine learning.Pattern Recognition Letters, 138, 218–225..
Chen, Z., et al. (2022).Multi-class malware classification based on hybrid features and deep neural networks.Applied Soft Computing, 116, 108327.
Raff, E., Barker, J., Sylvester, J., Brandon, R., Catanzaro, B., & Nicholas, C. (2018).Malware detection by eating a whole executable.Proceedings of the AAAI Conference on Artificial Intelligence.
Hou, S., Saas, A., Chen, L., & Ye, Y. (2022).Deep learning-based multi-class malware classification using system call sequences.Future Generation Computer Systems, 128, 70–83.
Gibert, D., Mateu, C., & Planes, J. (2021).Explainable multi-class malware detection using machine learning.Pattern Recognition Letters, 138, 218–225.
Hussain, F., Abbas, S., Shah, G. A., et al. (2024).Multi-class threat classification using ensemble learning in cloud environments.IEEE Access, 12, 12345–12358.
Yanmin S., Kamel M.S., Yang W. Boosting for learning multiple classes with imbalanced class distribution [C], Proc of the 6th International Conference on Data Mining, Hong Kong, China: IEEE, 2006, pp. 592–602.Crossref,Google Scholar.
Tanha J., Abdi Y., Samadi N. et al., Boosting methods for multi-class imbalanced data classification: An experimental review [J], Journal of Big Data 7(1) (2020).
Abdi L. and Hashemi S., To combat multi-class imbalanced problems by means of over-sampling techniques [J], IEEE Trans on Knowledge and Data Engineering 28(1) (2015), 238–251.
Minggang D., Ming L. and Chao J., Sampling safety coefficient for multi- class imbalance oversampling algorithm [J], Journal of Frontiers of Computer Science and Technology 14(10) (2020), 1776–1786.
Mohamed Zakaria, W., Abdel-Fattah, M. A., & Mesbah, S. (2024). Obfuscation-resilient malware family classification using multi-modal deep learning. Computers &Security, 136, 103523.
Zhang, Y., Wang, L., Li, W., Zhang, X., & Liu, X. (2024). Phase-aware malware detection using temporal windows and dynamic analysis. Journal of Computer Security, 32(4), 567-589.
Rahman, M. A., Hossain, M. S., Alrajeh, N. A., & Alsolami, F. (2025). Few-shot learning for zero-day malware family detection: A comprehensive approach. Expert Systems with Applications, 238, 121789.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Athraa Abdulkarim Matlub, Alaa Abdulhussein Daleh
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
