A Review of Deep Learning Approaches for Vehicle Classification and Tracking in Diverse Conditions
DOI:
https://doi.org/10.29304/jqcsm.2025.17.22196Keywords:
Vehicle Classification, Real-time Tracking, CNN, Computer Vision, Traffic Monitoring, Intelligent Transportation Systems (ITS)Abstract
Vehicle classification and tracking are fundamental components of intelligent transportation systems (ITS), enabling real-time traffic monitoring, surveillance, and autonomous driving. This paper provides a comprehensive review of modern deep learning and artificial intelligence methods used in vehicle classification and tracking. It highlights key methods such as CNNs, YOLO, Faster R-CNN, Kalman Filters, DeepSORT, and LSTM networks, with a particular focus on hybrid CNN-LSTM models that combine spatial and temporal features for robust performance in dynamic scenes. The review identifies major challenges reported in the literature, including occlusion, adverse weather conditions, real-time processing requirements, and data privacy concerns. It also outlines the most common application scenarios, such as smart surveillance, urban traffic control, and autonomous navigation. Based on the current trends, the paper recommends future directions involving vision transformers, reinforcement learning, edge computing, and multimodal sensor fusion. The goal is to offer researchers and practitioners a structured overview of the state-of-the-art, while highlighting opportunities for improving adaptability, scalability, and efficiency in smart transportation systems.
Downloads
References
Sharma, D. and Z.A. Jaffery, Categorical vehicle classification and tracking using deep neural networks. International Journal of Advanced Computer Science and Applications, 2021. 12(9).
Tan, Z., Vehicle classification with deep learning. Master's thesis, Fırat University, Institute of Science, Elazığ, 63p, 2019.
Neupane, B., T. Horanont, and J. Aryal, Real-time vehicle classification and tracking using a transfer learning-improved deep learning network. Sensors, 2022. 22(10): p. 3813.
Rad, R. and M. Jamzad, Real time classification and tracking of multiple vehicles in highways. Pattern Recognition Letters, 2005. 26(10): p. 1597-1607.
Mostafa, M., et al. Automatic Vehicle Classification and Speed Tracking. in 2023 2nd International Conference on Applied Artificial Intelligence and Computing (ICAAIC). 2023. IEEE.
Faruque, M.O., H. Ghahremannezhad, and C. Liu, Vehicle classification in video using deep learning. Machine Learning and Data Mining in Pattern Recognition, MLDM, 2019: p. 117-131.
Azimjonov, J. and A. Özmen, A real-time vehicle detection and a novel vehicle tracking systems for estimating and monitoring traffic flow on highways. Advanced Engineering Informatics, 2021. 50: p. 101393.
Liu, S., et al., Vehicle tracking by detection in UAV aerial video. Science China. Information Sciences, 2019. 62(2): p. 24101.
Xu, Y. and J. Wang, A unified neural network for object detection, multiple object tracking and vehicle re-identification. arXiv preprint arXiv:1907.03465, 2019.
Farhodov, X., et al., LSTM network with tracking association for multi-object tracking. Journal of Korea Multimedia Society, 2020. 23(10): p. 1236-1249.
Velazquez-Pupo, R., et al., Vehicle detection with occlusion handling, tracking, and OC-SVM classification: A high performance vision-based system. Sensors, 2018. 18(2): p. 374.
Ahn, H. and Y.-H. Lee, Vehicle classification and tracking based on deep learning. Journal of web engineering, 2022. 21(4): p. 1283-1294.
Saha, R., T. Debi, and M.S. Arefin. Developing a framework for vehicle detection, tracking and classification in traffic video surveillance. in Intelligent Computing and Optimization: Proceedings of the 3rd International Conference on Intelligent Computing and Optimization 2020 (ICO 2020). 2021. Springer.
Shokravi, H., et al., A review on vehicle classification and potential use of smart vehicle-assisted techniques. Sensors, 2020. 20(11): p. 3274.
Bathija, A. and G. Sharma, Visual object detection and tracking using yolo and sort. International Journal of Engineering Research Technology, 2019. 8(11): p. 345-355.
Kumar, S., et al., Fusion of deep sort and Yolov5 for effective vehicle detection and tracking scheme in real-time traffic management sustainable system. Sustainability, 2023. 15(24): p. 16869.
Gholamhosseinian, A. and J. Seitz, Vehicle classification in intelligent transport systems: An overview, methods and software perspective. IEEE Open Journal of Intelligent Transportation Systems, 2021. 2: p. 173-194.
Wu, J., et al., Automatic vehicle classification using roadside LiDAR data. Transportation Research Record, 2019. 2673(6): p. 153-164.
Li, T. and R. Stern, Car-following-response-based vehicle classification via deep learning. Journal on Autonomous Transportation Systems, 2024. 1(1): p. 1-23.
Wang, Y. and H.A. Karimi, Advanced deep learning models and algorithms for spatial-temporal data, in Big Data. 2024, CRC Press. p. 227-253.
Endo, T., Analysis of Conventional Feature Learning Algorithms and Advanced Deep Learning Models. Journal of Robotics Spectrum, 2023. 1: p. 001-012.
Janiesch, C., P. Zschech, and K. Heinrich, Machine learning and deep learning. Electronic markets, 2021. 31(3): p. 685-695.
Pramanik, A., et al., Granulated RCNN and multi-class deep sort for multi-object detection and tracking. IEEE Transactions on Emerging Topics in Computational Intelligence, 2021. 6(1): p. 171-181.
Jahan, N., S. Islam, and M.F.A. Foysal. Real-time vehicle classification using CNN. in 2020 11th International Conference on Computing, Communication and Networking Technologies (ICCCNT). 2020. IEEE.
Zuraimi, M.A.B. and F.H.K. Zaman. Vehicle detection and tracking using YOLO and DeepSORT. in 2021 IEEE 11th IEEE Symposium on Computer Applications & Industrial Electronics (ISCAIE). 2021. IEEE.
Wang, X., et al., Real-time vehicle type classification with deep convolutional neural networks. Journal of Real-Time Image Processing, 2019. 16: p. 5-14.
Khalkhali, M.B., A. Vahedian, and H.S. Yazdi, Vehicle tracking with Kalman filter using online situation assessment. Robotics and Autonomous Systems, 2020. 131: p. 103596.
Farahi, F. and H.S. Yazdi, Probabilistic Kalman filter for moving object tracking. Signal Processing: Image Communication, 2020. 82: p. 115751.
Sukkar, M., et al., Enhancing pedestrian tracking in autonomous vehicles by using advanced deep learning techniques. Information, 2024. 15(2): p. 104.
Azhar, M.I.H., et al. People tracking system using DeepSORT. in 2020 10th IEEE international conference on control system, computing and engineering (ICCSCE). 2020. IEEE.
Lin, L., et al., Realtime vehicle tracking method based on YOLOv5+ DeepSORT. Computational Intelligence and Neuroscience, 2023. 2023(1): p. 7974201.
Xiao, H., et al., An improved LSTM model for behavior recognition of intelligent vehicles. IEEE Access, 2020. 8: p. 101514-101527.
Hussain, L.A., et al., A predictive target tracking framework for IoT using CNN–LSTM. Internet of Things, 2023. 22: p. 100744.
Mead, M.A., Hybrid CNN and LSTM Model (HCLM) for Short-Term Traffic Volume Prediction. International Journal of Intelligent Computing and Information Sciences, 2022. 22(4): p. 51-61.
Chougule, A., et al., HybridSecNet: in-vehicle security on controller area networks through a hybrid two-step LSTM-CNN model. IEEE Transactions on Vehicular Technology, 2024.
Kothuri, S.R., et al. Hybrid CNN-LSTM machine learning algorithm for driver distraction detection. in 2024 International Conference on Integrated Circuits and Communication Systems (ICICACS). 2024. IEEE.
Ni, X. and H. Huttunen, Vehicle attribute recognition by appearance: computer vision methods for vehicle type, make and model classification. Journal of Signal Processing Systems, 2021. 93: p. 357-368.
Kim, H., Multiple vehicle tracking and classification system with a convolutional neural network. Journal of Ambient Intelligence and Humanized Computing, 2022. 13(3): p. 1603-1614.
Liu, B., D. Zhao, and H. Zhang, Road classification using 3D LiDAR sensor on vehicle. Measurement Science and Technology, 2023. 34(6): p. 065201.
Zhao, Z., et al., Point cloud features-based kernel SVM for human-vehicle classification in millimeter wave radar. IEEE Access, 2020. 8: p. 26012-26021.
Kanona, M.E., et al., A machine learning based vehicle classification in forward scattering radar. IEEE Access, 2022. 10: p. 64688-64700.
Nampally, R., Leveraging AI in Urban Traffic Management: Addressing Congestion and Traffic Flow with Intelligent Systems. Journal of Artificial Intelligence and Big Data, 2021. 1(1): p. 86-99.
Kumar, A., et al. Navigating Urban Mobility: A Review of AI-Driven Traffic Flow Management in Smart Cities. in 2024 11th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions)(ICRITO). 2024. IEEE.
Kumar, N. and M. Raubal, Applications of deep learning in congestion detection, prediction and alleviation: A survey. Transportation Research Part C: Emerging Technologies, 2021. 133: p. 103432.
Niharika, R., et al. Voice Controlled Moving Robot for Smart Surveillance. in 2024 5th International Conference on Circuits, Control, Communication and Computing (I4C). 2024. IEEE.
Avinashiappan, A., et al., Smart surveillance using deep learning. International Journal of Reconfigurable and Embedded Systems, 2023. 12(3): p. 423.
Mahajan, Y., et al., Smart CCTV Surveillance System with Visitor Management System, Face Detection, Fire Alarm, Parking Assistant for Residential Areas. International Journal for Research in Applied Science and Engineering Technology 2023. 11(11): p. 1340-1345.
Valera, Á., et al., Navigation of autonomous light vehicles using an optimal trajectory planning algorithm. Sustainability, 2021. 13(3): p. 1233.
Lodi, M., S. Sarkar, and A. Dadhich. Autonomous Drone Navigation in Dynamic and Uncertain Environments. in 2024 15th International Conference on Computing Communication and Networking Technologies (ICCCNT). 2024. IEEE.
Yang, R., Y. Tang, and G. Xiong. Autonomous Navigation Method Based on Deep Reinforcement Learning with Dual-Layer Perception Mechanism. in 2024 IEEE International Conference on Unmanned Systems (ICUS). 2024. IEEE.
Almeqbaali, N.N.A., et al. Parking and Travel Ticket Automation–Recent Research Solutions. in 2024 15th Annual Undergraduate Research Conference on Applied Computing (URC). 2024. IEEE.
Long, J., Qu, G., & Shi, S. , New parking toll collector management system. 2019.
Yang, D., Lin, G., Li, G., Song, Y., & Fang, Z., Automatic toll collection system and ETC-based intelligent mechanical garage system. 2020.
Kumar, N., D. Acharya, and D. Lohani, An IoT-based vehicle accident detection and classification system using sensor fusion. IEEE Internet of Things Journal, 2020. 8(2): p. 869-880.
Pemila, M., et al., Real time classification of vehicles using machine learning algorithm on the extensive dataset. IEEE Access, 2024.
Sekhar, G.B., et al. Vehicle Tracking and Speed Estimation Using Deep Sort. in 2022 Second International Conference on Interdisciplinary Cyber Physical Systems (ICPS). 2022. IEEE.
Hassaballah, M., et al., Vehicle detection and tracking in adverse weather using a deep learning framework. IEEE transactions on intelligent transportation systems, 2020. 22(7): p. 4230-4242.
Akita, T. and S. Mita. Object tracking and classification using millimeter-wave radar based on LSTM. in 2019 IEEE Intelligent Transportation Systems Conference (ITSC). 2019. IEEE.
Lin, C.-J., S.-Y. Jeng, and H.-W. Lioa, A Real‐Time Vehicle Counting, Speed Estimation, and Classification System Based on Virtual Detection Zone and YOLO. Mathematical Problems in Engineering, 2021. 2021(1): p. 1577614.
Komorkiewicz, M., et al., Intelligent Data Handling in Current and Next-Generation Automated Vehicle Development—A Review. IEEE Access, 2023. 11: p. 32061-32072.
Vaswani, A., et al., Attention is all you need. Advances in neural information processing systems, 2017. 30.
Maurício, J., I. Domingues, and J. Bernardino, Comparing vision transformers and convolutional neural networks for image classification: A literature review. Applied Sciences, 2023. 13(9): p. 5521.
Bhojanapalli, S., et al. Understanding robustness of transformers for image classification. in Proceedings of the IEEE/CVF international conference on computer vision. 2021.
Medina, M.C.C., B.J. Fernandes, and P.V. Barros, The use of reinforcement learning algorithms in object tracking: A systematic literature review. Neurocomputing, 2024: p. 127954.
Zhang, D., et al., Deep reinforcement learning for visual object tracking in videos. arXiv preprint arXiv:1701.08936, 2017.
Mozaffari, S., et al., Deep learning-based vehicle behavior prediction for autonomous driving applications: A review. IEEE Transactions on Intelligent Transportation Systems, 2020. 23(1): p. 33-47.
Ali, K.S., Edge AI and IoT: Direct integration for on-the-device data processing. Advances in Engineering Innovation, 2023. 5: p. 20-23.
Chenna, D., EdgeAI in Self-Sustaining Systems With AI and IoT. 2024.
Roheda, S., H. Krim, and B.S. Riggan, Robust multi-modal sensor fusion: An adversarial approach. IEEE Sensors journal, 2020. 21(2): p. 1885-1896.
Wang, H., et al., Recent progress on artificial intelligence-enhanced multimodal sensors integrated devices and systems. Journal of Semiconductors, 2025. 46(1): p. 011610.
Chen, L., et al. Multimodal sensor fusion for real-time location-dependent defect detection in laser-directed energy deposition. in International Design Engineering Technical Conferences and Computers and Information in Engineering Conference. 2023. American Society of Mechanical Engineers.
Wan, X., et al., Tracking beyond detection: learning a global response map for end-to-end multi-object tracking. IEEE Transactions on Image Processing, 2021. 30: p. 8222-8235.
Lei, M., et al., End-to-end network for pedestrian detection, tracking and re-identification in real-time surveillance system. Sensors, 2022. 22(22): p. 8693.
Zodage, P., et al., Explainable AI (XAI): History, basic ideas and methods. International Journal of Advanced Research in Science, Communication and Technology, 2024. 3.
Mathrani, R., Challenges and Innovations in Achieving Sustainable Computing: A Comprehensive Analysis. energy. 20305: p. 6.
Sustainable Computing—An Overview. 2022.
Sustainable computing. IEEE Access, 2024. 46(3).
Pandey, A., Mannepalli, P. K., Gupta, M., Dangi, R., & Choudhary, G. (2024). A deep learning-based hybrid CNN-LSTM model for location-aware web service recommendation. Neural Processing Letters, 56(5), 234.
Liu, X., & Rodelas, N. C. (2024). Vehicle Detection and Tracking Techniques Based on Deep Learning in Road Traffic Surveillance. [Journal/Conference Name if available], [Volume(Issue)], pages.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Zahraa Hussain Jumah, Adala M. Chyaid
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
