A Survey on Exploring Multimodal Fusion in Healthcare: Challenges and Solutions
DOI:
https://doi.org/10.29304/jqcsm.2024.16.41810Keywords:
Multimodal Fusion, Healthcare, Deep Learning, Clinical Diagnostics, Multimodal classificationAbstract
Multimodal fusion in healthcare has emerged as a powerful tool for integrating diverse data sources, including medical images, sensor data, and clinical records, to enhance diagnostic accuracy, treatment planning, and patient monitoring. However, existing surveys need more discussion of the technical details of proposed multimodal fusion approaches and their practical applications. Therefore, we propose a comprehensive survey on using multimodal fusion in healthcare applications. It provides an in-depth examination of the current landscape, discussing key developments that have propelled the field forward as well as ongoing challenges that researchers and practitioners continue to overcome in their efforts to realize the full potential of this technology. The survey explores the fundamentals and core concepts underlying multimodal fusion, explores the methods and techniques that have been developed, and examines the broad applications in various healthcare domains. Moreover, it provides a detailed discussion of the multimodal deep learning classification pipeline, highlighting the critical steps involved in pre-processing, feature extraction, information fusion, and model evaluation. Furthermore, the survey critically analyzes the data integration challenges that remain significant obstacles and identifies promising future directions that hold the potential to shape the future of this rapidly evolving field.
Downloads
References
A. W. A. N. D. A. T. A. N. D. R. P. R. A. N. D. P. S. A. N. D. G. Rasool, “Multimodal Data Integration for Oncology in the Era of Deep Neural Networks: A Review,” Jul. 2023. doi: 10.48550/arXiv.2303.
E. Warner et al., “Multimodal Machine Learning in Image-Based and Clinical Biomedicine: Survey and Prospects,” Int J Comput Vis, Apr. 2024, doi: 10.1007/s11263-024-02032-8.
Z. Ch. Oleiwi, E. N. AlShemmary, and S. Al-augby, “Adaptive Features Selection Technique for Efficient Heart Disease Prediction,” Journal of Al-Qadisiyah for Computer Science and Mathematics, vol. 15, no. 1, Feb. 2023, doi: 10.29304/jqcm.2023.15.1.1137.
A. Kline et al., “Multimodal machine learning in precision health: A scoping review,” NPJ Digit Med, vol. 5, no. 1, p. 171, 2022.
C. C. A. N. D. J. W. A. N. D. Y. W. A. N. D. S. Z. A. N. D. Z. A. A. N. D. L. A. C. A. N. D. K. T. W. A. N. D. B. A. L. A. N. D. Y. Huo, “Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Review,” Jul. 2022. doi: 10.48550/arXiv.2203.
M. Yaseen Abdullah, “Optimization Techniques for Human Multi-Biometric Recognition System,” Journal of Al-Qadisiyah for Computer Science and Mathematics, vol. 15, no. 3, Oct. 2023, doi: 10.29304/jqcsm.2023.15.31292.
M. A. Al-Antari, “Artificial Intelligence for Medical Diagnostics—Existing and Future AI Technology!,” Diagnostics, vol. 13, no. 4, p. 688, Feb. 2023, doi: 10.3390/diagnostics13040688.
F. Mohsen, H. Ali, N. El Hajj, and Z. Shah, “Artificial intelligence-based methods for fusion of electronic health records and imaging data,” Sci Rep, vol. 12, no. 1, p. 17981, Oct. 2022, doi: 10.1038/s41598-022-22514-4.
S.-C. Huang, A. Pareek, S. Seyyedi, I. Banerjee, and M. P. Lungren, “Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines,” NPJ Digit Med, vol. 3, no. 1, p. 136, 2020.
R. J. Al Gharrawi and A. A. Al-Joda, “A Survey of Medical Image Analysis Based on Machine Learning Techniques,” Journal of Al-Qadisiyah for Computer Science and Mathematics, vol. 15, no. 1, Feb. 2023, doi: 10.29304/jqcm.2023.15.1.1139.
R. N. Sadoon and A. M. Chaid, “Deep Learning Technique for The Classification of Lung Diseases from X-ray Images,” Journal of Al-Qadisiyah for Computer Science and Mathematics, vol. 16, no. 2, Jun. 2024, doi: 10.29304/jqcsm.2024.16.21544.
A. B. Menegotto, C. D. L. Becker, and S. C. Cazella, “Computer-aided diagnosis of hepatocellular carcinoma fusing imaging and structured health data,” Health Inf Sci Syst, vol. 9, no. 1, p. 20, 2021, doi: 10.1007/s13755-021-00151-x.
G. Muhammad, F. Alshehri, F. Karray, A. El Saddik, M. Alsulaiman, and T. H. Falk, “A comprehensive survey on multimodal medical signals fusion for smart healthcare systems,” Information Fusion, vol. 76, pp. 355–375, 2021, doi: https://doi.org/10.1016/j.inffus.2021.06.007.
E. Warner et al., “Multimodal Machine Learning in Image-Based and Clinical Biomedicine: Survey and Prospects,” Int J Comput Vis, 2024, doi: 10.1007/s11263-024-02032-8.
A. Holzinger, B. Malle, A. Saranti, and B. Pfeifer, “Towards multi-modal capability with Graph Neural Networks enabling information fusion for explainable AI,” Information Fusion, vol. 71, pp. 28–37, 2021, doi: https://doi.org/10.1016/j.inffus.2021.01.008.
Z. Salahuddin, H. C. Woodruff, A. Chatterjee, and P. Lambin, “Transparency of deep neural networks for medical image analysis: A review of interpretability methods,” Comput Biol Med, vol. 140, p. 105111, 2022.
E. H. Abdulsaed, M. Alabbas, and R. S. Khudeyer, “Optimizing the Architecture of Convolutional Neural Networks Using Modified Salp Swarm Algorithm,” Journal of Al-Qadisiyah for Computer Science and Mathematics, vol. 16, no. 1, Mar. 2024, doi: 10.29304/jqcsm.2024.16.11450.
K. Doi, “Diagnostic imaging over the last 50 years: research and development in medical imaging science and technology,” Phys Med Biol, vol. 51, no. 13, p. R5, 2006, doi: 10.1088/0031-9155/51/13/R02.
P. Shah et al., “Artificial intelligence and machine learning in clinical development: a translational perspective,” NPJ Digit Med, vol. 2, no. 1, p. 69, Jul. 2019, doi: 10.1038/s41746-019-0148-3.
O. Amft, “How Wearable Computing Is Shaping Digital Health,” Jul. 2018. doi: 10.1109/mprv.2018.011591067.
W. C. Sleeman IV, R. Kapoor, and P. Ghosh, “Multimodal classification: Current landscape, taxonomy and future directions,” ACM Comput Surv, vol. 55, no. 7, pp. 1–31, 2022.
S. Hameed Al-Amiry, “Survey on Diagnosing Retina Diseases in Optical Coherence Tomography Images Based on Deep Learning,” Journal of Al-Qadisiyah for Computer Science and Mathematics, vol. 16, no. 2, Jun. 2024, doi: 10.29304/jqcsm.2024.16.21568.
H.-I. Suk, S.-W. Lee, D. Shen, and A. D. N. Initiative, “Hierarchical feature representation and multimodal fusion with deep learning for AD/MCI diagnosis,” Neuroimage, vol. 101, pp. 569–582, 2014.
S. Liu et al., “Multimodal neuroimaging feature learning for multiclass diagnosis of Alzheimer’s disease,” IEEE Trans Biomed Eng, vol. 62, no. 4, pp. 1132–1140, 2014.
J. Shi, X. Zheng, Y. Li, Q. Zhang, and S. Ying, “Multimodal neuroimaging feature learning with multimodal stacked deep polynomial networks for diagnosis of Alzheimer’s disease,” IEEE J Biomed Health Inform, vol. 22, no. 1, pp. 173–183, 2017.
F. Zhang, Z. Li, B. Zhang, H. Du, B. Wang, and X. Zhang, “Multi-modal deep learning model for auxiliary diagnosis of Alzheimer’s disease,” Neurocomputing, vol. 361, pp. 185–195, 2019.
J. Song, J. Zheng, P. Li, X. Lu, G. Zhu, and P. Shen, “An effective multimodal image fusion method using MRI and PET for Alzheimer’s disease diagnosis,” Front Digit Health, vol. 3, p. 637386, 2021.
Z. Kong, M. Zhang, W. Zhu, Y. Yi, T. Wang, and B. Zhang, “Multi-modal data Alzheimer’s disease detection based on 3D convolution,” Biomed Signal Process Control, vol. 75, p. 103565, 2022.
M. Abdelaziz, T. Wang, and A. Elazab, “Fusing multimodal and anatomical volumes of interest features using convolutional auto-encoder and convolutional neural networks for Alzheimer's disease diagnosis,” Front Aging Neurosci, vol. 14, p. 812870, 2022.
Y. A. Mohammed and E. G. Saleh, “Investigating the Applicability of Logistic Regression and Artificial Neural Networks in Predicting Breast Cancer,” Journal of Al-Qadisiyah for Computer Science and Mathematics, vol. 12, no. 2, Jul. 2020, doi: 10.29304/jqcm.2020.12.2.697.
C. Cui et al., “Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review,” Progress in Biomedical Engineering, vol. 5, no. 2, p. 022001, 2023, doi: 10.1088/2516-1091/acc2fe.
R. M. A. N. D. F. W. A. N. D. S. W. A. N. D. X. J. A. N. D. J. T. Dudley, “Deep learning for healthcare: review, opportunities and challenges,” Jul. 2017. doi: 10.1093/bib/bbx044.
M. Salvi et al., “Automated assessment of glomerulosclerosis and tubular atrophy using deep learning,” Computerized Medical Imaging and Graphics, vol. 90, p. 101930, 2021, doi: https://doi.org/10.1016/j.compmedimag.2021.101930.
A. Abernethy et al., “The Promise of Digital Health: Then, Now, and the Future,” NAM Perspectives, vol. 6, no. 22, Jun. 2022, doi: 10.31478/202206e.
S. R. Stahlschmidt, B. Ulfenborg, and J. Synnergren, “Multimodal deep learning for biomedical data fusion: a review,” Brief Bioinform, vol. 23, no. 2, Mar. 2022, doi: 10.1093/bib/bbab569.
Zhang, T., & Shi, M. (2020). Multi-modal neuroimaging feature fusion for diagnosis of Alzheimer’s disease. Journal of neuroscience methods, 341, 108795.
Fang, X., Liu, Z., & Xu, M. (2020). Ensemble of deep convolutional neural networks based multimodality images for Alzheimer's disease diagnosis. IET Image Processing, 14(2), 318-326.
Chen, H., Guo, H., Xing, L., Chen, D., Yuan, T., Zhang, Y., & Zhang, X. (2023). Multimodal predictive classification of Alzheimer's disease based on attention‐combined fusion network: Integrated neuroimaging modalities and medical examination data. IET Image Processing, 17(11), 3153-3164.
Sun, Z., Wang, Z., & Xu, Y. (2023). Privacy protection in cross-platform recommender systems: techniques and challenges. Wireless Networks, 1-10.
Rahman, M. A. (2023, December). A Survey on Security and Privacy of Multimodal LLMs-Connected Healthcare Perspective. In 2023 IEEE Globecom Workshops (GC Wkshps) (pp. 1807-1812). IEEE.
Mohsen, F., Ali, H., El Hajj, N., & Shah, Z. (2022). Artificial intelligence-based methods for fusion of electronic health records and imaging data. Scientific Reports, 12(1), 17981.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Riyadh Hussein AL-Mosawi, Ahmed Al-Shammari
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
