A Critical Analysis of Deep Learning Methods for Video QoE Prediction
DOI:
https://doi.org/10.29304/jqcsm.2025.17.22176Keywords:
QoE, Deep Learning, CNN, LSTM, Classification, RegressionEachAbstract
Multimedia video applications significantly impact video quality prediction, widely regarded as one of the most challenging problems. The Quality of Experience (QoE) prediction of the video mimics the satisfaction of the content of the video as humans perceive it. Machine learning and deep learning models have applied numerous methods to obtain QoE predictions. Some of these methods are full reference or reduced reference (half reference); others are no reference. In this paper, we attempt to explore, evaluate, and analyze the different scenarios and models related to QoE predictions for videos using deep learning. We have conducted a comprehensive examination to address the limitations of the existing models. Moreover, we suggest a new framework to overcome the limitations of the existing models.
Downloads
References
Digital 2024, "Global Overview Report," STATISTA, p. 109, (2024). [Online]. Available: https://www.statista.com/statistics/1254810/top-video-content-type-by-global-reach
M. Haqi Al-Tai, B. M. Nema, and A. Al-Sherbaz, "Deep Learning for Fake News Detection: Literature Review," AL-MUSTANSIRIYAH JOURNAL OF SCIENCE, vol. 34, no. 2, pp. 70–81, (2023), doi: 10.23851/mjs.v34i2.1292.
H. S. Ibrahim, N. M. Shati, and A. A. Alsewari, "A Transfer Learning Approach for Arabic Image Captions," AL-MUSTANSIRIYAH JOURNAL OF SCIENCE, vol. 35, no. 3, pp. 81–90, (2024) , doi: 10.23851/mjs.v35i3.1485.
D. Chmieliauskas and Š. Paulikas, "Video Stream Recognition Using Bitstream Shape for Mobile Network QoE," Sensors, vol. 23, p. 2548, (2023), doi: 10.3390/s23052548.
B. Mahaboob and S. A. Kalaiselvan, "Experimental Investigation Based on Services of Video Streaming using Deep Neural Network for Continuous QoE Prediction," Journal of Theoretical and Applied Information Technology, vol. 101, no. 5, pp. 1954–1961, Mar. (2023). [Online]. Available: www.jatit.org.
F. Gu and Z. Zhang, "No-Reference Quality Assessment of Stereoscopic Video Based on Temporal Adaptive Model for Improved Visual Communication," Sensors, vol. 22, no. 21, p. 8084, (2022).
Y. Wang, S. Inguva, and B. Adsumilli, "YouTube UGC Dataset for Video Compression Research," in PROC. IEEE INT. WORKSHOP ON MULTIMEDIA SIGNAL PROCESSING (MMSP), Kuala Lumpur, Malaysia, (2019) , pp. 1–5, doi: 10.1109/MMSP.2019.8901772.
N. M. Khassaf and S. H. Shaker, "Image Retrieval based Convolutional Neural Network," AL-MUSTANSIRIYAH JOURNAL OF SCIENCE, vol. 31, no. 4, pp. 43–54, (2020), doi: 10.23851/mjs.v31i4.897.
J. Yan, L. Wu, Y. Fang, X. Liu, X. Xia, and W. Liu, "Video Quality Assessment for Online Processing: From Spatial to Temporal Sampling," IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 12_Part_2, pp. 13441–13451, Dec. (2024), doi: 10.1109/TCSVT.2024.3450085.
M. Ghosh and C. Singhal, "MO-QoE: Video QoE using multi-feature fusion based Optimized Learning Models," Signal Processing: Image Communication, vol. 107, (2022) , p. 116766. ISSN: 0923-5965
S. N. Shakya and P. Kancharla, "Deep priors for video quality prediction," arXiv preprint*, arXiv:2410.22566, 2024. doi: 10.48550/arXiv.2410.22566.
A. Telili, A. Ksentini, Y. Hadjadj-Aoul, V. Pégon, and A. C. Begen, "Bitrate ladder prediction methods for adaptive video streaming: A review and benchmark," IEEE Transactions on Multimedia, vol. 24, no. 8, pp. 2225–2236, (2022) , doi: 10.1109/TMM.2022.3158572.
Q. Zheng, Y. Wang, J. Zhang, K. Zhang, and W. Lin, "Video Quality Assessment: A Comprehensive Survey," arXiv preprint arXiv:2412.04508, Dec. (2024).
W. Kim, J. Kim, S. Ahn, J. Kim, and S. Lee, "Deep video quality assessor: From spatio-temporal visual sensitivity to a convolutional neural aggregation network," in European Conference on Computer Vision (ECCV), Springer, Cham, (2018), pp. 219-234. doi: 10.1007/978-3-030-01237-3_17.
K. Seshadrinathan, R. Soundararajan, A. C. Bovik, and L. K. Cormack, "Study of subjective and objective quality assessment of video," IEEE Trans. Image Process., vol. 19, no. 6, pp. 1427–1441, (2010).
Laboratory of Computational Perception & Image Quality, Oklahoma State University, "CSIQ video database," (2013).
H. Zhang, H. Hu, G. Gao, Y. Wen, and K. Guan, "DeepQoE: A multimodal learning framework for video quality of experience (QoE) prediction," IEEE Trans. Multimedia, vol. 22, no. 12, pp. 3210–3223,( 2020) , doi: 10.1109/TMM.2020.2973828.
H. Zhang, H. Hu, G. Gao, Y. Wen, and K. Guan, "DeepQoE: A Unified Framework for Learning to Predict Video QoE," in 2018 IEEE International Conference on Multimedia and Expo (ICME), San Diego, CA, USA, (2018), pp. 1-6.
Y. Zhang, M. Yuan, and Z. Chen, “WHU-MVQoE2016: A quality of experience dataset for mobile video research,” WHU Tech. Rep., Dec. (2016).
C. G. Bampis, Z. Li, A. K. Moorthy, I. Katsavounidis, A. Aaron and A. C. Bovik, "LIVE Netflix Video Quality of Experience Database," Online: http://live.ece.utexas.edu/research/LIVE_NFLXStudy/index.html, (2016).
H. Wanget al., “VideoSet: A large-scale compressed video quality dataset based on jnd measurement,”J. Visual Commun. Image Representation, vol. 46, pp. 292–302, (2017).
C. Cárdenas-Angelat, J. B. Polglase, C. J. Vaca-Rubio, and M. C. Aguayo-Torres, "Application of Deep Learning Techniques to Video QoE Prediction in Smartphones," in 2019 European Conference on Networks and Communications (EuCNC), Valencia, Spain, (2019), pp. 252-256. doi: 10.1109/EuCNC.2019.8801974.
T. N. Duc, C. M. Tran, P. X. Tan, and E. Kamioka, "Convolution Neural Networks for Continuous QoE Prediction in Video Streaming Services," IEEE Access, vol. 8, pp. 116268-116278, (2020). doi: 10.1109/ACCESS.2020.3004125.
S. Bai, J. Zico Kolter, and V. Koltun, "An empirical evaluation of generic convolutional and recurrent networks for sequence modeling," arXiv preprint, arXiv:1803.01271, (2018). [Online]. Available: http://arxiv.org/abs/1803.01271.
N. Eswara, K. Manasa, A. Kommineni, S. Chakraborty, H. P. Sethuram, K. Kuchi, A. Kumar, and S. S. Channappayya, “A
Continuous QoE Evaluation Framework for Video Streaming Over HTTP,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 28, no. 11, pp. 3236–3250, nov (2018). [Online]. Available: https://ieeexplore.ieee.org/document/8013810/
D. Ghadiyaram, J. Pan, and A. C. Bovik, “A Subjective and Objective Study of Stalling Events in Mobile Streaming Videos,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 29, no. 1, pp. 183–197, (2019). [Online]. Available: http://ieeexplore.ieee.org/document/8093636/
N. Eswara et al., "Streaming Video QoE Modeling and Prediction: A Long Short-Term Memory Approach," IEEE Transactions on Circuits and Systems for Video Technology, vol. 30, no. 3, pp. 661-673, Mar. (2020). doi: 10.1109/TCSVT.2019.2895223
C. Chen, L. K. Choi, G. de Veciana, C. Caramanis, R. W. Heath, and A. C. Bovik, "Modeling the time-varying subjective quality of HTTP video streams with rate adaptations," IEEE Transactions on Image Processing, vol. 23, no. 5, pp. 2206–2221, May (2014).
M. Ghosh and C. Singhal, "M-3R: A Memory Based Approach for Streaming QoE Prediction under 3R settings," in 2021 IEEE International Conference on Advanced Networks and Telecommunications Systems (ANTS), Hyderabad, India, (2021), pp. 432-437. doi: 10.1109/ANTS52808.2021.9936944.
LIVE -Labaratory for Image and Video Quality Engineering, an Image Quality Assessment Database. Available: http://live.ece.utexas.edu/research/LIVE NFLX II/live nflx plus.html.
Z. Tu, X. Yu, Y. Wang, N. Birkbeck, B. Adsumilli, and A. C. Bovik, "RAPIQUE: Rapid and accurate video quality prediction of user generated content," IEEE Open Journal of Signal Processing, vol. 2, pp. 425–440, (2021).
V. Hosu, F. Hahn, M. Jenadeleh, H. Lin, H. Men, T. Szir´anyi, S. Li,and, and D. Saupe, “The Konstanz natural video database (KoNViD-1k),” in Proc. 9th Int. Conf. Qual. Multimedia Exper. (QoMEX), (2017), pp. 1–6.
Z. Sinno and A. C. Bovik, “Large-scale study of perceptual video quality,” IEEE Trans. Image Process, vol. 28, no. 2, pp. 612–627, (2018).
F. Gu, & Z. Zhang, “No-Reference Quality Assessment of Stereoscopic Video Based on Temporal Adaptive Model for Improved Visual Communication,” Sensors, (2022), 22(21), 8084. https://doi.org/10.3390/s22218084.
M. Urvoy, M. Barkowsky, R. Cousseau, Y. Koudota, V. Ricorde, P.L. Callet, J. Gutierrez, and N. Garcia, “NAMA3DS1-COSPAD1: Subjective video quality assessment database on coding conditions introducing freely available high quality 3D stereoscopic sequences.” In Proceedings of the 2012 Fourth International Workshop on Quality of Multimedia Experience, Melbourne, Australia,5–7 July (2012).
J. Wang, S. Wang, Z. Wang, “Quality prediction of asymmetrically compressed stereoscopic videos.” In Proceedings of the 2015 IEEE International Conference on Image Processing (ICIP), Quebec City, QC, Canada, 27–30 September )2015(.
F. Qi, T. Jiang, X. Fan, S. Ma, and D. Zhao, “Stereoscopic video quality assessment based on stereo just-noticeable difference model,” In Proceedings of the 2013 IEEE International Conference on Image Processing, Melbourne, Australia, 15–18 September )2013(.
M. Ghosh, R. Wayal, and C. Singhal, "DeSVQ: Deep Learning Based Streaming Video QoE Estimation," in Proceedings of the 23rd International Conference on Distributed Computing and Networking (ICDCN 2022), January 4–7,)( 2022(, Delhi, India, ACM, New York, NY, USA, 7 pages. doi: 10.1145/3491003.3491023.
M. B. Islam, "Three-Stream 3D deep CNN for no-Reference stereoscopic video quality assessment," Intelligent Systems with Applications, Jan. )2022(. doi: 10.1016/j.iswa.2021.200059.
D. R. Bull and F. Zhang, Chapter 2 "The human visual system," in Intelligent Image and Video Compression (Second Edition), D. R. Bull and F. Zhang, Eds. Academic Press, )2021(, pp. 17-58. ISBN: 9780128203538. doi: 10.1016/B978-0-12-820353-8.00011-6.
B. Appina, S. V. R. Dendi, K. Manasa, S. S. Channappayya, and A. C. Bovik, "Study of subjective quality and objective blind quality prediction of stereoscopic videos," IEEE Transactions on Image Processing, vol. 28, no. 10, pp. 5027–5040, Oct.( 2019).
Z. A. Khan, A. Beghdadi, M. Kaaniche, F. Alaya-Cheikh, and O. Gharbi, "A neural network based framework for effective laparoscopic video quality assessment," Computerized Medical Imaging and Graphics, vol. 101, p. 102121, Oct. (2022). doi: 10.1016/j.compmedimag.
Z. A. Khan, A. Beghdadi, F. A. Cheikh, M. Kaaniche, E. Pelanis, R. Palomar, A. A. Fretland, B. Edwin, O. J. Elle, Towards a video quality assessment based framework for enhancement of laparoscopic videos, in: SPIE 33Medical Imaging: Image Perception, Observer Performance, and Technology Assessment, Vol. 11316, (2020), p. 113160P.
R. Elwerghemmi, M. Heni, R. Ksantini, and R. Bouallegue, "Online QoE Assessment Model Based on Incremental Stacked Multiclass Classifier," International Journal of Computing and Digital Systems, (2023).
R. Elwerghemmi, M. Heni, R. Ksantini, and R. Bouallegue, "An Efficient Stacked Deep Incremental Model for Online Streaming Video QoE Prediction,"International Journal of Computing and Digital Systems, vol. 13, no. 1, pp. 1-12, (2023). doi: 10.12785/ijcds.1301119.
R. Elwerghemmi, M. Heni, R. Ksantini, and R. Bouallegue, "Online QoE Prediction Model Based on Stacked Multiclass Incremental Support Vector Machine," in 2019 8th International Conference on Modeling Simulation and Applied Optimization (ICMSAO), Manama, Bahrain, (2019), pp. 1-5. doi: 10.1109/ICMSAO.2019.8880302.
Poqemon-QoE-Dataset," Github. [Online]. Available: https://github.com/Lamyne/Poqemon-QoE-Dataset
C. Liu, X. Chen, X. Wang, X. Xie, and Z. Guo, "QoE Assessment Model Based on Continuous Deep Learning for Video in Wireless Networks," IEEE Transactions on Mobile Computing, vol. 22, no. 6, pp. 3619-3633, (2023).
A. Botchkarev, "Performance metrics (error measures) in machine learning regression, forecasting and prognostics: Properties and typology," arXiv preprint arXiv:1809.03006, Sept. (2018).
V. Plevris, G. Solorzano, N.P. Bakas, and M.E.A Seghier, “Investigation of performance metrics in regression analysis and machine learning-based prediction models, ” In ECCOMAS Congress (2022).
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Salwa Aqeel Mahdi, Huda Abdulaali Abdulbaqi, Hazeem B. Taher
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
