Improving the Reliability and Accuracy of Image Captioning Systems Using Ensemble of FC, Softmax, and LSTM Deep Decoders

Abstract

In this work, a deep system for automatic image description is presented, which aims to produce fluent, meaningful, and structurally coherent sentences for input images. The proposed architecture is based on an encoder-decoder framework, in which high-level image features are first extracted by an Inception-v3 deep convolutional network and then fed as a compressed image representation to an LSTM-based language decoder to produce a word-by-word sentence. On this basic structure, a voting-based ensemble learning framework is designed, in which three deep paths, including a fully connected (FC) network, a Softmax linear model, and a sequence-oriented LSTM decoder, are trained independently, and the word probability vectors at the output level are combined with a maximum voting mechanism. The evaluation is performed on the standard Flickr8k database and using BLEU-1 to BLEU-4, METEOR, and ROUGE-L metrics. The results show that the best single LSTM model achieves values ​​of 0.64, 0.39, 0.23, and 0.16 for BLEU-1 to BLEU-4, and 0.22 and 0.50 for METEOR and ROUGE-L, respectively, while the Ensemble model improves the values ​​to 0.74, 0.50, 0.35, and 0.22 for BLEU-1 to BLEU-4, 0.475 for METEOR, and 0.55 for ROUGE-L; such that the relative improvements in BLEU-3 and BLEU-4 are 54% and 41%, respectively. The paired t-test also shows that the difference in Ensemble performance with single models is significant at the 95% confidence level, and compared to the existing methods on Flickr8k, competitive results are obtained and, in some measures, superior.