Identification and Captioning using Deep Learning Techniques for Multimodal Images

Identification and Captioning using Deep Learning Techniques for Multimodal Images

Bhairavi D. Naik, Nikita R. Pardeshi, Kinjal S. Patil, Aarti S. Shinde

MKSSS's Cummins College of Engineering for Women

(An Autonomous Institute Affiliated to Savitribai Phule Pune University)

Abstract

In recent years, with the development of deep learning, the combination of computer vision and natural language processing has aroused great attention in the past few years. With the rapid development of artificial intelligence, image caption has gradually attracted the attention of many researchers in the field of artificial intelligence and has become an interesting and arduous task. Captions have become one of the most needed tools in the modern world. Captions, which automatically generate natural language descriptions according to the content observed in an image, are an important part of scene understanding, combining computer vision knowledge and natural language processing. The process of generating image descriptions is called image Captioning. You need to recognize important objects, their attributes, and relationships between objects in the image. Generates sentences that are both syntactically and semantically correct. This article presents a deep learning model that uses computer vision and machine translation to describe images and generate captions. This paper aims at recognizing different objects in an image and recognizing relationships between these objects to generate captions. The dataset used was Flickr8k and the programming language used was Python3 to demonstrate the proposed project.

The application of image captions is extensive and significant, for example, the     realization of human-computer interaction.

Keywords: Image, Caption, CNN, VGG16, RNN, LSTM.