Image Caption Generator with Suitable Personal Assistance Using CNN and LSTM

Image Caption Generator with Suitable Personal Assistance Using CNN and LSTM

#Dr.Umapathi G R, Pavan R Shetty*, Ramakrishna Shivaram Hegde* , Tanmay Pandey*, R Kalaivani Indira*

#Faculty, ISE-Dept, AIT, Bangalore, Karnataka, 560107

*Students, ISE-Dept, AIT, Bangalore, Karnataka, 560107

Abstract—Visually impaired or partially sighted people face a lot of problems reading or identifying any local scenarios. To vanquish this situation, we will be developing an audio-based image captioner that will identify the objects in an image and form a meaningful sentence that gives the output in the aural form. Image processing is a widely used method for developing many new applications. Image processing library is also open source, so developers can use it easily. We used NLP (Natural Language Processing) to understand the description of an image and convert the text to speech. A combination of R-LSTM and CNN is used, which is nothing but a reference based long-short term memory which matches different text data and takes it as reference and gives the output. Some of the other applications of image captioning are social media platforms like Instagram, etc., virtual assistants, and video editing software. As technology is a key element to progress passing the information in a right manner is the need of the hour. The world is moving towards digitization so are the means of technology, in this context our work acts as a mediator and helps in actively carrying messages in the form of script and oration signals. Automatically decoding a simpler depiction was technically at ease. So, need was to decrypt depiction with hidden memo.

Keywords- Caption Generator, LSTM, CNN, NLP, Image Caption, Audio, Feature Extraction, Deep Learning, Flickr Datasets.