Image Captioning Using GPT-2 and CLIP

Image captioning is a fundamental task in vision- language understanding, where the model predicts a tex- tual informative caption to a given input image. In this paper, we present a simple approach to address this task. We use CLIP encoding as a prefix to the caption, by em- ploying a simple mapping network, and then fine-tunes a language model to generate the image captions. The re- cently proposed CLIP model contains rich semantic fea- tures which were trained with textual context, making it best for vision-language perception. Our key idea is that together with a pre-trained language model (GPT2), we ob- tain a wide understanding of both visual and textual data. Hence, our approach only requires rather quick training to produce a competent captioning model. Without addi- tional annotations or pre-training, it efficiently generates