The Magic of Speech/Text to Image Generation

The Magic of Speech/Text to Image Generation

1P.V.S Manisha, 2A.Laxmana Rao, 3K.Tejesh, 4Ch. Kumar, 5B. Venkata Karthik

1Assistant Professor, [2-5 ] B.Tech Students, LIET

[1,2,3,4,5] Computer Science and Information Technology, Lendi Institute of Engineering and Technology, Vizianagaram

---------------------------------------------------------------------***----------------------------------------------------------------

Abstract -Speech-to-image translation without text is an interesting and useful topic due to the potential applications in human computer interaction, art creation, computer-aided design. etc. Not to mention that many languages have no writing form. However, as far as we know, it has not been well-studied how to translate the speech signals into images directly and how well they can be translated. In this paper, we attempt to translate the speech & text signals into the image signals without the transcription stage by leveraging the advance of teacher-student learning and generative adversarial models. Specifically, a speech encoder is designed to represent the input speech signals as an embedding feature, and it is trained using teacher-student learning to obtain better generalization ability on new classes. Subsequently, a stacked adversarial generative network is used to synthesized high-quality images conditioned on the embedding feature encoded by the speech encoder. Experimental results on both synthesized and real data show that our proposed method is efficient to translate the raw speech signals into images without the middle text representation. We also have an additional feature where text is present if there is a problem in the system where voice detection is not present.

Keywords: ReactJS, OpenAI, API Keys, WebkitSpeech    Recognition ,  HTML5, CSS3, Bootstrap-5