Text Scene Synthesis: A Two-Phase Framework Using Diffusion Models and CLIP-Guided Video Generation

Abstract — The automatic generation of visual scenes from natural language descriptions represents a frontier challenge in artificial intelligence, bridging the gap between linguistic understanding and photorealistic visual synthesis. This paper presents a comprehensive two-phase framework for Text Scene Synthesis. In Phase 1, we leverage a state-of-the-art denoising diffusion probabilistic model (DDPM) conditioned on text embeddings to produce high-fidelity, semantically accurate images from arbitrary textual prompts. In Phase 2, we extend the framework to temporal visual synthesis by employing Contrastive Language-Image Pretraining (CLIP) alongside a video generation pipeline to produce coherent short-form video sequences directly from text. Experimental evaluation demonstrates that the proposed system significantly outperforms prior GAN-based, VAE-based, and standalone diffusion approaches in terms of image fidelity, semantic consistency, and temporal coherence. The system achieves a Fréchet Inception Distance (FID) of 12.4 on COCO-captions and a CLIPSIM score of 0.312 for video generation, representing meaningful improvements over prior methods. This work demonstrates the promise of unified text-to-scene pipelines for applications spanning entertainment, education, game development, and assistive technologies.