Deep Generative Framework for Multi Variant Style Fusion from a Single Image

In the present work, a deep generative model is presented which can generate several style variations from a single style reference while taking into account the content image. This technique can be considered as one of the most challenging issues in the area of neural style transfer. The proposed network architecture is based on the high-level perceptual layers of VGG19 as the generative model through GAN, which uses latent codes obtained from Gram matrices and adaptive layer normalization to control the styling process in all scales. This paper aims to develop a deep generative model to incorporate the content image and create multiple variations of a given style. This type of work can be seen as one of the most challenging problems in the neural style transfer. In the generative model, this research applies the high-level perceptual features extracted by VGG19 networks in the generator that applies the GAN framework to condition the style process at all scales, using the latent code generated by Gram matrix and Adaptive Layer Normalization. As demonstrated in the experiments, the approach outperforms the state-of-the-art approaches, such as AdaIN and Johnson et al., in quantitative metrics including SSIM, PSNR, FID and LPIPS (62.3 FID vs 78.2 FID for AdaIN, 86.4 FID for Johnson et al.). Moreover, generation time becomes 14ms per image due to the feed-forward process of the pipeline, which is about 900 times faster than the iterative Gatys algorithm. In the ablation study, all components are revealed to be unique in this model, such as Multi-variant Fusion, Perceptual Loss, and AdaLN Conditioning. The results show that single-image multi-style synthesis can be achieved within a single generative framework while maintaining a high fidelity to the content of the input image and fast inference time. The proposed system creates a viable basis for multi-style visual style in real-time deployment environments with limited resources.

[1]     L. A. Gatys, A. S. Ecker, and M. Bethge, "A neural algorithm of artistic style," arXiv preprint arXiv:1508.06576, 2015. [2]   J. Johnson, A. Alahi, and L. Fei-Fei, "Perceptual losses for real-time style transfer and super-resolution," in Proc. ECCV, 2016, pp. 694–711. [3]    D. Ulyanov, V. Lebedev, A. Vedaldi, and V. Lempitsky, "Texture networks: Feed-forward synthesis of textures and stylized images," in Proc. ICML, 2016, pp. 1349–1357. [4]      V. Dumoulin, J. Shlens, and M. Kudlur, "A learned representation for artistic style," in Proc. ICLR, 2017. [5]   X. Huang and S. Belongie, "Arbitrary style transfer in real- time with adaptive instance normalization," in Proc. ICCV, 2017, pp. 1501–1510. [6]   A. Mahendran and A. Vedaldi, "Understanding deep image representations by inverting them," in Proc. CVPR, 2015, pp. 5188–5196. [7]     D. Ulyanov, A. Vedaldi, and V. Lempitsky, "Instance normalization: The missing ingredient for fast stylization," arXiv preprint arXiv:1607.08022, 2016. [8]   Y. Li, C. Fang, J. Yang, Z. Wang, X. Lu, and M.-H. Yang, "Universal style transfer via feature transforms," in Proc. NeurIPS, 2017, pp. 386–396. [9]     I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative adversarial nets," in Proc. NeurIPS, 2014, pp. 2672–2680. [10]   P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, "Image-to- image translation with conditional adversarial networks," in Proc. CVPR, 2017, pp. 5967–5976.