Improved Semantic Inpainting Architecture Augmented with a Facial Landmark Detector
|
Mirza Sami School of Computing, University of Alabama at Birmingham, USA This email address is being protected from spambots. You need JavaScript enabled to view it. |
Israt Naiyer Department of Computer Science and Engineering, Brac University, Bangladesh This email address is being protected from spambots. You need JavaScript enabled to view it. |
Ehsanul Khan Department of Computer Science and Engineering, Brac University, Bangladesh This email address is being protected from spambots. You need JavaScript enabled to view it. |
Jia Uddin AI and Big Data Department, Endicott College, Woosong University, South Korea This email address is being protected from spambots. You need JavaScript enabled to view it. |
Abstract: This paper presents an augmented method for image completion, particularly for images of human faces by leveraging on deep learning based inpainting techniques. Face completion generally tend to be a daunting task because of the relatively low uniformity of a face attributed to structures like eyes, nose, etc. Here, understanding the top level context is paramount for proper semantic completion. The method presented improves upon existing inpainting techniques that reduce context difference by locating the closest encoding of the damaged image in the latent space of a pre-trained deep generator. However, these existing methods fail to consider key facial structures (eyes, nose, jawline, etc.,) and their respective location to each other. This paper mitigates this by introducing a face landmark detector and a corresponding landmark loss. This landmark loss is added to the construction loss between the damaged and generated image and the adversarial loss of the generative model. The model was trained with the celeb A dataset, tools like pyamg, pillow and the OpenCV library was used for image manipulation and facial landmark detection. There are three main weighted parameters that balance the effect of the three loss functions in this paper, namely context loss, landmark loss and prior loss. Experimental results demonstrate that the added landmark loss attributes to better understanding of top-level context and hence the model can generate more visually appealing in painted images than the existing model.The model obtained average Structural Similarity Index (SSIM) and Peak Signal-to-Noise Ratio (PNSR) scores of 0.851 and 33.448 for different orientations of the face and 0.896 and 31.473, respectively, for various types masks.
Keywords: Structural image inpainting, generative adversarial networks, facial landmark, synthetic image.
Received December 27, 2019; accepted February 2, 2021