StyleFaceUV: 3D Face Image Synthesis for View Consistency

Creating consistent face images across different viewpoints has always been a challenge for image generation models. Recent advancements in deep image generation, like StyleGAN2, produce high-quality 2D face images but struggle with multi-view consistency. StyleFaceUV offers a groundbreaking solution, leveraging a 3D face UV map generator for synthesizing view-consistent face images, bridging the gap between 2D image quality and 3D consistency. This approach not only ensures the face appearance remains consistent when viewed from different angles but also offers explicit control over face attributes and smooth transitions between different identities.

The Problem of Inconsistency in 2D Face Image Synthesis

Recent deep image generation models, such as StyleGAN2, have demonstrated the capability to synthesize highly realistic 2D face images.

However, these models often face challenges in maintaining consistency when generating face images from multiple viewpoints. This means that the same face, when rendered from different angles, may appear significantly different, which undermines the realism and applicability of these models in scenarios that require 3D understanding.

Multi-view consistency is paramount when synthesized faces are used in applications like virtual reality, augmented reality, and 3D modeling, where the ability to view a face from any angle without visual inconsistencies is crucial. The core issue lies in the fact that 2D image generation models typically lack an explicit understanding of the underlying 3D structure of a face, making it difficult to ensure geometric and textural consistency across views.

This inconsistency problem motivates the need for novel approaches that can incorporate 3D information into the face image synthesis process. StyleFaceUV offers an innovative solution to address this challenge by integrating 3D face UV maps, enabling the generation of face images that are not only realistic but also consistent across multiple viewpoints. By approaching the problem from a 3D perspective, StyleFaceUV minimizes the discrepancies and artifacts commonly found in purely 2D-based methods.

Introducing StyleFaceUV: A 3D Solution

StyleFaceUV presents a Novel approach to bridge the gap between high-quality 2D face image synthesis and 3D consistency. This model leverages a 3D face UV map generator, pre-trained StyleGAN2 model, and a parametric face model (3DMM) to achieve view-consistent face image generation.

By synthesizing 3D face meshes and encouraging their appearance to be consistent across different views, StyleFaceUV significantly enhances the realism and applicability of generated faces.

Key benefits of StyleFaceUV include:

• View-consistent generation: Face images maintain a coherent appearance from all viewing angles.
• 3D face mesh synthesis: Detailed 3D face structures derived from StyleGAN2.
• Compatibility: Seamless integration with StyleGAN2's latent space.
• Attribute control: Precise manipulation of facial features.
• Application breadth: Opens possibilities for various downstream tasks.

The Multi-View Loss

In some instances, the synthesized texture may fail on one side of the face, leading to inconsistencies and artifacts.

To address this, StyleFaceUV incorporates a multi-view loss function that encourages the faces to be consistent across both original and symmetric views. This is achieved by preparing symmetric-view StyleGAN2 images and their corresponding 3DMM coefficients for training.

Yaw angle shifting is utilized to turn the view angle of the original style code to its symmetric view, ensuring that the generated faces maintain a consistent appearance from all angles. Additionally, a weighted mask is used to combine face information in the overlapping region between two views, minimizing artifacts and improving the overall quality of the synthesized faces.

3D Coefficient Predictor

The first key component of StyleFaceUV is the 3D coefficient predictor.

This module’s primary task is to predict the corresponding 3D Morphable Model (3DMM) coefficients based on the input style code. These 3DMM coefficients are essential for defining the Shape and structure of the generated face, providing the geometric information needed for rendering.

• Geometry encoding: The 3DMM coefficients encode the three-dimensional shape of the face, enabling realistic and accurate representations.

3D Generator

The Second key component is the 3D Generator.

This module generates the diffuse map and displacement map based on the style code. These maps provide the appearance information for the face. StyleFaceUV integrates a modified StyleGAN2 generator as its 3D generator, leveraging its pre-trained capabilities to synthesize high-quality face textures. StyleFaceUV's integration with StyleGAN2 enables the generation of diverse and realistic face textures, enhancing the visual quality and realism of the synthesized faces.

Photo Loss

To encourage the synthesized face to have high-fidelity quality and identity, reconstruction loss containing photo loss is used.

It uses the L2 norm.

The Reconstruction Loss

To enhance the fidelity and identity of the synthesized face, StyleFaceUV employs a reconstruction loss function that contains both photo loss and perceptual loss.

The photo loss measures the L2 norm between images to minimize pixel-level differences, while the perceptual loss extracts feature maps to capture high-level details. A multi-view loss encourages faces to be consistent in both original and symmetric views.

View-Consistent Face Image Generation

StyleFaceUV’s architecture synthesizes 3D face meshes and encourages their appearance to remain consistent regardless of the viewing angle.

By rendering the generated mesh as images in different views, the model ensures that face images maintain a coherent and realistic appearance, addressing one of the primary limitations of traditional 2D face generation methods.

Latent Space Compatibility

One of the significant advantages of StyleFaceUV is its compatibility with the original StyleGAN2 latent space. This means that the 3D face models synthesized from latent codes match the face images generated directly from StyleGAN2, allowing for seamless integration and manipulation of facial features.

This feature enables users to leverage StyleGAN2’s powerful capabilities while benefiting from the added consistency and control offered by StyleFaceUV.

Explicit Face Attribute Control

StyleFaceUV utilizes a parameterized face model called 3DMM as its base Blend shape. This enables explicit tuning of face attributes, such as expression and lighting, by directly manipulating the 3DMM coefficients.

Unlike other methods that require indirect style code editing, StyleFaceUV provides a more intuitive and disentangled approach to facial feature manipulation.

Explicit face attribute control allows for precise adjustments to facial expressions, ensuring that the generated faces convey the desired emotions or characteristics. Additionally, the ability to control lighting enables the creation of realistic and visually appealing renderings.

Increased utility for downstream tasks

By generating the 3D face meshes we can provide more applications for downstream tasks