Abstract
Generating photo-realistic talking face videos with natural head movements and accurate lip synchronization is critical for applications in virtual reality, entertainment, and assistive technologies. Existing methods often result in static head postures when solely utilizing audio for driving, or generate distorted speaker identities when incorporating supplementary reference video for talking face generation, thereby undermining realism in both scenarios. We propose a deep learning framework that disentangles audio-driven lip synchronization and facial expressions from video-guided head postures, ensuring robust identity preservation. Our approach presents three integrated modules: an Audio-Driven Motion Flow Encoder that maps audio mel-spectrograms with facial landmarks from source images to produce a magnitude vector. Secondly, a Video-Guided Pose Refinement Network is employed to warp the source image utilizing posture parameters derived from a reference video. This is followed by an Identity Preservation Codebook Learning module, which integrates the outputs of the preceding two modules into a Vector Quantized Variational Autoencoder (VQ-VAE) to generate realistic video frames. Evaluations on the VoxCeleb dataset demonstrate state-of-the-art performance in video quality (PSNR: 30.25, SSIM: 0.88), head motion diversity (SDHMF: 0.29), and identity preservation (FID: 25.10), with competitive lip synchronization. This study advances efficient video processing for low-power devices, offering significant implications for talking face video technology.
The Architecture
Overview of the proposed framework. The Flow Encoder translates the intermediate representations of audio and static facial inputs into a magnitude vector, while a warping network utilizes posture parameters from a reference video to manipulate the static face. Ultimately, a VQ-VAE improves the distorted images into realistic generated video frames, guided by the magnitude vector, trained to reconstruct the ground-truth frames. During Inference; audio, face image and a reference video is passed to the model to generate photorealistic talking face.
Our specific contributions are summarized as follows:
- We propose a novel deep learning framework that disentangles the learning of dynamic facial attributes (e.g., head posture and eye blinks) from the preservation of the speaker’s identity. This decoupling enhances the realism and controllability of generated talking faces by isolating transient motions from speaker-specific characteristics.
- We leverage a Vector-Quantized Variational Autoencoder (VQ-VAE) to encode gesture patterns from reference videos into a frozen codebook. This enables prosody-driven animation of static images, producing personalized facial expressions and head movements that retain an individual’s unique talking style.
- We demonstrate the capability to generate realistic talking video frames with lip and head movements synchronized to the speaker’s speech prosody. This addresses the limitations of existing approaches that merely transfer motion features from reference videos, which often fail to align lip movements with the intended speech, resulting in unrealistic talking faces.
The source Image
Results
Description
The static source image was trained to talk like the reference video on the middle.
To the left is the outcome of model without disentangling the identity and motion,
while to the right is the outcome of model with disentangling the identity and motion.
The static source image was trained to talk like the reference video at the top right corner.
The output at the left shows the static image speaking in Cantones language with dynamic
head
motions learned from the reference video, while the middle output sings with the same head
motions.
