We propose GauFace, a novel Gaussian Splatting representation, tailored for efficient animation and rendering of physically-based facial assets. Leveraging strong geometric priors and constrained optimization, GauFace ensures a neat and structured Gaussian representation, delivering high fidelity and real-time facial interaction of 30fps@1440p on a Snapdragon 8 Gen 2 mobile platform.
We introduce TransGS, a diffusion transformer that instantly translates physically-based facial assets into the corresponding GauFace representations.
Once trained, TransGS can instantly translate facial assets with lighting conditions to GauFace representation. With the rich conditioning modalities, it also enables editing and animation capabilities reminiscent of traditional CG pipelines.
Left: we collect 143 facial assets under 134 lighting conditions, with a total of 1,023 combinations. Middle: for each combination, we render 1,071 frames under 153 different expressions with random camera positions. Right: our GauFace asset defines the center of Gaussians on the UV map consistent across different identities and introduces dynamic shadow vectors to disentangle the deformation-dependent and deformation-agnostic shading effects.
We condition TransGS on the image textures I, geometry code G and HDRI map L, to generate the GauFace asset A in a patch-based manner. Left: during training, a random global offset q is sampled, and the corresponding Image patch I_q and GauFace patch A_q are fed to the diffusion transformer. Right: at inference, the full GauFace asset can be synthesized in a single pass.
Conditioning on the PBR facial asset, TransGS generates the GauFace counterpart in 5 seconds. Here's some generated results:
The PBR facial assets are obtained from diverse sources. Row 1-3: Generated from ChatAvatar, a text/image-to-face generator. Row 4: Downloaded from web. Row 5: Scanned from a Light Stage. All figures are rendered under our cross-platform Unity3D GauFace render engine.
Thanks to the efficient representation of GauFace, we can render the facial assets in real-time on mobile devices using our Unity3D GauFace renderer.
Thanks to the rich conditions of TransGS, editing of textures can be seamlessly transferred to the GauFace representation in an interactive speed.
Adding a brand-new SIGGRAPH logo to the diffuse texture. Note that the GauFace counterpart maintains sharp logo boundary and harmonic lightning.
Add wrinkles via the normal texture. All details are faithfully transferred to GauFace with full 4K resolution.
TBD