Instant Facial Gaussians Translator for Relightable and Interactable Facial Rendering

Instant Facial Gaussians Translator for Relightable and Interactable Facial Rendering

Dafei Qin1,3, Hongyang Lin2,3, Qixuan Zhang 2,3, Kaichun Qiao2,3, Longwen Zhang2,3, Zijun Zhao2,3, Jun Saito4, Jingyi Yu2, Lan Xu2, Taku Komura1,
1University of Hong Kong, 2ShanghaiTech University 3Deemos Technology Co., Ltd. 4Adobe Research
arXiv(Upcoming) Video Code(Upcoming) Dataset(Upcoming)

TransGS convert PBR facial assets to high-quality 3DGS in 5 seconds, delivering unprecedented real-time rendering quality on mobile platforms.

Abstract

We propose GauFace, a novel Gaussian Splatting representation, tailored for efficient animation and rendering of physically-based facial assets. Leveraging strong geometric priors and constrained optimization, GauFace ensures a neat and structured Gaussian representation, delivering high fidelity and real-time facial interaction of 30fps@1440p on a Snapdragon 8 Gen 2 mobile platform.

We introduce TransGS, a diffusion transformer that instantly translates physically-based facial assets into the corresponding GauFace representations.

Once trained, TransGS can instantly translate facial assets with lighting conditions to GauFace representation. With the rich conditioning modalities, it also enables editing and animation capabilities reminiscent of traditional CG pipelines.

Video

Methods

GauFace Representation & Dataset

Left: we collect 143 facial assets under 134 lighting conditions, with a total of 1,023 combinations. Middle: for each combination, we render 1,071 frames under 153 different expressions with random camera positions. Right: our GauFace asset defines the center of Gaussians on the UV map consistent across different identities and introduces dynamic shadow vectors to disentangle the deformation-dependent and deformation-agnostic shading effects.

TransGS Structure

We condition TransGS on the image textures I, geometry code G and HDRI map L, to generate the GauFace asset A in a patch-based manner. Left: during training, a random global offset q is sampled, and the corresponding Image patch I_q and GauFace patch A_q are fed to the diffusion transformer. Right: at inference, the full GauFace asset can be synthesized in a single pass.

Instant Facial Gaussian Splatting Generation

Conditioning on the PBR facial asset, TransGS generates the GauFace counterpart in 5 seconds. Here's some generated results:

The PBR facial assets are obtained from diverse sources. Row 1-3: Generated from ChatAvatar, a text/image-to-face generator. Row 4: Downloaded from web. Row 5: Scanned from a Light Stage. All figures are rendered under our cross-platform Unity3D GauFace render engine.

Applications

Real-time Rendering on Mobile devices

Thanks to the efficient representation of GauFace, we can render the facial assets in real-time on mobile devices using our Unity3D GauFace renderer.

Interactive Editing

Thanks to the rich conditions of TransGS, editing of textures can be seamlessly transferred to the GauFace representation in an interactive speed.

Adding a brand-new SIGGRAPH logo to the diffuse texture. Note that the GauFace counterpart maintains sharp logo boundary and harmonic lightning.

Add wrinkles via the normal texture. All details are faithfully transferred to GauFace with full 4K resolution.

BibTeX


      TBD