We propose an end-to-end deep-learning approach for automatic rigging and retargeting of 3D models of human faces in the wild. Our approach, called Neural Face Rigging (NFR), holds three key properties:
(i) NFR's expression space maintains human-interpretable editing parameters for artistic controls;
(ii) NFR is readily applicable to arbitrary facial meshes with different connectivity and expressions;
(iii) NFR can encode and produce fine-grained details of complex expressions performed by arbitrary subjects.
To the best of our knowledge, NFR is the first approach to provide realistic and controllable deformations of in-the-wild facial meshes, without the manual creation of blendshapes or correspondence. We design a deformation autoencoder and train it through a multi-dataset training scheme, which benefits from the unique advantages of two data sources: a linear 3DMM with interpretable control parameters as in FACS, and 4D captures of real faces with fine-grained details. Through various experiments, we show NFR's ability to automatically produce realistic and accurate facial deformations across a wide range of existing datasets as well as noisy facial scans in-the-wild, while providing artist-controlled, editable parameters.
NFR can transfer facial animations to arbitrary meshes, including those with different connectivity and expressions.
Retargeting ICT FaceKit animation to TripleGangers, FacewareHouse and Raw Scan.
Retargeting Multiface speaking animation to TripleGangers, FacewareHouse and Raw Scan.
NFR provides a FACS like rigging space for direct controls.
Given the same expression codes, NFR maintains pausible expressions while linear rigging fails.
Our project is build on top of existing works that are topology-agnostic. Neural Jacobian Fields enables us to process the facial deformation in an expressive and accurate manner. DiffusionNet helps to aggregate information on the input meshes with varying topologies.
@inproceedings{qin2023NFR,
author = {Qin, Dafei and Saito, Jun and Aigerman, Noam and Groueix Thibault and Komura, Taku},
title = {Neural Face Rigging for Animating and Retargeting Facial Meshes in the Wild},
year = {2023},
booktitle = {SIGGRAPH 2023 Conference Papers},
}