MI-NeRF: Learning a Single Face NeRF from
Multiple Identities

Aggelina Chatziagapi1
Grigorios G. Chrysos2
Dimitris Samaras1

1Stony Brook University
2University of Wisconsin-Madison

arXiv 2024




Our proposed multi-identity NeRF (MI-NeRF) learns a single network to model complex non-rigid human face motion of multiple identities from just monocular videos. In contrast, standard approaches train multiple per-identity NeRFs.


Abstract

In this work, we introduce a method that learns a single dynamic neural radiance field (NeRF) from monocular talking face videos of multiple identities. NeRFs have shown remarkable results in modeling the 4D dynamics and appearance of human faces. However, they require per-identity optimization. Although recent approaches have proposed techniques to reduce the training and rendering time, increasing the number of identities can be expensive. We introduce MI-NeRF (multi-identity NeRF), a single unified network that models complex non-rigid facial motion for multiple identities, using only monocular videos of arbitrary length. The core premise in our method is to learn the non-linear interactions between identity and non-identity specific information with a multiplicative module. By training on multiple videos simultaneously, MI-NeRF not only reduces the total training time compared to standard single-identity NeRFs, but also demonstrates robustness in synthesizing novel expressions for any input identity. We present results for both facial expression transfer and talking face video synthesis. Our method can be further personalized for a target identity given only a short video.


Method



Overview of MI-NeRF. Given monocular talking face videos from multiple identities, MI-NeRF learns a single network to model their 4D geometry and appearance. A multiplicative module with shared weights across all identities learns non-linear interactions between identity codes and facial expressions. MI-NeRF can synthesize high-quality videos of any input identity.


Demo



BibTeX

If you find our work useful, please consider citing our paper: 
                
                    @misc{chatziagapi2024minerf,
                        title={MI-NeRF: Learning a Single Face NeRF from Multiple Identities}, 
                        author={Aggelina Chatziagapi and Grigorios G. Chrysos and Dimitris Samaras},
                        year={2024},
                        eprint={2403.19920},
                        archivePrefix={arXiv},
                        primaryClass={cs.CV}
                    }