A Survey on Deep Generative 3D-aware Image Synthesis

ACM Computing Surveys, 2023
Weihao Xia · Jing-Hao Xue

Introduction

This project lists representative papers/codes/datasets about deep 3D-aware image synthesis. Besides 3D-aware Generative Models (GANs and Diffusion Models) discussed in this survey, this project additionally covers novel view synthesis studies, especially those based on implicit neural representations such as NeRF.

We aim to constantly update the latest relevant papers and help the community track this topic. Please feel free to join us and contribute to the project. Please do not hesitate to reach out if you have any questions or suggestions.

Survey paper

• A Survey on Deep Generative 3D-aware Image Synthesis
  Weihao Xia and Jing-Hao Xue. ACM Computing Surveys, 2023.

3D Control of 2D GANs

3D Control Latent Directions

For 3D control over diffusion models simiar to GAN, please refer to semantic manipulation in diffusion latent spaces.

• SeFa: Closed-Form Factorization of Latent Semantics in GANs.
  Yujun Shen, Bolei Zhou.
  CVPR 2021. [Paper] [Project] [Code]

• GANSpace: Discovering Interpretable GAN Controls.
  Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, Sylvain Paris.
  NeurIPS 2020. [Paper] [Code]

• Interpreting the Latent Space of GANs for Semantic Face Editing.
  Yujun Shen, Jinjin Gu, Xiaoou Tang, Bolei Zhou.
  CVPR 2020. [Paper] [Project] [Code]

• Unsupervised Discovery of Interpretable Directions in the GAN Latent Space.
  Andrey Voynov, Artem Babenko.
  ICML 2020. [Paper] [Code]

• On the "steerability" of generative adversarial networks.
  Ali Jahanian, Lucy Chai, Phillip Isola.
  ICLR 2020. [Paper] [Project] [Code]

3D Parameters as Controls

• 3D-FM GAN: Towards 3D-Controllable Face Manipulation.
  Yuchen Liu, Zhixin Shu, Yijun Li, Zhe Lin, Richard Zhang, and Sun-Yuan Kung.
  ECCV 2022. [Paper] [Project]

• GAN-Control: Explicitly Controllable GANs.
  Alon Shoshan, Nadav Bhonker, Igor Kviatkovsky, Gerard Medioni.
  ICCV 2021. [Paper] [Project] [Code]

• CONFIG: Controllable Neural Face Image Generation.
  Marek Kowalski, Stephan J. Garbin, Virginia Estellers, Tadas Baltrušaitis, Matthew Johnson, Jamie Shotton.
  ECCV 2020. [Paper] [Code]

• DiscoFaceGAN: Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning.
  Yu Deng, Jiaolong Yang, Dong Chen, Fang Wen, Xin Tong.
  CVPR 2020. [Paper] [Code]

• StyleRig: Rigging StyleGAN for 3D Control over Portrait Images.
  Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, Christian Theobalt.
  CVPR 2020 (oral). [Paper] [Project]

• PIE: Portrait Image Embedding for Semantic Control.
  Ayush Tewari, Mohamed Elgharib, Mallikarjun B R., Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, Christian Theobalt.
  TOG (SIGGRAPH Asia) 2020. [Paper] [Project]

3D Prior Knowledge as Constraints

• 3D-Aware Indoor Scene Synthesis with Depth Priors.
  Zifan Shi, Yujun Shen, Jiapeng Zhu, Dit-Yan Yeung, Qifeng Chen.
  ECCV 2022 (oral). [Paper] [Project] [Code]

• NGP: Towards a Neural Graphics Pipeline for Controllable Image Generation.
  Xuelin Chen, Daniel Cohen-Or, Baoquan Chen, Niloy J. Mitra.
  Eurographics 2021. [Paper] [Code]

• Lifting 2D StyleGAN for 3D-Aware Face Generation.
  Yichun Shi, Divyansh Aggarwal, Anil K. Jain.
  CVPR 2021. [Paper] [Code]

• RGBD-GAN: Unsupervised 3D Representation Learning From Natural Image Datasets via RGBD Image Synthesis.
  Atsuhiro Noguchi, Tatsuya Harada.
  ICLR 2020. [Paper] [Code]

• Visual Object Networks: Image Generation with Disentangled 3D Representation.
  Jun-Yan Zhu, Zhoutong Zhang, Chengkai Zhang, Jiajun Wu, Antonio Torralba, Joshua B. Tenenbaum, William T. Freeman.
  NeurIPS 2018. [Paper] [Project] [Code]

• 3D Shape Induction from 2D Views of Multiple Objects.
  Matheus Gadelha, Subhransu Maji, Rui Wang.
  3DV 2017. [Paper] [Project] [Code]

• Generative Image Modeling using Style and Structure Adversarial Networks.
  Xiaolong Wang, Abhinav Gupta.
  ECCV 2016. [Paper] [Project] [Code]

3D-aware GANs for a Single Image Category

Unconditional 3D Generative Models

• BallGAN: 3D-aware Image Synthesis with a Spherical Background.
  Minjung Shin, Yunji Seo, Jeongmin Bae, Young Sun Choi, Hyunsu Kim, Hyeran Byun, Youngjung Uh.
  ICCV 2023. [Paper] [Project] [Code]

• Mimic3D: Thriving 3D-Aware GANs via 3D-to-2D Imitation.
  Xingyu Chen, Yu Deng, Baoyuan Wang.
  ICCV 2023. [Paper] [Project]

• GRAM-HD: 3D-Consistent Image Generation at High Resolution with Generative Radiance Manifolds.
  Jianfeng Xiang, Jiaolong Yang, Yu Deng, Xin Tong.
  ICCV 2023. [Paper] [Project]

• Live 3D Portrait: Real-Time Radiance Fields for Single-Image Portrait View Synthesis.
  Alex Trevithick, Matthew Chan, Michael Stengel, Eric R. Chan, Chao Liu, Zhiding Yu, Sameh Khamis, Manmohan Chandraker, Ravi Ramamoorthi, Koki Nagano.
  TOG (SIGGRAPH) 2023. [Paper] [Project]

• VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Grids.
  Katja Schwarz, Axel Sauer, Michael Niemeyer, Yiyi Liao, Andreas Geiger.
  NeurIPS 2022. [Paper] [Code]

• GeoD: Improving 3D-aware Image Synthesis with A Geometry-aware Discriminator.
  Zifan Shi, Yinghao Xu, Yujun Shen, Deli Zhao, Qifeng Chen, Dit-Yan Yeung.
  NeurIPS 2022. [Paper] [Project]

• EpiGRAF: Rethinking training of 3D GANs.
  Ivan Skorokhodov, Sergey Tulyakov, Yiqun Wang, Peter Wonka.
  NeurIPS 2022. [Paper] [Project] [Code]

• VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Grids.
  Schwarz, Katja, Sauer, Axel, Niemeyer, Michael, Liao, Yiyi, and Geiger, Andreas.
  NeurIPS 2022. [Paper] [Project]

• Injecting 3D Perception of Controllable NeRF-GAN into StyleGAN for Editable Portrait Image Synthesis.
  Jeong-gi Kwak, Yuanming Li, Dongsik Yoon, Donghyeon Kim, David Han, Hanseok Ko.
  ECCV 2022. [Paper] [Project] [Code]

• Generative Multiplane Images: Making a 2D GAN 3D-Aware.
  Xiaoming Zhao, Fangchang Ma, David Güera, Zhile Ren, Alexander G. Schwing, Alex Colburn.
  ECCV 2022. [Paper] [Project] [Code]

• 3D-FM GAN: Towards 3D-Controllable Face Manipulation.
  Yuchen Liu, Zhixin Shu, Yijun Li, Zhe Lin, Richard Zhang, and Sun-Yuan Kung.
  ECCV 2022. [Paper] [Project]

• EG3D: Efficient Geometry-aware 3D Generative Adversarial Networks.
  Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, Gordon Wetzstein.
  CVPR 2022. [Paper] [Project] [Code]

• StylizedNeRF: Consistent 3D Scene Stylization as Stylized NeRF via 2D-3D Mutual Learning.
  Yi-Hua Huang, Yue He, Yu-Jie Yuan, Yu-Kun Lai, Lin Gao.
  CVPR 2022. [Paper]

• Multi-View Consistent Generative Adversarial Networks for 3D-aware Image Synthesis.
  Xuanmeng Zhang, Zhedong Zheng, Daiheng Gao, Bang Zhang, Pan Pan, Yi Yang.
  CVPR 2022. [Paper] [Code]

• Disentangled3D: Learning a 3D Generative Model with Disentangled Geometry and Appearance from Monocular Images.
  Ayush Tewari, Mallikarjun B R, Xingang Pan, Ohad Fried, Maneesh Agrawala, Christian Theobalt.
  CVPR 2022. [Paper] [Project]

• GIRAFFE HD: A High-Resolution 3D-aware Generative Model.
  Yang Xue, Yuheng Li, Krishna Kumar Singh, Yong Jae Lee.
  CVPR 2022. [Paper] [Code]

• StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation.
  Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, Ira Kemelmacher-Shlizerman.
  CVPR 2022. [Paper] [Project] [Code]

• FENeRF: Face Editing in Neural Radiance Fields.
  Jingxiang Sun, Xuan Wang, Yong Zhang, Xiaoyu Li, Qi Zhang, Yebin Liu, Jue Wang.
  CVPR 2022. [Paper] [Code]

• LOLNeRF: Learn from One Look.
  Daniel Rebain, Mark Matthews, Kwang Moo Yi, Dmitry Lagun, Andrea Tagliasacchi.
  CVPR 2022. [Paper] [Project]

• GRAM: Generative Radiance Manifolds for 3D-Aware Image Generation.
  Yu Deng, Jiaolong Yang, Jianfeng Xiang, Xin Tong.
  CVPR 2022. [Paper] [Project] [Code]

• VolumeGAN: 3D-aware Image Synthesis via Learning Structural and Textural Representations.
  Yinghao Xu, Sida Peng, Ceyuan Yang, Yujun Shen, Bolei Zhou.
  CVPR 2022. [Paper] [Project] [Code]

• Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks.
  Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Woo Ha, Jinwoo Shin.
  ICLR 2022. [Paper] [Project] [Code]

• StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis.
  Jiatao Gu, Lingjie Liu, Peng Wang, Christian Theobalt.
  ICLR 2022. [Paper] [Project]

• MOST-GAN: 3D Morphable StyleGAN for Disentangled Face Image Manipulation.
  Safa C. Medin, Bernhard Egger, Anoop Cherian, Ye Wang, Joshua B. Tenenbaum, Xiaoming Liu, Tim K. Marks.
  AAAI 2022. [Paper]

• A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis.
  Xingang Pan, Xudong Xu, Chen Change Loy, Christian Theobalt, Bo Dai.
  NeurIPS 2021. [Paper]

• pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis.
  Eric R. Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, Gordon Wetzstein.
  CVPR 2021. [Paper] [Project] [Code]

• GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields.
  Michael Niemeyer, Andreas Geiger.
  CVPR 2021 (Best Paper). [Paper] [Project] [Code]

• BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images.
  Thu Nguyen-Phuoc, Christian Richardt, Long Mai, Yong-Liang Yang, Niloy Mitra.
  NeurIPS 2020. [Paper] [Project] [Code]

• GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis.
  Katja Schwarz, Yiyi Liao, Michael Niemeyer, Andreas Geiger.
  NeurIPS 2020. [Paper] [Project] [Code]

• HoloGAN: Unsupervised learning of 3D representations from natural images.
  Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, Yong-liang Yang.
  ICCV 2019. [Paper] [Project] [Code]

Conditional 3D Generative Models

• Sem2NeRF: Converting Single-View Semantic Masks to Neural Radiance Fields.
  Yuedong Chen, Qianyi Wu, Chuanxia Zheng, Tat-Jen Cham, Jianfei Cai.
  ECCV 2022. [Paper] [Project] [Code]

• IDE-3D: Interactive Disentangled Editing for High-Resolution 3D-aware Portrait Synthesis.
  Jingxiang Sun, Xuan Wang, Yichun Shi, Lizhen Wang, Jue Wang, Yebin Liu.
  SIGGRAPH Asia 2022. [Paper] [Project] [Code]

• NeRFFaceEditing: Disentangled Face Editing in Neural Radiance Fields.
  Kaiwen Jiang, Shu-Yu Chen, Feng-Lin Liu, Hongbo Fu, Lin Gao.
  SIGGRAPH Asia 2022. [Paper] [Project]

• GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds.
  Zekun Hao, Arun Mallya, Serge Belongie, Ming-Yu Liu.
  ICCV 2021. [Paper] [Project] [Code]

3D-aware Diffusion Models for a Single Image Category

• Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction.
  Hansheng Chen, Jiatao Gu, Anpei Chen, Wei Tian, Zhuowen Tu, Lingjie Liu, Hao Su.
  ICCV 2023. [PDF] [Project] [Code]

• 3D-aware Image Generation using 2D Diffusion Models.
  Jianfeng Xiang, Jiaolong Yang, Binbin Huang, Xin Tong.
  ICCV 2023. [Paper] [Project] [Code]

• HoloFusion: Towards Photo-realistic 3D Generative Modeling.
  Animesh Karnewar, Niloy J. Mitra, Andrea Vedaldi, David Novotny.
  ICCV 2023. [Paper] [Project]

• HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusion.
  Ziya Erkoç, Fangchang Ma, Qi Shan, Matthias Nießner, Angela Dai.
  ICCV 2023. [Paper] [Project]

• LatentSwap3D: Semantic Edits on 3D Image GANs.
  Enis Simsar, Alessio Tonioni, Evin Pınar Örnek, Federico Tombari.
  ICCV 2023 Workshop on AI3DCC. [Paper]

• DiffusioNeRF: Regularizing Neural Radiance Fields with Denoising Diffusion Models.
  Jamie Wynn and Daniyar Turmukhambetov.
  CVPR 2023. [Paper] [Supplementary material] [COde]

• NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion Models.
  Seung Wook Kim, Bradley Brown, Kangxue Yin, Karsten Kreis, Katja Schwarz, Daiqing Li, Robin Rombach, Antonio Torralba, Sanja Fidler.
  CVPR 2023. [Paper] [Project]

• Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion.
  Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, Baining Guo.
  CVPR 2023. [Paper] [Project]

• DiffRF: Rendering-guided 3D Radiance Field Diffusion.
  Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulò, Peter Kontschieder, Matthias Nießner.
  CVPR 2023 (Highlight). [Paper] [Project]

• RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation.
  Titas Anciukevičius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy J. Mitra, Paul Guerrero.
  CVPR 2023. [Paper] [Project] [Code]

• SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstruction.
  Zhizhuo Zhou, Shubham Tulsiani.
  CVPR 2023. [Paper] [Project] [Code]

• HoloDiffusion: Training a 3D Diffusion Model using 2D Images.
  Animesh Karnewar, Andrea Vedaldi, David Novotny, Niloy Mitra.
  CVPR 2023. [Paper] [Project]

• 3DiM: Novel View Synthesis with Diffusion Models.
  Daniel Watson, William Chan, Ricardo Martin-Brualla, Jonathan Ho, Andrea Tagliasacchi, Mohammad Norouzi.
  ICLR 2023. [Paper] [Project]

• 3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models.
  Biao Zhang, Jiapeng Tang, Matthias Niessner, Peter Wonka.
  SIGGRAPH 2023. [Paper] [Project] [Code]

• GAUDI: A Neural Architect for Immersive 3D Scene Generation.
  Miguel Angel Bautista, Pengsheng Guo, Samira Abnar, Walter Talbott, Alexander Toshev, Zhuoyuan Chen, Laurent Dinh, Shuangfei Zhai, Hanlin Goh, Daniel Ulbricht, Afshin Dehghan, Josh Susskind.
  NeurIPS 2022. [Paper] [Project]

• Learning a Diffusion Prior for NeRFs.
  Guandao Yang, Abhijit Kundu, Leonidas J. Guibas, Jonathan T. Barron, Ben Poole.
  arxiv 2023. [Paper]

• Adding 3D Geometry Control to Diffusion Models.
  Wufei Ma, Qihao Liu, Jiahao Wang, Angtian Wang, Yaoyao Liu, Adam Kortylewski, Alan Yuille.
  arxiv 2023. [Paper]

• Generative Novel View Synthesis with 3D-Aware Diffusion Models.
  Eric R. Chan, Koki Nagano, Matthew A. Chan, Alexander W. Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, Gordon Wetzstein.
  arxiv 2023. [Paper] [Project] [Code]

• 3D-LDM: Neural Implicit 3D Shape Generation with Latent Diffusion Models.
  Gimin Nam, Mariem Khlifi, Andrew Rodriguez, Alberto Tono, Linqi Zhou, Paul Guerrero.
  arxiv 2022. [Paper]

3D-Aware Generative Models on ImageNet

• VQ3D: Learning a 3D-Aware Generative Model on ImageNet.
  Kyle Sargent, Jing Yu Koh, Han Zhang, Huiwen Chang, Charles Herrmann, Pratul Srinivasan, Jiajun Wu, Deqing Sun.
  ICCV 2023 (Oral). [Paper] [Project]

• 3D Generation on ImageNet.
  Ivan Skorokhodov, Aliaksandr Siarohin, Yinghao Xu, Jian Ren, Hsin-Ying Lee, Peter Wonka, Sergey Tulyakov.
  ICLR 2023 (Oral). [Paper] [Project] [Code]

3D-aware Video Synthesis

• 3D-Aware Video Generation.
  Sherwin Bahmani, Jeong Joon Park, Despoina Paschalidou, Hao Tang, Gordon Wetzstein, Leonidas Guibas, Luc Van Gool, Radu Timofte.
  TMLR 2023. [Paper] [Project] [Code]

• Streaming Radiance Fields for 3D Video Synthesis.
  Lingzhi Li, Zhen Shen, Zhongshu Wang, Li Shen, Ping Tan.
  NeurIPS 2022. [Paper] [Code]

INR-based 3D Novel View Synthesis

Neural Scene Representations

• Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations.
  Mehdi S. M. Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lucic, Daniel Duckworth, Alexey Dosovitskiy, Jakob Uszkoreit, Thomas Funkhouser, Andrea Tagliasacchi.
  CVPR 2022. [Paper] [Project] [Code]

• Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering.
  Vincent Sitzmann, Semon Rezchikov, William T. Freeman, Joshua B. Tenenbaum, Fredo Durand.
  NeurIPS 2021 (Spotlight). [Paper] [Project] [Code]

• Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields.
  Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, Pratul P. Srinivasan.
  ICCV 2021. [Paper] [Project] [Github]

• NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis.
  Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng.
  ECCV 2020. [Paper] [Project] [Gtihub-Tensorflow] [krrish94-PyTorch] [yenchenlin-PyTorch]

• Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision.
  Michael Niemeyer, Lars Mescheder, Michael Oechsle, Andreas Geiger.
  CVPR 2020. [Paper] [Code]

• Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations.
  Vincent Sitzmann, Michael Zollhöfer, Gordon Wetzstein.
  NeurIPS 2019 (Oral, Honorable Mention "Outstanding New Directions"). [Paper] [Project] [Code] [Dataset]

• LLFF: Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines.
  Ben Mildenhall, Pratul Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, Abhishek Kar.
  SIGGRAPH 2019. [Paper] [Project] [Code]

• DeepVoxels: Learning Persistent 3D Feature Embeddings.
  Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, Michael Zollhöfer.
  CVPR 2019 (Oral). [Paper] [Project] [Code]

Acceleration

• Instant Neural Graphics Primitives with a Multiresolution Hash Encoding.
  Thomas Müller, Alex Evans, Christoph Schied, Alexander Keller.
  SIGGRAPH (TOG) 2022. [Paper] [Project] [Code]

• DIVeR: Real-time and Accurate Neural Radiance Fields with Deterministic Integration for Volume Rendering.
  Liwen Wu, Jae Yong Lee, Anand Bhattad, Yuxiong Wang, David A. Forsyth.
  CVPR 2022. [Paper] [Project] [Code]

• KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs.
  Christian Reiser, Songyou Peng, Yiyi Liao, Andreas Geiger.
  ICCV 2021. [Paper] [Code]

• FastNeRF: High-Fidelity Neural Rendering at 200FPS.
  Stephan J. Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, Julien Valentin.
  ICCV 2021. [Paper]

• PlenOctrees for Real-time Rendering of Neural Radiance Fields.
  Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, Angjoo Kanazawa.
  ICCV 2021. [Paper] [Project] [Code]

• Baking Neural Radiance Fields for Real-Time View Synthesis.
  Peter Hedman, Pratul P. Srinivasan, Ben Mildenhall, Jonathan T. Barron, Paul Debevec.
  ICCV 2021 (oral). [Paper] [Project] [Code]

• AutoInt: Automatic Integration for Fast Neural Volume Rendering.
  David B. Lindell, Julien N. P. Martel, Gordon Wetzstein.
  CVPR 2021 (oral). [Paper] [Project] [Code]

• NSVF: Neural Sparse Voxel Fields.
  Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, Christian Theobalt.
  NeurIPS 2020. [Paper] [Project] [Code]

From Constrained to In-the-wild Conditions

Few Images

• GRF: Learning a General Radiance Field for 3D Representation and Rendering.
  Alex Trevithick, Bo Yang.
  ICCV 2021. [Paper] [Code]

• MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo.
  Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, Hao Su.
  ICCV 2021. [Paper] [Project] [Code]

• CodeNeRF: Disentangled Neural Radiance Fields for Object Categories.
  Wonbong Jang, Lourdes Agapito.
  ICCV 2021. [Paper] [Project] [Code]

• pixelNeRF: Neural Radiance Fields from One or Few Images.
  Alex Yu, Vickie Ye, Matthew Tancik, Angjoo Kanazawa.
  CVPR 2021. [Paper] [Project] [Code]

• IBRNet: Learning Multi-View Image-Based Rendering.
  Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Martin-Brualla, Noah Snavely, Thomas Funkhouser.
  CVPR 2021. [Paper] [Project] [Code]

• NeRF-VAE: A Geometry Aware 3D Scene Generative Model.
  Adam R. Kosiorek, Heiko Strathmann, Daniel Zoran, Pol Moreno, Rosalia Schneider, Soňa Mokrá, Danilo J. Rezende.
  ICML 2021. [Paper]

Pose-free

• Self-Calibrating Neural Radiance Fields.
  Yoonwoo Jeong, Seokjun Ahn, Christopher Choy, Animashree Anandkumar, Minsu Cho, Jaesik Park.
  ICCV 2021. [Paper] [Project] [Code]

• BARF: Bundle-Adjusting Neural Radiance Fields.
  Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, Simon Lucey.
  ICCV 2021. [Paper] [Code]

• NeRF--: Neural Radiance Fields Without Known Camera Parameters.
  Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, Victor Adrian Prisacariu.
  arxiv 2021. [Paper] [Project] [Code]

Varying Appearance

• NeRFReN: Neural Radiance Fields with Reflections.
  Yuan-Chen Guo, Di Kang, Linchao Bao, Yu He, Song-Hai Zhang.
  CVPR 2022. [Paper] [[Project](https://bennyguo.github.io/nerfren/]

• NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections.
  Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, Daniel Duckworth.
  CVPR 2021 (oral). [Paper] [Code]

Large-scale Scene

• Grid-guided Neural Radiance Fields for Large Urban Scenes.
  Linning Xu, Yuanbo Xiangli, Sida Peng, Xingang Pan, Nanxuan Zhao, Christian Theobalt, Bo Dai, Dahua Lin.
  CVPR 2023. [Paper] [Project]

• S3-NeRF: Neural Reflectance Field from Shading and Shadow under a Single Viewpoint.
  Wenqi Yang, Guanying Chen, Chaofeng Chen, Zhenfang Chen, Kwan-Yee K. Wong.
  NeurIPS 2022. [Paper] [Project]

• BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering.
  Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, Dahua Lin.
  ECCV 2022. [Paper] [Project]

• Block-NeRF: Scalable Large Scene Neural View Synthesis.
  Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P. Srinivasan, Jonathan T. Barron, Henrik Kretzschmar.
  CVPR 2022. [Paper] [Project]

• Urban Radiance Fields.
  Konstantinos Rematas, Andrew Liu, Pratul P. Srinivasan, Jonathan T. Barron, Andrea Tagliasacchi, Thomas Funkhouser, Vittorio Ferrari.
  CVPR 2022. [Paper] [Project]

• Mega-NERF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs.
  Haithem Turki, Deva Ramanan, Mahadev Satyanarayanan.
  CVPR 2022. [Paper] [Code]

• Shadow Neural Radiance Fields for Multi-view Satellite Photogrammetry.
  Dawa Derksen, Dario Izzo.
  CVPR 2021. [Paper] [Code]

Dynamic Scene

• NeRFPlayer: A Streamable Dynamic Scene Representation with Decomposed Neural Radiance Fields.
  Liangchen Song, Anpei Chen, Zhong Li, Zhang Chen, Lele Chen, Junsong Yuan, Yi Xu, Andreas Geiger.
  TVCG 2023. [Paper] [Project]

• Generative Deformable Radiance Fields for Disentangled Image Synthesis of Topology-Varying Objects.
  Ziyu Wang, Yu Deng, Jiaolong Yang, Jingyi Yu, Xin Tong.
  Pacific Graphics 2022. [Paper] [Code]

• Neural Surface Reconstruction of Dynamic Scenes with Monocular RGB-D Camera.
  Hongrui Cai, Wanquan Feng, Xuetao Feng, Yan Wang, Juyong Zhang.
  NeurIPS 2022. [Paper] [Project] [Code]

• LoRD: Local 4D Implicit Representation for High-Fidelity Dynamic Human Modeling.
  Boyan Jiang, Xinlin Ren, Mingsong Dou, Xiangyang Xue, Yanwei Fu, Yinda Zhang.
  ECCV 2022. [Paper] [Code]

• Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-time.
  Liao Wang, Jiakai Zhang, Xinhang Liu, Fuqiang Zhao, Yanshun Zhang, Yingliang Zhang, Minye Wu, Lan Xu, Jingyi Yu.
  CVPR 2022 (Oral). [Paper] [Project]

• CoNeRF: Controllable Neural Radiance Fields.
  Kacper Kania, Kwang Moo Yi, Marek Kowalski, Tomasz Trzciński, Andrea Taliasacchi.
  CVPR 2022. [Paper] [Project]

• Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Deforming Scene from Monocular Video.
  Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, Christian Theobalt.
  ICCV 2021. [Paper] [Project] [Code]

• NeRFlow: Neural Radiance Flow for 4D View Synthesis and Video Processing.
  Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B. Tenenbaum, Jiajun Wu.
  ICCV 2021. [Paper] [Project]

• Nerfies: Deformable Neural Radiance Fields.
  Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, Ricardo-Martin Brualla.
  ICCV 2021. [Paper] [Project] [Code]

• D-NeRF: Neural Radiance Fields for Dynamic Scenes.
  Albert Pumarola, Enric Corona, Gerard Pons-Moll, Francesc Moreno-Noguer.
  CVPR 2021. [Paper] [Project] [Code] [Data]

• Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction.
  Guy Gafni, Justus Thies, Michael Zollhöfer, Matthias Nießner.
  CVPR 2021. [Paper] [Project] [Video]

• NSFF: Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes.
  Zhengqi Li, Simon Niklaus, Noah Snavely, Oliver Wang.
  CVPR 2021. [Paper] [Project] [Code]

• Space-time Neural Irradiance Fields for Free-Viewpoint Video.
  Wenqi Xian, Jia-Bin Huang, Johannes Kopf, Changil Kim.
  CVPR 2021. [Paper] [Project]

The following papers are not directly related to 3D-aware image synthesis. But it would be beneficial to pay attention to those works. For example, in our survey, inverse rendering are not classified as 3D-aware image synthesis as they are not deliberately designed for this purpose. But with the inferred intrinsic components, photorealistic images can be rendered. 3D reconstruction models geometry only with no appearance information, meaning them not able to render images with photorealistic textures. But these representations have been introduced as the geometric representation along with a textural representation (e.g., Texture Field) for 3D-aware image synthesis.

3D Representations

• K-Planes: Explicit Radiance Fields in Space, Time, and Appearance.
  Sara Fridovich-Keil, Giacomo Meanti, Frederik Warburg, Benjamin Recht, Angjoo Kanazawa.
  CVPR 2023. [Paper] [Project] [Code]

• HexPlane: A Fast Representation for Dynamic Scenes.
  Ang Cao, Justin Johnson.
  CVPR 2023. [Paper] [Project] [Code]

• GIFS: Neural Implicit Function for General Shape Representation.
  Jianglong Ye, Yuntao Chen, Naiyan Wang, Xiaolong Wang.
  CVPR 2022. [Paper] [Project] [Code]

• Geometry-Consistent Neural Shape Representation with Implicit Displacement Fields.
  Wang Yifan, Lukas Rahmann, Olga Sorkine-Hornung.
  ICLR 2022. [Paper] [Project] [Code]

• Neural Volumes: Learning Dynamic Renderable Volumes from Images.
  Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, Yaser Sheikh.
  TOG 2019. [Paper] [Code]

• DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation.
  eong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, Steven Lovegrove.
  CVPR 2019. [Paper] [Code]

• Occupancy Networks: Learning 3D Reconstruction in Function Space.
  Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, Andreas Geiger.
  CVPR 2019. [Paper] [Project] [Code]

Neural Inverse Rendering (Neural De-rendering)

Inverse rendering is to infer underlying intrinsic components of a scene from rendered 2D images. These properties include shape (surface, depth, normal), material (albedo, reflectivity, shininess), and lighting (direction, intensity), which can be further used to render photorealistic images.

• NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination.
  Xiuming Zhang, Pratul P. Srinivasan, Boyang Deng, Paul Debevec, William T. Freeman, Jonathan T. Barron.
  SIGGRAPH Asia 2021. [Paper] [Project] [Code]

• Extracting Triangular 3D Models, Materials, and Lighting From Images.
  Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, Sanja Fidler.
  CVPR 2022. [Paper] [Code] [Project]

• Modeling Indirect Illumination for Inverse Rendering.
  Yuanqing Zhang, Jiaming Sun, Xingyi He, Huan Fu, Rongfei Jia, Xiaowei Zhou.
  CVPR 2022. [Paper]

• IRISformer: Dense Vision Transformers for Single-Image Inverse Rendering in Indoor Scenes.
  Rui Zhu, Zhengqin Li, Janarbek Matai, Fatih Porikli, Manmohan Chandraker.
  CVPR 2022. [Paper]

• De-rendering 3D Objects in the Wild.
  Felix Wimbauer, Shangzhe Wu, Christian Rupprecht.
  CVPR 2022. [Paper] [Project] [Code]

• GAN2X: Non-Lambertian Inverse Rendering of Image GANs.
  Xingang Pan, Ayush Tewari, Lingjie Liu, Christian Theobalt.
  3DV 2022. [Paper] [Project]

• PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting.
  Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, Noah Snavely.
  CVPR 2021. [Paper] [Project]

• Unified Shape and SVBRDF Recovery using Differentiable Monte Carlo Rendering.
  Fujun Luan, Shuang Zhao, Kavita Bala, Zhao Dong.
  EGSR 2021. [Paper] [Project] [Video]

• Invertible Neural BRDF for Object Inverse Rendering.
  Zhe Chen, Shohei Nobuhara, Ko Nishino.
  ECCV 2020. [Paper] [Code]

• Polarimetric Multi-View Inverse Rendering.
  Jinyu Zhao, Yusuke Monno, Masatoshi Okutomi.
  ECCV 2020. [Paper]

• Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF From a Single Image.
  Zhengqin Li, Mohammad Shafiei, Ravi Ramamoorthi, Kalyan Sunkavalli, Manmohan Chandraker.
  CVPR 2020.[Paper] [Project] [Code]

• DRWR: A Differentiable Renderer without Rendering for Unsupervised 3D Structure Learning from Silhouette Images.
  Zhizhong Han, Chao Chen, Yu-Shen Liu, Matthias Zwicker.
  ICML 2020. [Paper]

• Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer.
  Wenzheng Chen, Jun Gao, Huan Ling, Edward J. Smith, Jaakko Lehtinen, Alec Jacobson, Sanja Fidler.
  NeurIPS 2019. [Paper] [Code]

• InverseRenderNet: Learning Single Image Inverse Rendering.
  Ye Yu, William A. P. Smith.
  CVPR 2019. [Paper] [Code] [IIW Dataset]

Neural Rerendering

• Hybrid Neural Fusion for Full-frame Video Stabilization.
  Yu-Lun Liu, Wei-Sheng Lai, Ming-Hsuan Yang, Yung-Yu Chuang, Jia-Bin Huang.
  ICCV 2021. [Paper] [Code]

• Neural Lumigraph Rendering.
  Petr Kellnhofer, Lars Jebe, Andrew Jones, Ryan Spicer, Kari Pulli, Gordon Wetzstein.
  CVPR 2021. [Paper] [Project] [Data]

• Neural Re-Rendering of Humans from a Single Image.
  Kripasindhu Sarkar, Dushyant Mehta, Weipeng Xu, Vladislav Golyanik, Christian Theobalt.
  ECCV 2020. [Paper]

• Neural Rerendering in the Wild.
  Moustafa Meshry, Dan B Goldman, Sameh Khamis, Hugues Hoppe, Rohit Pandey, Noah Snavely, Ricardo Martin-Brualla.
  CVPR 2019. [Paper]

• Revealing Scenes by Inverting Structure from Motion Reconstructions.
  Francesco Pittaluga, Sanjeev J. Koppal, Sing Bing Kang, Sudipta N. Sinha.
  CVPR 2019. [Paper]

Datasets

Summary of popular 3D-aware image synthesis datasets.

Multi-view image collections

The images are rendered or collected according to different experimental settings, such as Synthetic-NeRF dataset, the DTU dataset, and the Tanks and Temples dataset for general purposes, the crowded Phototourism dataset for varying lighting conditions, the Blender Forward Facing (BLEFF) dataset to benchmark camera parameter estimation and novel view synthesis quality, and the San Francisco Alamo Square Dataset for large-scale scenes.

Examples of multi-view image datasets.

Single-view image collections

Summary of popular single-view image datasets organized by their major categories and sorted by their popularity.

Citation

If this repository benefits your research, please consider citing our paper.

  @inproceedings{xia2023survey,
    title={A Survey on Deep Generative 3D-aware Image Synthesis},
    author={Xia, Weihao and Xue, Jing-Hao},
    booktitle={ACM Computing Surveys (CSUR)},
    year={2023}
  }

License

This work is licensed under a Creative Commons Attribution 4.0 International License.