GHNeRF: Learning Generalizable Human Features with Efficient Neural Radiance Fields

Arnab Dey1, Di Yang2, Rohith Agaram3, Antitza Dantcheva2, Andrew I. Comport1, Srinath Sridhar4, Jean Martinet1,
1I3S/CNRS University Cote d'Azur 2INRIA 3International Institute of Information Technology, Hyderabad 4Brown University
Paper arXiv CVPR Code (soon) Data (soon)

More details coming soon

GHNeRF Method

Figure 2. Overview of the GHNeRF pipeline: Given an input image I, human features fh and multi-resolution image features fimg can be extracted using a 2D image encoder and a 2D CNN respectively. Subsequently, fimg is used to form a cost volume for depth prediction. The predicted depth is used for depth-guided sampling to reduce the number of samples along the ray. For each 3D sample point x along the ray, we combine image and voxel features to input an MLP gNeRF, generating the intermediate NeRF feature VNeRF. Finally, the intermediate NeRF feature VNeRF and the human feature fh are concatenated and fed into a smaller MLP gh to produce heatmaps. Furthermore, VNeRF and the view direction d are combined in another MLP gc to derive color c. The final pixel color and heatmaps are generated using volume rendering technique.

Abstract

Recent advances in Neural Radiance Fields (NeRF) have demonstrated promising results in 3D scene representa- tions, including 3D human representations. However, these representations often lack crucial information on the un- derlying human pose and structure, which is crucial for AR/VR applications and games. In this paper, we intro- duce a novel approach, termed GHNeRF, designed to ad- dress these limitations by learning 2D/3D joint locations of human subjects with NeRF representation. GHNeRF uses a pre-trained 2D encoder streamlined to extract essential human features from 2D images, which are then incorpo- rated into the NeRF framework in order to encode human biomechanic features. This allows our network to simulta- neously learn biomechanic features, such as joint locations, along with human geometry and texture. To assess the effec- tiveness of our method, we conduct a comprehensive com- parison with state-of-the-art human NeRF techniques and joint estimation algorithms. Our results show that GHNeRF can achieve state-of-the-art results in near real-time. The project website is available at arnabdey.co/ghnerf.github.io/.

Interactive Interface

RenderPeople dataset

ZJU-MoCap dataset

Related Links

BibTeX

@misc{dey2024ghnerf,
      title={GHNeRF: Learning Generalizable Human Features with Efficient Neural Radiance Fields},
      author={Arnab Dey and Di Yang and Rohith Agaram and Antitza Dantcheva and Andrew I. Comport and Srinath Sridhar and Jean Martinet},
      year={2024},
      eprint={2404.06246},
      archivePrefix={arXiv},
      primaryClass={cs.CV}}