Neural 3D Mesh Renderer
CVPR 20183D Mesh Reconstruction
2D-to-3D Style Transfer
3D DeepDream
Highlights
- The first work to introduce differentiable polygon mesh rendering into neural networks.
- The first work to apply neural style transfer to 3D models.
- Recipient of the NVIDIA Pioneering Research Award, with over 1,000 citations.
Abstract
For modeling the 3D world behind 2D images, which 3D representation is most appropriate? A polygon mesh is a promising candidate for its compactness and geometric properties. However, it is not straightforward to model a polygon mesh from 2D images using neural networks because the conversion from a mesh to an image, or rendering, involves a discrete operation called rasterization, which prevents back-propagation. Therefore, in this work, we propose an approximate gradient for rasterization that enables the integration of rendering into neural networks. Using this renderer, we perform single-image 3D mesh reconstruction with silhouette image supervision and our system outperforms the existing voxel-based approach. Additionally, we perform gradient-based 3D mesh editing operations, such as 2D-to-3D style transfer and 3D DeepDream, with 2D supervision for the first time. These applications demonstrate the potential of the integration of a mesh renderer into neural networks and the effectiveness of our proposed renderer.
Results
Single-image 3D reconstruction
Our rendering layer enables neural networks to learn to reconstruct a 3D mesh from a single image.
2D-to-3D style transfer
Our method transfers the style of a painting to both the shape and texture of a 3D model. Notice how the models’ outlines change to reflect the style of each painting.
The style images used are Thomson No. 5 (Yellow Sunset) (D. Coupland, 2011), The Tower of Babel (P. Bruegel the Elder, 1563), The Scream (E. Munch, 1910), and Portrait of Pablo Picasso (J. Gris, 1912).
3D DeepDream
We extend DeepDream to 3D meshes.
Technical overview
Understanding the 3D world from 2D images is a fundamental challenge in computer vision. Rendering connects the two by converting 3D scenes into 2D images. Polygon meshes provide a compact, expressive, and intuitive representation of 3D shapes, motivating the development of a mesh renderer that supports backpropagation.
Conventional mesh rendering cannot be directly integrated into neural networks trained with backpropagation because rasterization is a discrete operation. We propose an approximate gradient for rasterization, enabling end-to-end training of neural networks that include a mesh renderer.
We used this renderer for the applications shown above. The figure below illustrates the reconstruction pipeline.
We trained the 3D mesh generator using silhouette images, minimizing the difference between the silhouettes of the reconstructed shapes and the ground-truth silhouettes.
For 2D-to-3D style transfer, we optimized the mesh’s shape and texture to minimize a style loss computed on the rendered images. We used a similar approach for 3D DeepDream.
In both applications, our renderer propagates gradients from 2D image space back to the 3D mesh.
More details can be found in the paper.
Citation
@InProceedings{kato2018renderer,
title = {Neural 3D Mesh Renderer},
author = {Kato, Hiroharu and Ushiku, Yoshitaka and Harada, Tatsuya},
booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2018}
}
Related Projects
powered by Academic Project Page Template