Monocular 3D Shape Recovery

Abstract

Recovering the 3D shape of a nonrigid surface from a single viewpoint is known to be both ambiguous and challenging. Resolving the ambiguities typically requires prior knowledge about the most likely deformations that the surface may undergo. Here we extend the Laplacian formalism, which was first introduced in the Graphics  community to regularize  3D meshes, to achieve this goal given  correspondences with  a reference image in which the shape is known.

Our  approach   allows  us   to  reduce  the   dimensionality  of   the  surface reconstruction problem without sacrificing accuracy, thus allowing for real-time implementations.

The algorithm runs in realtime on tablets. It can detect which deformable object appears in the camera stream, then track and reconstruct the corresponding 3D shape as the object deforms over time.

We also use our algorithm to build an augmented reality coloring book App, which runs in realtime on tablets. The App tracks a possibly non-planar drawing page, precisely takes its colors to texture a virtual animated character. This work was done in colloboration with Disney Research Zurich.

Results

Real-Time application on Tablets

Real-Time Demo on PC

Modeling the Impact of a Baseball

We collaborated with Prof. Lloyd Smith, WSU’s Sport Science Laboratory and obtained 7000 frames-per-second videos of a baseball colliding with a cylinder that serves as the bat to study its behavior. The videos show the very strong deformation that occurs at the moment of impact. We take the reference frame to be the first one where the ball is undeformed and can be represented by a spherical triangulation of diameter 73.52 mm. Then, we can reconstructed the 3D shapes of the baseball when it hits the cylinder.

Reprojection of 3D reconstructed mesh on image plane

3D reconstruction of the baseball during the collision with a bat

On-contact vertices during the impact

Code and Supplementary Material

Code and instructions for running it are available here.

Supplementary material can be downloaded as a zip-file supplementary.zip or viewed on a webpage README.html.

References

T. D. Ngo; J. Östlund; P. Fua : Template-based Monocular 3D Shape Recovery using Laplacian Meshes; IEEE Transactions on Pattern Analysis and Machine Intelligence. 2016. DOI : 10.1109/Tpami.2015.2435739.
J. O. M. Östlund; A. Varol; T. D. Ngo; P. Fua : Laplacian Meshes for Monocular 3D Shape Recovery. 2012. European Conference on Computer Vision, Florence, October 2012.
M. Salzmann; P. Fua : Linear Local Models for Monocular Reconstruction of Deformable Surfaces; IEEE Transactions on Pattern Analysis and Machine Intelligence. 2011. DOI : 10.1109/TPAMI.2010.158.