Local Models for Monocular Reconstruction of Deformable 3D Surfaces

Click on the images to jump to corresponding results.

Recovering the 3D shape of a nonrigid surface from a single viewpoint is known to be both ambiguous and challenging. Resolving the ambiguities typically requires prior knowledge about the most likely deformations that the surface may undergo. It often takes the form of a global deformation model that can be learned from training data. While effective, this approach suffers from the fact that a new model must be learned for each new surface, which means acquiring new training data and may be impractical. In this work, we replaced the global models by linear local ones for surface patches, which can be assembled to represent arbitrary surface shapes as long as they are made of the same material. Not only do they eliminate the need to retrain the model for different surface shapes, they also let us formulate 3D shape reconstruction from correspondences as either an algebraic problem that can be solved in closed-form or a convex optimization problem whose solution can be found using standard numerical packages.

M. Salzmann, P. Fua

Results

We recover the 3D shape of a deforming surface represented by a triangulated mesh given point correspondences between one or more input images, in which the shape is unknown, and a reference image in which it is known. In each one of the videos below, we show

• Top left: The 3D mesh recovered by enforcing temporal consistency across three-frame sequences and reprojected on the original images.
• Bottom left: The same mesh seen from a different viewpoint.
• Top right: The mesh recovered from individual frames and without enforcing temporal consistency, which yields more jittery results.
• Bottom right: Results obtained by enforcing temporal consistency across the whole sequence as a post-processing step, which yields smoother results.

Deforming TShirt [top] The reference image.
Deforming Cloth [top] The reference image.
Deforming Cushion [top] The reference image.

Code and Data

• Code from Salzmann and Fua, CVPR 2009. The linked .tgz file contains the cardboard data also available here. It requires the SeDuMi optimization software package that can be downloaded from here.

Materials

• CVPR'09 Slides
• Tutorial slides on Modeling Deformable Surfaces from Single Videos

References

• M. Salzmann, J. Pilet, S. Ilic, P. Fua, Surface Deformation Models for Non-Rigid 3--D Shape Recovery, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 29, Nr. 8, pp. 1481 - 1487, August 2007.
• M. Salzmann and P. Fua, Linear Local Models for Monocular Reconstruction of Deformable Surfaces, IEEE Transactions on Pattern Analysis and Machine Intelligence, In Press, 2010.
• M. Salzmann and P. Fua, Deformable Surface 3D Reconstruction from Monocular Images, accepted to G. Medioni and S. Dickinson, Morgan-Claypool, 2011, Unedited draft of forthcoming monograph.

Contact

Last Updated: Monday, November 15, 2010