Deformable Surface 3D Tracking

Recovering the shape of 3D deformable surfaces from monocular video sequences is a severely underconstrained problem. In this work, we explore several approaches to resolving the resulting ambiguities when the surfaces are represented as 3D triangulated meshes and 3D to 2D correspondences between surfaces and images can be established.

Click on the images to play the videos. They are encoded in MPEG4.
For Windows users: Windows Media Player or ffdshow.
For Apple users: QuickTime Player.
For linux/unix users: mplayer.

Formalizing the Nature of the Ambiguities

We have studied from a theoretical standpoint the ambiguities that occur when tracking a generic deformable surface under monocular perspective projection given 3--D to 2--D correspondences. We have shown that, additionally to the known scale ambiguity, a set of potential ambiguities can be clearly identified.
From this, we have deduced a minimal set of constraints required to disambiguate the problem and incorporate them into a working algorithm that runs on real noisy data, as shown in the examples below.

Convex Optimization

A traditional approach to resolving the ambiguities inherent to monocular 3D shape reconstruction is to introduce deformation models that enforce smoothness constraints. However, such constraints would not work in cases such as the ones below where the surfaces deform in non-smooth fashion. Instead, we simply disallow large changes of edge orientation between consecutive frames, which is a generally applicable constraint when tracking a surface at 25 frames-per-second. Furthermore, this can be formulated as a Second Order Cone Programming feasibility problem, which yields a well-posed optimization problem with a single minimum.

Dimensionality Reduction

In the examples above, the shape recovery problem was formulated in terms of the coordinates of the vertices of a mesh. This leads to very high-dimensional problems that can be very inefficient. Machine learning methods have been used in other vision domains to reduce the dimensionality of large problems. One of their main limitations is that they require training examples that might be very hard to obtain and register. Instead, we have proposed an approach to creating such examples by randomly setting a subset of the angles between the facets of the mesh, that we show to be sufficient to fully determine the shape of the surface. We then apply Principal Component Analysis on these examples, and track deforming surfaces at a frame-rate ranging between 0.5 and 2.5 frames-per-second.

Closed-Form 3D Reconstruction

Up to now, the proposed methods tracked the surface from frame to frame in a video sequence. This has the weaknesses of requiring a good initialization in the first image, and of yielding results that drift along the sequence. To overcome these issues, we have proposed a closed-form solution to shape recovery. We first formulate the correspondence problem as the solution of a linear system, which still lets us use our PCA model. This system suffers from ambiguities that can be identified by the eigenvectors corresponding to small eigenvalues. We can then write the reconstruction as a linear combination of such eigenvectors, and seek the weight that satisfy inextensibility constraints set on the mesh. These constraints are expressed as a system quadratic equations whose solution can be found in closed-form via Extended Linearization.

Local Low-Dimensional Models

The low-dimensional representation proposed above are global models of the surface. Their major disadvantage is that they are only valid for a particular object. Therefore, a new model must be re-built for every new object, even though it is made of the same material. Furthermore, learning a mapping from the low-dimensional space to the high-dimenional one requires many training examples to cover all the possible deformations, especially in the case of highly flexible material. Here, we have introduced local deformation models that can be learned from much smaller training sets, and that can be combined to form surfaces of arbitrary shapes. In particular, we represent our local models as Gaussian Process Latent Variable Models (GPLVM), and combine them following a Product of Expert (PoE) framework.

Datasets

Datasets for deformable surface reconstruction

Relevant Publications

M. Salzmann, V. Lepetit and P. Fua, Deformable Surface Tracking Ambiguities, Conference on Computer Vision and Pattern Recognition, Minneapolis, MI, June 2007.

M. Salzmann, R. Hartley and P. Fua, Convex Optimization for Deformable Surface 3-D Tracking, IEEE International Conference on Computer Vision, Rio de Janeiro, Brazil, October 2007.

M. Salzmann, J.Pilet, S.Ilic and P.Fua,Surface Deformation Models for Non-Rigid 3--D Shape Recovery,IEEE Transactions on Pattern Analysis and Machine Intelligence, August 2007.

M. Salzmann, F. Moreno-Noguer, V. Lepetit and P. Fua, Closed-Form Solution to Non-Rigid 3D Surface Registration, European Conference on Computer Vision, Marseille, France, October 2008.

M. Salzmann, R. Urtasun and P. Fua, Local Deformation Models for Monocular 3D Shape Recovery, Conference on Computer Vision and Pattern Recognition, Anchorage, Alaska, June 2008.

Contact:Mathieu Salzmann mathieu.salzmann@epfl.ch