Augmenting Deformable Surfaces

Click on the images to jump to corresponding results.

Starting from our earlier approach to fast keypoint matching, we developed real-time techniques to detect deformable surfaces. Using wide-baseline correspondences between an input image and a model image, we recover quickly and accurately the 2D or 3D shape of a deforming surface. In addition, we also model the lighting and detect occlusions.

J. Pilet, P. Fua

Results

Painting a Platypus a Deforming TShirt [top] Given the TShirt and platypus images shown below, we recover the deformations of the tshirt in the video shown to the right. We then deform the platypus image accordingly, relight it, and add the result to the original images while taking occlusions into account.
Magic Book [top] The same technology was used by Camille Scherrer, an designer, to create a magic book.
Augmenting a 3D Surface[top] In the examples above, we only recover 2D surface deformations. We can also use our techniques for monocular 3D shape recovery to recover the deformations of the paper and augment it with fully 3D objects.

Code

• BazAR: A vision based fast detection library.
• A background subtraction algorithm, robust to sudden light changes.

Materials

• Tutorial slides on Modeling Deformable Surfaces from Single Videos

References

• J. Pilet, V. Lepetit, and P. Fua, Fast Non-Rigid Surface Detection, Registration and Realistic Augmentation, International Journal of Computer Vision, Vol. 76, Nr. 2, February 2008.
• C. Scherrer, J. Pilet, V. Lepetit and P. Fua, Souvenirs du Monde des Montagnes, Leonardo, special issue on ACM SIGGRAPH, Vol. 42, Nr. 4, pp. 350 - 355, August 2009.

Contacts

Last Updated: Monday, November 15, 2010