Implicit Meshes For 3-D Reconstruction And Tracking
The novel ways, both to represent the static surfaces, and to parameterize their deformations, are developed. This can be used both by automated algorithms for efficient 3–D shape reconstruction, and by graphics designers for editing and animation
We propose a method that combines the strength of explicit and implicit surface representations to avoid their drawbacks, and in this way build robust surface representation, called implicit mesh, suitable for automated shape recovery from video sequences.
We also advocate the use of Dirichlet Free Form Deformation (DFFD) as generic surface deformation technique that can be used to parameterize objects of arbitrary geometry defined as explicit meshes.
Combining DFFD as a generic parameterization method for explicit surfaces and implicit meshes as a generic surface representation we obtained a powerfull tool for automated shape recovery from images. However,we also argue that any other available surface parameterization can be used.
We demonstrate the applicability of our technique in case of 3–D reconstruction from images and tracking of deformable objects in monocular scenes. The examples of those reconstruction and tracking results are depicted in Figure 1. bellow.
Building Implicit Meshes
The implicit mesh model is a surface representation that unifies explicit (polygonal) and implicit surface representations in such a way that it preserves the strengths of both representations and eliminates their drawbacks. To create an implicit mesh model we must address two problems:
- Creating an implicit surface that closely approximates the shape of the initial explicit mesh
- Creating an implicit surface that closely approximates the shape of the initial explicit mesh
Our approach is depicted by Figure 2. We now discuss its components. To create implicit surface of the explicit triangular mesh we convert each mesh’ facets to a metaball primitive. Metaball primitives we consider here are:
- spherical metaballs
- triangular metaballs
Spherical metaball is an isosurface of the scalar field generated from the skeleton which is the center of the circle circumscribed around the facet as shown in Figure 2(b). Triangular metaball is an isosurface of the scalar field generated from the skeleton which is now whole facet instead of the point as in the case of spherical metaball. The parameters for the metaball include the position of the skeleton, and the potential filed function depending on the distance from the skeleton. Triangular metaball is depicted in Figure 2 (c).
Figure 2: Converting an explicit surface into an implicit shell. Upper row: Single triangle as the part of the explicit mesh (a), converted to the spherical metaball shown transparent (b) and to the triangular metaball (c). Middle row: Initial explicit surface (d), converted to the transparent implicit shell with the spherical metaballs (e) and converted to the implicit shell using triangular metaballs (f). Bottom row: Deformed explicit mesh (g), and corresponding deformed implicit shells shown transparent around the explicit one with spherical metaballs (h) and with the triangular materials(i).
Videos of Reconstruction and Tracking results
Here you can see and download some more results. Results are short movies in avi format. First we present results of 3D reconstruction and then tracking of deformable objects.
3D Reconstruction form uncalibrated video sequences
Original video. | Textured reconstructed model. |
The above images and corresponding downloadable movies demonstrate 3D reconstruction from uncalibrated video sequences of the upper body. We used the generic upper body surface model, represented as triangular implicit mesh and parameterized in terms of DFFD control points. The sequence was first calibrated using model-based bundle adjustment technique, and then stereo data cloud is extracted using simple correlation based algorithm. Silhouettes are obtained using snakes starting from manually outlined initial contours. In the left video the images form the original video sequence together with the outlined silhouettes are depicted. The depicted video is composed of 6 images, we used for the reconstruction, out of much longer sequence. Note that the small number of images did not influence quality and precision of the reconstruction shown by the video on the right. The result is textured model reprojected in the same view as the originals. The correct model alignment with the depicted silhouettes indicates quality of the reconstruction.
These images and movies depict our use of the morphable PCA face model of Blanz and Vetter to reconstruct the faces and recover the camera motion. First we used feature points on the face and the explicit triangulated mesh representing morhable face model. However, it was not sufficient to obtain the precise reconstruction on the occluding contours (silhouettes) as depicted by the middle image of the given video. To improve the reconstruction result we used our implicit mesh formalism. We then automatically detected silhouettes in the images, which are depicted in yellow in the given video, and used them together with the feature points to improve the recovered shape as shown in the right end image of the video. The implicit mesh surface representation was actually used for robust automatic silhouette handling, which is important when one wants to track deformable objects in the scenes with clutter, changing background and partial occlusions.
Tracking of 3D deformable objects in monocular videos
Here we use single video sequences to track upper body motion and the deformation of a piece of paper. The results are shown in the videos bellow.
In the first paper sequence, the background changes because of the translating book. In the second one, the hand occludes the sheet of paper. Our algorithm is impervious to both of these difficulties.
Publications
Field Guide to Northern Tree-related Microhabitats: Descriptions and size limits for their inventory in boreal and hemiboreal forests of Europe and North America
Swiss Federal Institute for Forest, Snow and Landscape Research WSL, Switzerland, 2024.Data Champions Lunch Talks – Green Bytes: Data-Driven Approaches to EPFL Sustainability
Data Champions Lunch Talks – Green Bytes: Data-Driven Approaches to EPFL Sustainability, EPFL, CM 1 221, April 18, 2024.Comparison of Three Viral Nucleic Acid Preamplification Pipelines for Sewage Viral Metagenomics
Food and Environmental Virology. 2024. DOI : 10.1007/s12560-024-09594-3.How to Support Students to Develop Skills that Promote Sustainability
Teaching Transversal Skills for Engineering Studens: A Practical Handbook of Activities with Tangibles; EPFL, 2024.How to Support Students Giving Each Other Constructive Feedback, Especially When It Is Difficult to Hear
Teaching Transversal Skills for Engineering Studens: A Practical Handbook of Activities with Tangibles; EPFL, 2024.How teachers can use the 3T PLAY trident framework to design an activity that develops transversal skills
Teaching Transversal Skills for Engineering Studens: A Practical Handbook of Activities with Tangibles; EPFL, 2024.The conceptual foundations of innate immunity: Taking stock 30 years later
Immunity. 2024-04-09. Vol. 57, num. 4, p. 613-631. DOI : 10.1016/j.immuni.2024.03.007.Radio-Activities: Architecture and Broadcasting in Cold War Berlin
Cambridge, MA; London: MIT Press, 2024.No Last One
Revue Matières. 2024. num. 18.All That is Solid
Transcalar Prospects in Climate Crisis; Zurich: Lars Müller, 2024.Implicit Meshes for Surface Reconstruction
IEEE Transactions on Pattern Analysis and Machine Intelligence. 2006. Vol. 28, num. 2, p. 328-333. DOI : 10.1109/TPAMI.2006.37.Field Guide to Northern Tree-related Microhabitats: Descriptions and size limits for their inventory in boreal and hemiboreal forests of Europe and North America
Swiss Federal Institute for Forest, Snow and Landscape Research WSL, Switzerland, 2024.Data Champions Lunch Talks – Green Bytes: Data-Driven Approaches to EPFL Sustainability
Data Champions Lunch Talks – Green Bytes: Data-Driven Approaches to EPFL Sustainability, EPFL, CM 1 221, April 18, 2024.Comparison of Three Viral Nucleic Acid Preamplification Pipelines for Sewage Viral Metagenomics
Food and Environmental Virology. 2024. DOI : 10.1007/s12560-024-09594-3.How to Support Students to Develop Skills that Promote Sustainability
Teaching Transversal Skills for Engineering Studens: A Practical Handbook of Activities with Tangibles; EPFL, 2024.How to Support Students Giving Each Other Constructive Feedback, Especially When It Is Difficult to Hear
Teaching Transversal Skills for Engineering Studens: A Practical Handbook of Activities with Tangibles; EPFL, 2024.How teachers can use the 3T PLAY trident framework to design an activity that develops transversal skills
Teaching Transversal Skills for Engineering Studens: A Practical Handbook of Activities with Tangibles; EPFL, 2024.The conceptual foundations of innate immunity: Taking stock 30 years later
Immunity. 2024-04-09. Vol. 57, num. 4, p. 613-631. DOI : 10.1016/j.immuni.2024.03.007.Radio-Activities: Architecture and Broadcasting in Cold War Berlin
Cambridge, MA; London: MIT Press, 2024.No Last One
Revue Matières. 2024. num. 18.All That is Solid
Transcalar Prospects in Climate Crisis; Zurich: Lars Müller, 2024.Field Guide to Northern Tree-related Microhabitats: Descriptions and size limits for their inventory in boreal and hemiboreal forests of Europe and North America
Swiss Federal Institute for Forest, Snow and Landscape Research WSL, Switzerland, 2024.Data Champions Lunch Talks – Green Bytes: Data-Driven Approaches to EPFL Sustainability
Data Champions Lunch Talks – Green Bytes: Data-Driven Approaches to EPFL Sustainability, EPFL, CM 1 221, April 18, 2024.Comparison of Three Viral Nucleic Acid Preamplification Pipelines for Sewage Viral Metagenomics
Food and Environmental Virology. 2024. DOI : 10.1007/s12560-024-09594-3.How to Support Students to Develop Skills that Promote Sustainability
Teaching Transversal Skills for Engineering Studens: A Practical Handbook of Activities with Tangibles; EPFL, 2024.How to Support Students Giving Each Other Constructive Feedback, Especially When It Is Difficult to Hear
Teaching Transversal Skills for Engineering Studens: A Practical Handbook of Activities with Tangibles; EPFL, 2024.How teachers can use the 3T PLAY trident framework to design an activity that develops transversal skills
Teaching Transversal Skills for Engineering Studens: A Practical Handbook of Activities with Tangibles; EPFL, 2024.The conceptual foundations of innate immunity: Taking stock 30 years later
Immunity. 2024-04-09. Vol. 57, num. 4, p. 613-631. DOI : 10.1016/j.immuni.2024.03.007.Radio-Activities: Architecture and Broadcasting in Cold War Berlin
Cambridge, MA; London: MIT Press, 2024.No Last One
Revue Matières. 2024. num. 18.All That is Solid
Transcalar Prospects in Climate Crisis; Zurich: Lars Müller, 2024.Field Guide to Northern Tree-related Microhabitats: Descriptions and size limits for their inventory in boreal and hemiboreal forests of Europe and North America
Swiss Federal Institute for Forest, Snow and Landscape Research WSL, Switzerland, 2024.Data Champions Lunch Talks – Green Bytes: Data-Driven Approaches to EPFL Sustainability
Data Champions Lunch Talks – Green Bytes: Data-Driven Approaches to EPFL Sustainability, EPFL, CM 1 221, April 18, 2024.Comparison of Three Viral Nucleic Acid Preamplification Pipelines for Sewage Viral Metagenomics
Food and Environmental Virology. 2024. DOI : 10.1007/s12560-024-09594-3.How to Support Students to Develop Skills that Promote Sustainability
Teaching Transversal Skills for Engineering Studens: A Practical Handbook of Activities with Tangibles; EPFL, 2024.How to Support Students Giving Each Other Constructive Feedback, Especially When It Is Difficult to Hear
Teaching Transversal Skills for Engineering Studens: A Practical Handbook of Activities with Tangibles; EPFL, 2024.How teachers can use the 3T PLAY trident framework to design an activity that develops transversal skills
Teaching Transversal Skills for Engineering Studens: A Practical Handbook of Activities with Tangibles; EPFL, 2024.The conceptual foundations of innate immunity: Taking stock 30 years later
Immunity. 2024-04-09. Vol. 57, num. 4, p. 613-631. DOI : 10.1016/j.immuni.2024.03.007.Radio-Activities: Architecture and Broadcasting in Cold War Berlin
Cambridge, MA; London: MIT Press, 2024.No Last One
Revue Matières. 2024. num. 18.All That is Solid
Transcalar Prospects in Climate Crisis; Zurich: Lars Müller, 2024.Field Guide to Northern Tree-related Microhabitats: Descriptions and size limits for their inventory in boreal and hemiboreal forests of Europe and North America
Swiss Federal Institute for Forest, Snow and Landscape Research WSL, Switzerland, 2024.Data Champions Lunch Talks – Green Bytes: Data-Driven Approaches to EPFL Sustainability
Data Champions Lunch Talks – Green Bytes: Data-Driven Approaches to EPFL Sustainability, EPFL, CM 1 221, April 18, 2024.Comparison of Three Viral Nucleic Acid Preamplification Pipelines for Sewage Viral Metagenomics
Food and Environmental Virology. 2024. DOI : 10.1007/s12560-024-09594-3.How to Support Students to Develop Skills that Promote Sustainability
Teaching Transversal Skills for Engineering Studens: A Practical Handbook of Activities with Tangibles; EPFL, 2024.How to Support Students Giving Each Other Constructive Feedback, Especially When It Is Difficult to Hear
Teaching Transversal Skills for Engineering Studens: A Practical Handbook of Activities with Tangibles; EPFL, 2024.How teachers can use the 3T PLAY trident framework to design an activity that develops transversal skills
Teaching Transversal Skills for Engineering Studens: A Practical Handbook of Activities with Tangibles; EPFL, 2024.The conceptual foundations of innate immunity: Taking stock 30 years later
Immunity. 2024-04-09. Vol. 57, num. 4, p. 613-631. DOI : 10.1016/j.immuni.2024.03.007.Radio-Activities: Architecture and Broadcasting in Cold War Berlin
Cambridge, MA; London: MIT Press, 2024.No Last One
Revue Matières. 2024. num. 18.All That is Solid
Transcalar Prospects in Climate Crisis; Zurich: Lars Müller, 2024.Contacts
Slobodan Ilic | [email] |