GarNet: A Two-stream Network for Fast and Accurate 3D Cloth Draping

While Physics-Based Simulation (PBS) can highly accurately drape a 3D garment model on a 3D body, it remains too costly for real-time applications, such as virtual try-on. By contrast, inference in a deep network, that is, a single forward pass, is typically quite fast.

In this collaborative project with Fision AG, we leverage this property and introduce a novel architecture to fit a 3D garment template to a 3D body model. Specifically, we build upon the recent progress in 3D point-cloud processing with deep networks to extract garment features at varying levels of detail, including point-wise, patch-wise and global features. We then fuse these features with those extracted in parallel from the 3D body, so as to model the cloth-body interactions. The resulting two-stream architecture is trained with a loss function inspired by physics-based modeling, and delivers realistic garment shapes whose 3D points are, on average, less than 1.5cm away from those of a PBS method, while running 40 times faster.

Method

Our two stream network

Our model takes the target body and the template garment, as input. It consists of a Garment stream and a Body stream. The Body one uses an architecture similar to PointNet to extract local and global body features. The Garment stream exploits the global body features to compute point-wise, patch-wise and global body features for the template garment mesh. These features, along with the global body ones, are then fed to a fusion subnetwork to predict the shape of the fitted garment. To further increase the fitting accuracy, we introduce an auxiliary stream that computes separate local body features for each garment vertex.

We incorporate appropriate loss terms for physical constraints (e.g. skin-cloth penetration, bending) in the objective function we minimize during training. Hence, at test time, we avoid the need for extra post-processing steps that PBS tools might require.

Result videos

Prediction Results

The following videos show our results and the ground truth side-by-side.

Female T-Shirt

Male T-Shirt

Female Jeans

Male Jeans

Architecture Comparison

The following video shows the GarNet-Naive, GarNet-Local and ground truth side-by-side.

Ablation Study

The following video depicts our results when turning on some terms in the loss function during training.

Links

Main paper (arXiv)

Supplementary Material

Fision AG

Citation

Please cite:

@article{gundogdu18garnet,
title={Garnet: A Two-stream Network for Fast and Accurate 3D Cloth Draping},
author={Gundogdu, Erhan and Constantin, Victor and Seifoddini, Amrollah and Dang, Minh and Salzmann, Mathieu and Fua, Pascal},
journal={arXiv preprint arXiv:1811.10983},
year={2018}
}