FaceLift: Semi-supervised 3D Facial Landmark Localization

3D facial landmark localization has proven to be of particular use for applications, such as face tracking, 3D face modeling and image-based 3D face reconstruction. In the supervised learning case, such methods usually rely on 3D landmark datasets derived from 3DMM-based registration that often lack spatial definition alignment, as compared with that chosen by hand-labeled human consensus, e.g. how are eyebrow landmarks defined? This creates a gap between landmark datasets generated via high-quality 2D human labels and 3DMMs, and it ultimately limits their effectiveness. To address this issue, we introduce a novel semi-supervised learning approach that learns 3D landmarks by directly lifting (visible) hand-labeled 2D landmarks and ensures better definition alignment, without the need for 3D landmark datasets. To lift 2D landmarks to 3D, we leverage 3D-aware GANs for better multi-view consistency learning, and in-the-wild multi-frame videos for robust cross-generalization. Furthermore, we contribute a novel 3D facial landmark evaluation scheme to handle comparison across various 3D landmark definitions by exploiting recent advancements in photogrammetric face mesh tracking. Empirical experiments demonstrate that our method not only achieves better definition alignment between 2D-3D landmarks but also outperforms other supervised learning 3D landmark localization methods on ground-truth 3D datasets.