Abstract: We propose a novel 3D gaze estimation approach that learns spatial relationships between the subject, scene, and objects and outputs 3D gaze direction. Our method targets unconstrained settings, including cases where close-up views of the subject’s eyes are unavailable, such as when the subject is distant or facing away. Previous methods rely on either 2D appearance alone or limited spatial cues by using depth maps in non-learnable post-processing. Estimating 3D gaze direction from 2D observations in these settings is challenging; besides variations in subject, scene, and gaze direction, different camera poses produce various 2D appearances and 3D gaze directions, even when targeting the same 3D scene. To address this issue, we propose gaze-aware 3D context encoding. Our method represents the subject and scene as 3D poses and object positions as 3D context, learning spatial relationships in 3D space. Inspired by human vision, we geometrically align this context in a subject-centric space, significantly reducing the spatial complexity. Furthermore, we propose D$^3$ (distance-direction-decomposed) positional encoding to better capture the spatial relationship between 3D context and gaze direction in direction and distance space. Experiments show substantial improvement, reducing mean angle error of leading baselines by 13\%–37\% on benchmark datasets in single-view settings.