Skip to yearly menu bar Skip to main content


MS-MANO: Enabling Hand Pose Tracking with Biomechanical Constraints

Pengfei Xie · Wenqiang Xu · Tutian Tang · Zhenjun Yu · Cewu Lu

Arch 4A-E Poster #213
[ ]
Wed 19 Jun 10:30 a.m. PDT — noon PDT


This work proposes a novel learning framework for visual hand dynamics analysis that takes into account the physiological aspects of hand motion. The existing models, which are simplified joint-actuated systems, often produce unnatural motions. To address this, we integrate a musculoskeletal system with a learnable parametric hand model, MANO, to create a new model, MS-MANO. This model emulates the dynamics of muscles and tendons to drive the skeletal system, imposing physiologically realistic constraints on the resulting torque trajectories. We further propose a simulation-in-the-loop pose refinement framework, BioPR, that refines the initial estimated pose through a multi-layer perceptron (MLP) network. Our evaluation of the accuracy of MS-MANO and the efficacy of the BioPR is conducted in two separate parts. The accuracy of MS-MANO is compared with MyoSuite, while the efficacy of BioPR is benchmarked against two large-scale public datasets and two recent state-of-the-art methods. The results demonstrate that our approach consistently improves the baseline methods both quantitatively and qualitatively. Code and models will be made publicly available.

Live content is unavailable. Log in and register to view live content