M-Adaptor: Text-driven Whole-body Human Motion Generation

Text-driven whole-body human motion generation, which involves the creation of motion sequences based on textual descriptions, has attracted much attention in the communities of computer vision and artificial intelligence. It aims to extend text-driven motion generation tasks to accommodate complex whole-body human motions, encompassing facial expressions and hand gestures. Researchers have recently developed large-scale 3D expressive whole-body motion datasets enriched with semantic labels and pose descriptions. Nonetheless, there remains a considerable demand within the community for a straightforward and effective framework for generating and evaluating whole-body human motion based on textual descriptions. To address the above issues, we introduce M-Adaptor, a two-stage Low-Rank Adaptation (LoRA)-based generator for whole-body motion generation tasks, to improve the quality and diversity of body motions, facial expressions, and hand gestures. In particular, it first generates initial coarse-grained body motion tokens from textual prompts to enhance the stability of generated motions, then iterates fine-grained facial expressions with the LoRA-based adaptor to enhance motion expressiveness. Furthermore, we extend the existing state-of-the-art CLaM model to CLaM-H and CLaM-X for evaluation of SMPL-H and SMPL-X based motion generation. Extensive qualitative and quantitative evaluations demonstrate our framework's superior performance, with a significant R-Precision improvement for text-driven whole-body motion generation.