We present Hierarchical Motion Representation (HiMoR), a novel deformation representation for 3D Gaussian primitives that is capable of achieving high-quality monocular dynamic 3D reconstruction. HiMoR's foundation is the insight that motions in everyday scenes can be decomposed into coarser motion, which forms the basis for finer details. Using a tree structure, HiMoR's nodes represent different levels of motion detail, with shallower nodes depicting coarse motion for temporal smoothness and deeper nodes denoting finer motion. Additionally, our model uses a few shared motion bases to represent motions of different set of nodes, aligning with the assumption of motion being generally low-rank. This motion representation design provides Gaussians with a more rational deformation, maximizing the use of temporal relationships to tackle the challenging task of monocular dynamic 3D reconstruction. We also propose using a more reliable perceptual metric as a substitute, given that pixel-level metrics for evaluating monocular dynamic 3D reconstruction can sometimes fail to effectively reflect true reconstruction quality. Extensive experiments demonstrate our method's efficacy in achieving superior novel view synthesis from challenging monocular videos with complex motions.