Multi-agent trajectory prediction is essential in autonomous driving, risk avoidance, and traffic flow control. However, the \textbf{heterogeneous traffic density} on \textbf{interactions}, which caused by physical laws, social norms and so on, is often overlooked in existing methods. When the density varies, the number of agents involved in interactions and the corresponding interaction probability change dynamically. To tackle this issue, we propose a new method, called \underline{\textbf{D}}ensity-\underline{\textbf{A}}daptive Model based on \underline{\textbf{M}}otif \underline{\textbf{M}}atrix for Multi-Agent Trajectory Prediction (DAMM), to gain insights into multi-agent systems. Here we leverage the \textbf{motif matrix} to represent dynamic connectivity in a higher-order pattern, and distill the interaction information from the perspectives of the spatial and the temporal dimensions. Specifically, in spatial dimension, we utilize multi-scale feature fusion to adaptively select the optimal range of neighbors participating in interactions for each time slot. In temporal dimension, we extract the temporal interaction features and adapt a pyramidal pooling layer to generate the interaction probability for each agent. Experimental results demonstrate that our approach surpasses state-of-the-art methods on autonomous driving dataset.