Abstract: Point clouds have gained growing interest in gait recognition. However, current methods, which typically convert point clouds into 3D voxels, often fail to extract essential gait-specific features. In this paper, we explore gait recognition within 3D point clouds from the perspectives of architectural designs and gait representation modeling. We indicate the significance of local and body size features in 3D gait recognition and introduce S$^3$GaitNet, a novel framework combining advanced local representation learning techniques with a novel size-aware learning mechanism. Specifically, S$^3$GaitNet utilizes Set Abstraction (SA) layer and Pyramid Point Pooling (P$^3$) layer for learning locally fine-grained gait representations from 3D point clouds directly. Both the SA and P$^3$ layers can be further enhanced with size-aware learning to make the model aware of the actual size of the subjects. In the end, S$^3$GaitNet not only outperforms current state-of-the-art methods, but it also consistently demonstrates robust performance and great generalizability on two benchmarks. Our extensive experiments validate the effectiveness of size and local features in gait recognition. The code will be publicly available.