We present a method for constructing anatomical atlases on the fly. An atlas is an image that represents the prototypical structure of a collection of images. Among other uses, atlases play a key role in studies of anatomical variability across populations. Existing atlas construction methods are computationally prohibitive, requiring days to weeks of computation. Consequently, many scientific studies are forced to use suboptimal atlases constructed for different population groups, negatively impacting downstream analyses. In this work, we present MultiMorph, a model that rapidly produces 3D anatomical atlases for any set of brain MRI images. MultiMorph enables medical researchers with no machine learning background to rapidly construct high-quality population-specific atlases in a single forward network pass, without requiring any fine tuning or optimization. MultiMorph is based on a linear group-interaction layer that aggregates and shares features within the group of input images. We demonstrate that MultiMorph outperforms state-of-the-art optimization-based and machine-learning-based atlas construction methods in both small and large population settings. It generates better atlases with a 100-fold reduction in computational time. Further, we demonstrate generalization to new imaging modalities and population groups at test-time.