Abstract: Face recognition is central to many authentication, security, and personalized applications. Yet, it suffers from significant privacy risks, particularly concerning unauthorized access to sensitive biometric data. This paper presents CryptoFace, the first end-to-end encrypted face recognition system that ensures secure processing of facial data from acquisition through storage and matching without exposing raw facial images or features at any stage. CryptoFace leverages fully homomorphic encryption (FHE) for encrypted feature extraction, feature matching, and comparison while maintaining high face recognition performance. It employs a mixture of shallow patch convolutional networks (PCNNs), which can be evaluated in parallel with FHE and lead to much faster inference. It is scalable to high-resolution face data without sacrificing inference speed and optimizes homomorphic neural architecture by minimizing the multiplicative depth. We evaluate the performance and computational efficiency of CryptoFace on multiple encrypted benchmark face datasets. CryptoFace exhibits a significant acceleration of $7.2\times$ ($9,845$s to $1,364$s per image on CPU) compared to state-of-the-art FHE-based neural networks adapted for face recognition. CryptoFace will facilitate the deployment of secure biometric authentication systems in applications requiring strict privacy and security guarantees.