Face anti-spoofing (FAS) is critical to applications which heavily rely on the authenticity of detected faces. Many FAS methods have focused on learning discriminative features from both live and spoof training data. However, since not every possible attack type is available in the training stage, these FAS methods usually fail to detect unseen attacks in the inference stage. In comparison, One-Class Classification, where the training data are from only a single positive class (e.g., live faces), enables a more practical setting for FAS to detect whether a test face image belongs to the live class or not. In this paper, we address the one-class FAS detection problem and propose a novel One-Class Spoof Cue Map estimation Network (OC-SCMNet) to detect various spoof attacks by learning exclusively from the live class. We construct OC-SCMNet with one latent feature extractor, one Spoof Cue Map (SCM) estimator, and one SCM-guided generator. Our first goal is to learn to extract latent spoof features from live images so that their estimated SCMs should have zero responses. To avoid trapping to a trivial solution, we devise a novel SCM-guided feature learning by combining many SCMs as pseudo ground-truths to guide a conditional generator to generate non-trivial latent spoof features for spoof data. Our second goal is to approximately simulate the potential out-of-distribution spoof attacks under one-class constraint. To this end, we propose using a memory bank to dynamically preserve a set of sufficiently “independent” latent spoof features to encourage the generator to probe the latent spoof feature space. Extensive experiments conducted on eight FAS benchmark datasets demonstrate that the proposed OC-SCMNet not only outperforms previous one-class FAS methods but also achieves comparable performances to state-of-the-art two-class FAS method.