Table tennis is beyond a game of physical skill. Competitive players overcome the challenging high-speed and highly dynamic environment by anticipating their opponent’s intent – buying themselves the necessary time to respond. In this work, we take one step towards designing such an agent. Previous works have developed systems capable of real-time table tennis gameplay, though they often do not leverage anticipation; of the works that forecast opponent actions, their approaches are limited by dataset size and variety. Our paper contributes (1) a scalable system for reconstructing monocular video of table tennis matches in 3D, (2) a real-time Transformer-based model for anticipating opponent actions, and (3) an uncertainty-aware anticipatory controller for a KUKA robot arm. We demonstrate in simulation that our policy improves the ball return rate on high-speed hits from 44.3% to 61.1% as compared to a baseline non-anticipatory policy.