In this paper, we speed up robust estimation from dense two-view correspondences. Previous work has shown that dense matchers can significantly improve both the accuracy and robustness of two-view relative pose estimation. However, the large number of matches comes with a significantly increased runtime during robust estimation. To avoid this, we propose an efficient match summarization scheme which provides comparable accuracy to using the full set of dense matches, while having 10-100x faster runtime. We validate our approach on standard benchmark datasets together with multiple state-of-the-art dense matchers.