Generalized Category Discovery (GCD) is an intriguing open-world problem that has garnered increasing attention within the research community. Given a dataset that includes both labelled and unlabelled images, GCD aims to categorize all images in the unlabelled subset, regardless of whether they belong to known or unknown classes. In GCD, the common practice typically involves applying a sphere projection operator at the end of the self-supervised pretrained backbone. This approach leads to subsequent operations, being conducted within Euclidean or spherical geometry. However, both of these geometries have been shown to be suboptimal for encoding samples that possess hierarchical structures. In contrast, hyperbolic space exhibits a unique property where its volume grows exponentially relative to the radius, making it particularly suitable for embedding hierarchical data. In this paper, we are the first to investigate category discovery within hyperbolic space. We propose a simple yet effective framework for learning hierarchy-aware representations and classifiers for GCD through the lens of hyperbolic geometry. Specifically, we initiate this process by transforming a self-supervised backbone pretrained in Euclidean space into hyperbolic space via exponential mapping, then performing all representation and classifier optimization in the hyperbolic domain. We evaluate our proposed framework on widely used parametric and non-parametric GCD baselines, as well as the previous state-of-the-art (SOTA) method, achieving significant improvements and establishing a new SOTA.