Classifiers are crucial to computer vision, yet their "black box" nature obscures the decision-making process, limiting the ability to trace the influence of individual features. Traditional interpretability methods, including GAN-based attribute editing, are constrained by domain and resource demands, often requiring extensive labeling and model-specific training. Text-to-image diffusion models, while promising for broader applications, lack precise semantics for classifier interpretation without extensive user input. We introduce DiffEx, a training-free framework that combines large language models (LLMs) and pre-trained diffusion models to improve classifier explainability. DiffEx leverages Vision-Language Models (VLMs) to build a comprehensive, hierarchical semantic corpus and applies a novel algorithm to rank impactful features, offering broad and fine-grained attributes that influence classifier scores. Our experiments show that DiffEx provides nuanced, interpretable insights across diverse domains, including medical diagnostics, making it versatile, scalable, and well-suited for understanding complex classifiers in critical applications.