We introduce Scenario Dreamer, a fully data-driven generative simulator for autonomous vehicle planning that generates both the initial traffic scene (comprising the lane graph and agent bounding boxes) and closed-loop agent behaviours. Existing methods for generating driving simulation environments encode the initial traffic scene as a rasterized image and, as such, require parameter-heavy networks that perform unnecessary computation due to many empty pixels in the rasterized scene. Moreover, we find that existing methods that employ rule-based agent behaviours lack diversity and realism. Scenario Dreamer instead comprises a novel vectorized latent diffusion model for initial traffic scene generation and a return-conditioned autoregressive Transformer for agent behaviour simulation. Scenario Dreamer supports scene extrapolation via diffusion inpainting, enabling the generation of unbounded simulation environments. We validate that Scenario Dreamer environments are more realistic than those generated from existing methods while requiring fewer parameters and lower generation latency. We confirm its practical utility by showing that RL-based planning agents are more challenged in Scenario Dreamer environments than traditional non-generative simulation environments, especially on long and adversarial driving environments. All code will be open-sourced.