Large Language Models (LLMs) have achieved significant success recently, leading to growing enthusiasm to expand their generative capabilities from general text to specialized domains. This paper discusses generating parametric sequences of computer-aided design (CAD) models via LLMs. This serves as an entry point for 3D shape generation by LLMs, as CAD model parameters directly map to shapes in 3D space. This task is non-trivial despite LLMs' strong generative abilities, as they have neither encountered parametric sequences during pretraining nor can they directly perceive 3D shapes. Therefore, we propose CAD-Llama, a paradigm empowering pretrained LLMs for parametric 3D CAD model generation. Specifically, we introduce a code-like format to unify parametric 3D CAD command sequences and a hierarchical annotation pipeline to convert intricate parametric 3D CAD shapes into Python-like pseudo-code, an LLM-friendly text format. Additionally, we propose adaptive pre-training on Structured Parametric CAD Code (SPCC) and fine-tuning with CAD-related instructions to imbue LLMs with spatial information conveyed in parametric sequences. Experimental results show that our framework significantly outperforms previous autoregressive methods and prevailing LLM baselines.