We propose PyTorchGeoNodes, a differentiable module for 3D object reconstruction from images using interpretable shape programs. Unlike traditional CAD model retrieval, shape programs allow semantic reasoning, editing, and a low memory footprint. Despite their potential, shape programs for 3D scene understanding have been largely overlooked. Our key contribution is enabling gradient-based optimization by translating shape programs, like those in Blender, into efficient PyTorch code. Additionally, we show that a combination of PyTorchGeoNodes with Genetic Algorithms is a method of choice to optimize both discrete and continuous shape program parameters for 3D reconstruction, and can be further integrated with other reconstruction algorithms such as Gaussian Splats. Our experiments on the ScanNet dataset show that our method achieves accurate reconstructions.