Plug-and-play (PnP) and deep-unrolling solvers have become the de-facto standard tools for inverse problems in single-pixel imaging (SPI). The former, a class of iterative algorithms where regularization is implicitly performed by an off-the-shelf deep Gaussian denoiser, behaves well on interpretation and generalization, but poorly on accuracy and running time. By contrast, the latter results in better accuracy with faster inference, but cannot generalize well on varying degradations and cannot be interpreted as the proximal operator of any function. In this paper, we tackle the challenges of integrating the merits of both kinds of solvers into one. To this end, we propose a proximal optimization trajectory loss and an image restoration network to develop learned proximal restorers (LPRs) as PnP priors. LPRs approximate the proximal operator of an explicit restoration regularization and guarantees the fast convergence of PnP-HQS (half-qauadratic splitting algorithm) and PnP-ADMM (alternating direction method of multipliers). Besides, extensive experiments show that our algorithms not only achieve state-of-the-art accuracy in almost real time but also generalize well on varying degradations caused by changing imaging settings (resolution and compression ratio). The source code will be released soon.