Crowd counting has achieved significant progress by training regressors to predict head positions. In heavily crowded scenarios, however, regressors are challenged by uncontrollable annotation variance, which causes density map bias and context information inaccuracy. In this study, we propose mutual prompt learning (mPrompt), which leverages a regressor and a segmenter as guidance for each other, alleviating the bias and inaccuracy caused by annotation variance while distinguishing foreground from background. In specific, mPrompt leverages point annotations to tune the segmenter and predict pseudo head masks in a way of point prompt learning. It then uses the predicted segmentation masks, which serve as spatial constraint, to rectify biased point annotations as context prompt learning. From a perspective of mutual information maximization, mPrompt mitigates the impact of annotation variance while improving the model accuracy. Experiments show that mPrompt respectively reduces the Mean Average Error (MAE) significantly on four popular datasets, demonstrating the superiority of mutual prompt learning.Code is enclosed in the supplementary material.