Vision-based 3D occupancy prediction is an emerging scene perception task aiming at predicting the occupancy in the voxelized representation of the 3D scene, which provides fundamental scene perception information for subsequent decisions in autonomous driving systems. Existing methods focus on depth map estimation to construct 3D scene representations from 2D input images, but fail to exploit the semantic and geometry consistency on the pixel/voxel-level, leading to missing occupancy and coarse boundary in prediction results. To address this issue, we propose a semantic-geometry consistent 3D occupancy prediction (SGC-Occ) framework to generate more realistic 3D occupancy prediction results, which follows the popular Bird's-Eye-View (BEV) paradigm, and centers at promoting the semantic consistency during view transformation and geometry consistency in voxelized BEV grids. Specifically, we first propose a recurrent semantic consistency enhancement (RSCE) module, which maintains a dynamic updating occupancy bank to bridge the semantic gap between 2D image features and BEV features. Then we introduce a coarse-to-fine geometry refinement (CFGR) module to resolve missing occupancy and coarse boundary by aggregating local geometric correlations, generating more precise and realistic occupancy predictions. Extensive experiments and analyses are conducted to demonstrate the effectiveness of our SGC-Occ framework.