This paper presents FloVD, a novel optical-flow-based video diffusion model for camera-controllable video generation. FloVD leverages optical flow maps to represent motions of the camera and moving objects. Since optical flow can be directly estimated from videos, this approach allows for the use of arbitrary training videos without ground-truth camera parameters. To synthesize natural object motion while supporting detailed camera control, we divide the camera-controllable video generation task into two sub-problems: object motion synthesis and flow-conditioned video synthesis. Specifically, we introduce an object motion synthesis model along with 3D-structure-based warping to generate foreground and background motions, which are fed into the flow-conditioned video diffusion model as conditional input, enabling camera-controlled video synthesis. Extensive experiments demonstrate the superiority of our method over previous approaches in terms of accurate camera control and natural object motion synthesis.