ShowMak3r: Compositional TV Show Reconstruction

Reconstructing dynamic radiance fields from video clips is challenging, especially when entertainment videos like TV shows are given. Many challenges make the reconstruction difficult due to (1) actors occluding with each other and having diverse facial expressions, (2) cluttered stages, and (3) small baseline views or sudden shot changes. To address these issues, we present VirtualPCR, a comprehensive reconstruction pipeline that allows the editing of scenes like how video clips are made in a production control room (PCR). In VirtualPCR, a 3DLocator module locates recovered actors on the stage using depth prior, and it estimates unseen human poses via interpolation. The proposed ShotMatcher module tracks the actors under shot changes. VirtualPCR has a face-fitting branch that dynamically recover the actors’ expressions. Experiments on the Sitcoms3D dataset show that our pipeline can reassemble TV shows with new cameras at different time steps. We also show that VirtualPCR offers interesting applications, such as scene editing, synthetic shot-making, relocating actors, and human pose manipulation.