Virtual view specification and synthesis in free viewpoint television application

In conventional TV, users can get only a single view of a real 3D world. The view is determined not by a user but by the camera position during acquisition. Free viewpoint television (FTV) is a new concept which aims at giving viewers the flexibility to select a novel viewpoint by employing multiple video streams as the input. Current proposed solutions for FTV include those based on ray-space sampling and interpolation. Most existing work makes the assumption that the cameras are calibrated. Moreover, virtual view specification is done in ways that would not be practical for actual use (e.g., some methods specify the virtual view by a rotation matrix and a translation vector). This paper proposes a framework for FTV based on image-based rendering, complete with key algorithms for color-based segmentation and correspondence, and multiple-image-based virtual view synthesis from uncalibrated cameras. Moreover, we propose an intuitive approach to specifying the virtual viewpoint, based on any two views chosen by the user. This makes the specification of the virtual view very intuitive and thus practical for the FTV application. Experiments with real video streams were performed to validate the proposed approaches.