ReViSE : Proposal of Framework which Enables Seamless and Flexible Integration of Real and Virtual Objects for Video See-Through MR

Recent advancements in deep learning technol- ogy have significantly improved the performance of instance segmentation to a practical level. This technology enables the detection, segmentation, and extraction of object regions, offering substantial potential for applications in mixed reality (MR). While most research has focused on detection and segmentation, the application of extraction in realizing MR has received limited attention. In this paper, we propose a framework called ReViSE (Real and Virtual Seamless Editor), which integrates instance segmentation with virtual reality (VR) technology to deliver a wide range of MR experiences. This framework generates diverse MR visuals by applying instance segmentation on original images captured by a camera, and then replacing specified arbitrary object regions with virtual objects. Then, the MR visuals are presented through a head-mounted display (HMD) to provide users with a highly immersive visual experience. Evaluation experiments with a basic implementation show a processing time of 52.67 ms/frame and a display performance of 32.73 fps. The framework has also demonstrated its ability to accurately extract target objects and deliver a high-quality visual experience.