Endoscopes are widely used for examination and treatment of gastrointestinal diseases. For medical treatment, measurement of sizes of the tumors is important to determine treatment policies. Many of such measurements, In many cases, such measurements currently rely on visual estimation by humans. However, such estimation is often susceptible to human errors. If objective measurements of shapes of tumors can be performed using endoscope devices, it is a large step for solving such problems.

We have been developing a 3D endoscope system based on an active stereo approach. The system configurations are shown in the figure below. A micro pattern projector is inserted through the instrument channel of the endoscope, and images are captured using the endoscopic camera while the pattern illumination is projected onto the target surface. Then,the correspondences between the captured image and the projected pattern are estimated using image processing. 3D measurement by triangulation is performed using those correspondences.

Since the pattern projector is just inserted through the instrument channel of the endoscopes the projector is not fixed with respect to the endoscope camera. Thus, it is necessary to estimate the positional relationship between the endoscope camera and the pattern projector in advance. We developed a calibration method using a sphere(EMBC 2014), and an auto-calibration method where we can estimate the relationship without using calibration objects such as sphere(EMBC 2015).

For 3D endoscope systems, the measurement targets are living tissues. When a pattern is projected onto a living tissue, the projected patterns often become blurred due to subsurface scattering of the tissues. Also, in endoscopic imaging environments, noises and disturbances are more significant than ordinary camera environments. For this reason, pattern projection method that is more robust against blur and noise is required. To that end, we have developed a grid-like pattern where its grid points are modulated by gaps between grid edges (gap-coded grid pattern). Furthermore, we have developed a pattern projector that can sharply project such patterns using an optical element called DOE (EMBC 2016, ECCV 2016, PSIVT 2017).

The figures below show an experimental example where a phantom model of the stomach is sequentially captured using the system and the results of frame-by-frame 3D shapes are integrated into a unified shape. The left figure shows the appearance of the model, the area captured sequentially (blue lines), and an area restored from a single frame (red line). The right figure shows the integrated shape. It is shown that the small details of the phantom model shape are captured within the final shape. (CARE 2018).

• Furukawa, R., Aoyama, M., Hiura, S., Aoki, H., Kominami, Y., Sanomura, Y., Yoshida, S., Tanaka, S., Sagawa, R., and Kawasaki, H., Calibration of a 3d endoscopic system based on active stereo method for shape measurement of biological tissues and specimen Engineering in Medicine and Biology Conference, pp.4991–4994, 2014
• Furukawa, R., Masutani, R., Miyazaki, D., Baba, M., Hiura, S., Visentini-Scarzanella, M., Morinaga, H., Kawasaki, H., and Sagawa, R., 2-dof auto-calibration for a 3d endoscope system based on active stereo Engineering in Medicine and Biology Conference, pp.7937–7941, 2015.
• Furukawa, R., Sanomura, Y., Tanaka, S., Yoshida, S., Sagawa, R., Visentini-Scarzanella, M., and Kawasaki, H., 3d endoscope system using doe projector Engineering in Medicine and Biology Conference, pp.2091–2094, 2016.
• Furukawa, R., Morinaga, H., Sanomura, Y., Tanaka, S., Yoshida, S., and Kawasaki, H., Shape acquisition and registration for 3d endoscope based on grid pattern projection European Conference on Computer Vision, pp.399–415, 2016.
• Furukawa, R., Naito, M., Miyazaki, D., Baba, M., Hiura, S., Sanomura, Y., Tanaka, S., and Kawasaki, H., Auto-calibration Method for Active 3D Endoscope System Using Silhouette of Pattern Projector Image and Video Technology - 8th Pacific-Rim Symposiu (PSIVT), pp.222–236, 2017.
• Furukawa, R., Mizomori, M., Hiura, S., Oka, S., Tanaka, S., and Kawasaki, H., Wide-Area Shape Reconstruction by 3D Endoscopic System Based on CNN Decoding, Shape Registration and Fusion Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis, pp.139–150, 2018.