Depth Mapping on Arducam Stereo Camera HAT with OpenCV

Introduction

The Arducam Stereo Camera HAT is a revolutionary product that enables you to use two synchronized Pi camera modules on a single standard Raspberry Pi model. The stereo camera setup could be used in lots of applications that are hard to realize in the past. This documentation will show a demo of using the Arducam stereo camera HAT for depth mapping on a Raspberry Pi 3 with the help of our OV5647 Stereo Camera Board.

Credits to StereoPi, on which this tutorial and the codes used are based. Thanks to them for the work they have done.

Video Tutorial

We also provide a video demo of this tutorial, and you can check that below:

Hardware setup

Software Prerequisite

• Raspbian Buster (kernel 4.19.57-v7+), latest version recommended
• Python 3.5.3
• OpenCV 3.4.4 (pre-compiled, ‘pip’ from Python Wheels)
• StereoVision lib 1.0.3 (https://github.com/erget/StereoVision)

Installing Dependencies and Libraries

• Installing the latest Raspbian
• sudo pip install opencv-python
• sudo pip3 install stereovision
• sudo pip3 install matplotlib
• git clone https://github.com/ArduCAM/MIPI_Camera.git

Capture Images

The 1_test.py script will be used in this step.

Open the console and go to the examples folder:

cd MIPI_Camera/RPI/stereo_depth_demo

Console Command:

python 1_test.py

Select the camera mode after running this program, and then you will see a preview window. Press Q to stop the process and save the last image captured, which will be used in the next scripts for tuning the parameters.

It helps you make sure the hardware is working and captures the first picture. Camera modes and resolutions(width and height) will be recorded for later uses.

You can also see from the video above about how the first script works:

Step 2: Collect Images for Calibration

A perfect depth mapping sets a high standard for the camera setup, where two identical cameras have their optical, vertical and horizontal axis all in parallel. However, you can reach that perfection in real life, so calibration is required. The calibration software samples pictures of these two cameras on a specific object, analyzes them, and generates parameters for the correction. A chessboard is used in this demo.

The program starts with a 5-second countdown for positioning the object and then takes the picture. This process will be looped for 30 times by default, and repositioning is required between each countdown.

Console command:

python 2_chess_cycle.py

You can see from the video above to see more about this.

30 stereoscopic photos will then be saved in /scenes folder.

Step 3: Separate Captured Images

The 3_pairs_cut.py will separate the photos captured in the last step into left and right halves and save them in the /pairs folder. This separating process can run on-the-fly without saving, but this step could be helpful otherwise for subsequent uses. Image pairs could be saved from different capture series. You can process these images with your own codes or putting other stereoscopic images in this folder. You can also go through every stereo pair before the separation and sort the bad frames out. Console Command:

python 3_pairs_cut.py

You can see from the video above to see more about this.

Step 4: Calibration

4_calibration.py loads all previously saved pairs and figures out the correction matrices. It searches for the chessboard on the photo, and move on if the chessboard is not found. Therefore, the bad photos in the series won’t break the script. After the calculation is finished, it will rectify the last image and show the resulting “fixed” images. You can check the quality of the calibration in this step, which might take 1-2 minutes.

You can see from the video above to see more about this.

Console Command:

python 4_calibration.py 

Calibration script doing his job:

Step 5: Depth Map Tuning

The image in script 1 and the calibration results from step 4 will be loaded in 5_dm_tune.py. It shows a depth map with an interface for fine-tuning. Press the Load Settings button to use the default parameters. It’s recommended to take a photo with 3 objects from different distances to find the right settings before you tune the parameters. The closest will be red, and the furthest far away.

You can see from the video above to see more about this.

python 5_dm_tune.py 

Here is how this looks like:

Step 6: Real-Time Depth Map Using Video

6_dm_video.py uses the results from the above steps to generate a depth map in realtime.

You can see from the video above to see more about this.

Console Command:

python 6_dm_video.py

The result: