Capture image error when using multiple OV7251 parallel

Hi forum

I have a really strange problem. We have a setup, where we use three OV7251 in parallel, each with its own UC-628 + UC-425 USB shield. All three cameras are together on a USB hub and connected to a laptop, where I modified the C++ Arducam Streaming_Demo on Windows 10. I just open everything three times instead of once.
The problem is this: when I connect the cameras and run the program, there is a possibility that sometimes (say 50% of all cases) one camera reports either the USB_CAMERA_DATA_LEN_ERROR or the USB_CAMERA_USB_TIMEOUT_ERROR and does not display any picture. The configuration and start capturing always works, the errors occur only on the ArduCam_captureImage function of the captureImage_thread. A picture for the “faulty” cam is never displayed as there is always an error. If every camera displays a picture, the program runs very robust for a very long time without any problems and frame drops. The setup is not touched in between restarting the program. The picture is very clear when it is working and the data looks very good (it is used for machine learning).
I see also the same problem when I use three times the GUI USBTest program as described in the Hardware Trigger with External Signal (Featuring Multiple OV9281 Global Shutter Camera). We have a very similar setup.
We actually use a triggered camera configuration file but also with different config files without triggering this error occurs the same way.
This problem is extremely annoying, because we need to restart all the time our application until all three cameras work which takes some time in our setup.

What I have done so far:

• Used different config files for the UC-628 + UC-425
• tried a lot of delays in the cpp application between various setup and start calls
• tried to use a lot of global variables instead of local and checked that the handles are truly unique and not overwritten
• connected the cameras directly to the laptop instead of the USB hub
• Used the GUI as mentioned above, sometimes start a camera fully before configure and starting another or configuring sequentially and starting them sequentially
• A coworker did a separate implementation of the adapted Arducam Streaming_demo program and encountered the very same problem on a different laptop
• Use different USB hubs

The error always persisted and it is not possible to say that one action decreased the possibility of a startup error.

My specs:

• camera: OV7251
• USB Shield: UC-628 + UC-425
• OS: Windows 10 Education 2004
• Visual Studio 16.5.0
• Arducam Git Checkout: ArduCAM_USB_Camera_Shield 68b42fe

Is it possible that you try this out on your side? Connect three OV7251 to the computer and use the GUI to open them simultaneously? Or better: use the cpp streaming_demo to do so? At that stage, I’m almost certain that the problem lies on the driver side, as we already used more than a week of work and multiple people to solve this issue.

Thank you very much for a quick reply
Best Regards
Matthias

Hi @matthi

Very strange phenomenon, can you provide your modified program? I need to perform some tests.

I modified a 30fps configuration, you can try it, LEN_ERROR may be caused by too much bandwidth.

 

Hi @wong

Thank you for your reply
I will compile my test program and get back to you asap.

Thanks
Matthi

Hi @wong

First of all, thanks again for the config file. By using this file (i guess with 30fps instead of 100fps) the program does now almost always work with three cameras. “Almost”, because it sometimes does not work at the first time, when I connect the cameras. I now get a USB_CAMERA_USB_CREATE_ERROR for the third camera I try to open.

As I said, it is always the first time. Afterwards the program performs very robust and always starts up correctly.

I have attached my modified ArduCam_Demo.cpp file.
The main code is after line 483. The config files for the three cameras will be given as a parameter (a config file name for each camera). I also need to call ArduCam_scan a lot of times, because after the first time it takes up to 1 minute until the ArduCam_scan returns the correct IDs of the shields. Before the ID’s are empty. Is this a normal behavior?
As you can see, I added a lot of delays in between functions to make it robuster.

Do you experience the same problem?

In summary, my initial problem was solved by a degree, but during cold-start the problem remains.

Thanks
Matthias

int main(int argc, char **argv)
{
#ifdef linux
static struct termios oldt, newt;
tcgetattr(STDIN_FILENO, &oldt);
newt = oldt;
newt.c_lflag &= ~(ICANON);
tcsetattr(STDIN_FILENO, TCSANOW, &newt);
#endif

const char * config_file_name_A;
const char* config_file_name_B;
const char* config_file_name_C;
if (argc > 1) {
config_file_name_A = argv[1];
config_file_name_B = argv[2];
config_file_name_C = argv[3];
}
else {
showHelp();
return 0;
}

// find all cameras
ArduCamIndexinfo pUsbIdxArray[16];
int camera_num = 0;
Uint8 cam_A_USB_idx = 0xFF;
Uint8 cam_B_USB_idx = 0xFF;
Uint8 cam_C_USB_idx = 0xFF;

// scan every camera until all three serial numbers are found
while ( (cam_A_USB_idx | cam_B_USB_idx | cam_C_USB_idx) == 0xFF )
{
camera_num = ArduCam_scan(pUsbIdxArray);

Sleep(2000);
printf(“Found %d cameras\n”, camera_num);
char serial[16];
unsigned char* u8TmpData;
for (int i = 0; i < camera_num; i++) {
u8TmpData = pUsbIdxArray[i].u8SerialNum;
sprintf(serial, “%c%c%c%c-%c%c%c%c-%c%c%c%c”,
u8TmpData[0], u8TmpData[1], u8TmpData[2], u8TmpData[3],
u8TmpData[4], u8TmpData[5], u8TmpData[6], u8TmpData[7],
u8TmpData[8], u8TmpData[9], u8TmpData[10], u8TmpData[11]);
//printf(“index:%4d\tSerial:%s\n”, pUsbIdxArray[i].u8UsbIndex, serial);
}
Sleep(1000);

cam_A_USB_idx = 0xFF;
cam_B_USB_idx = 0xFF;
cam_C_USB_idx = 0xFF;
// search for each camera by serial number
for (int idx = 0; idx < camera_num; idx++)
{
std::string temp_string(reinterpret_cast<char const*>(pUsbIdxArray[idx].u8SerialNum));
std::cout << temp_string << “\n”;
if (strcmp(temp_string.c_str(), “AU3S20280010”) == 0)
{
cam_A_USB_idx = idx;
}
if (strcmp(temp_string.c_str(), “AU3S20280003”) == 0)
{
cam_B_USB_idx = idx;
}
if (strcmp(temp_string.c_str(), “AU3S20280017”) == 0)
{
cam_C_USB_idx = idx;
}
}
}

printf(“camera A USB index: %d\n”, cam_A_USB_idx);
printf(“camera B USB index: %d\n”, cam_B_USB_idx);
printf(“camera C USB index: %d\n”, cam_C_USB_idx);

bool a_init = false;
bool b_init = false;
bool c_init = false;

Sleep(2000);

printf(“init camera A\n”);
camera_initFromFile(config_file_name_A, cameraHandle_A, cameraCfg, cam_A_USB_idx);
Sleep(2000);

printf(“init camera B\n”);
camera_initFromFile(config_file_name_B, cameraHandle_B, cameraCfg, cam_B_USB_idx);
Sleep(2000);

printf(“init camera C\n”);
camera_initFromFile(config_file_name_C, cameraHandle_C, cameraCfg, cam_C_USB_idx);
Sleep(2000);

Sleep(10);
//ArduCam_writeSensorReg(cameraHandle_A, 0x3823, 0x20);
//ArduCam_writeSensorReg(cameraHandle_B, 0x3823, 0x20);
printf(“enabled external trigger\n”);

// read out chip id of cameras:
unsigned long low_value = 0;
unsigned long high_value = 0;
ArduCam_readSensorReg(cameraHandle_A, 0x300A, &high_value);
ArduCam_readSensorReg(cameraHandle_A, 0x300B, &low_value);
printf(“Camera A has chip ID: 0x%x%x \n”, high_value, low_value);

low_value = 0;
high_value = 0;
ArduCam_readSensorReg(cameraHandle_B, 0x300A, &high_value);
ArduCam_readSensorReg(cameraHandle_B, 0x300B, &low_value);
printf(“Camera B has chip ID: 0x%x%x \n”, high_value, low_value);

low_value = 0;
high_value = 0;
ArduCam_readSensorReg(cameraHandle_C, 0x300A, &high_value);
ArduCam_readSensorReg(cameraHandle_C, 0x300B, &low_value);
printf(“Camera C has chip ID: 0x%x%x \n”, high_value, low_value);

Sleep(2000);

std::thread captureThread_A(captureImage_thread, cameraHandle_A, false);
std::thread readThread_A(readImage_thread, cameraHandle_A, “Camera_A”, 0);
Sleep(5000);
std::thread captureThread_B(captureImage_thread, cameraHandle_B, false);
std::thread readThread_B(readImage_thread, cameraHandle_B, “Camera_B”, 1);
Sleep(5000);
std::thread captureThread_C(captureImage_thread, cameraHandle_C, true);
std::thread readThread_C(readImage_thread, cameraHandle_C, “Camera_C”, 2);

I somehow can’t attach files, so I posted my code (it’s not very readable now…)

Hi @matthi ,

I also need to call ArduCam_scan a lot of times, because after the first time it takes up to 1 minute until the ArduCam_scan returns the correct IDs of the shields. Before the ID’s are empty. Is this a normal behavior?

It takes a while to find the device after inserting the USB device for the first time.

 

For the code of multiple cameras, you can refer to this: https://github.com/glddiv/Arducam_Opencv_Demo/blob/master/Arducam_Opencv_Demo/src/ArduCam_Demo.cpp

Hi @wong

Thank you for your answer. I use the cpp streaming demo for the ArduCam USB Shield which captures constantly in one thread and reading the image constantly in another thread. So I’m not sure, if the demo code you provided also applies to this.
The Arducam example for the USB Shield camera does state it this way. Or do you suppose, I need to do it like the ArduCam_Demo.cpp you provided? Those functions are not mentioned in the software SDK and API for C++.

Cheers
Matthias

Hi @matthi ,

This example program uses the same principle as streamer_demo, but wraps it in C++ and provides a more friendly interface.

https://github.com/glddiv/Arducam_Opencv_Demo/blob/master/Arducam_Opencv_Demo/src/Arducam.cpp

Hi @wong

Thank you for the code. The example code does exactly what I do in my camera application: open the cameras, each with its separate handle, start a thread for each capture as well as read and display the picture. The example uses two cameras, I use three.

What is the maximum amount of cameras in parallel you have ever read out with this method? Did you ever encountered a limit? If yes, what was the error / behavior you got?
I actually beginning to suspect that there is something not correct in the driver for the cameras, when there are multiple instances of it.

Cheers
Matthias

Hi @matthi ,

The biggest problem I have encountered is the USB bandwidth problem, because our camera uses the RAW format, so its data volume is relatively large.Once the connected cameras increase and exceed the USB bandwidth limit, it will not be able to receive pictures normally.

Hi @wong

Ok, that’s strange that you never encountered such a problem. Could you tell me, how the problem shows itself when the bandwidth is not enough? are there sporadically frame drops, or did you receive a specific error? was there an error for/frame drop for some cameras only, or for all together? how was this issue behaving?

Maybe this is what we encountered…

Hi @matthi ,

If it is a bandwidth problem, the USB_CAMERA_DATA_LEN_ERROR error will appear.

So your current situation should be a bandwidth problem, because you use a low frame rate configuration and no longer have the USB_CAMERA_DATA_LEN_ERROR error.