Maximum Framerate IMX298 + RPI4

[kylemcdonald]

What is the maximum framerate for the IMX298 on RPI4 at full resolution?

I wrote some short code in C to test. I set auto exposure off, exposure time to 1, resolution to 4656×3496 and I used IMAGE_ENCODING_RAW_BAYER. I measured 4.4fps (not including the first frame which takes longer).

Is it possible to increase the speed? What is the primary bottleneck?

I also tried IMAGE_ENCODING_I420, but the speed is similar (even though the bandwidth is larger). I also tried IMAGE_ENCODING_JPEG, but it causes a crash:

`
usrvcsm: [vcsm_malloc_cache]: [5925] [mmal_vc_port buffer]: ioctl mem-alloc FAILED [-1] (hdl: 0)
usrvcsm: [vcsm_vc_hdl_from_hdl]: [5925]: invalid device or handle!
usrvcsm: [vcsm_lock]: [5925]: invalid device or invalid handle!
mmal: mmal_vc_shm_alloc: could not allocate 48944896 bytes of shared memory (handle 0) – mem (nil), vc_hdl 00000000
mmal: mmal_vc_port_payload_alloc: vc.ril.image_encode:out:0(JPEG): failed to allocate 48944896 bytes of shared memory
mmal: mmal_pool_initialise_buffer_headers: failed to allocate payload 0/1
mmal: Failed to create isp output pool.
`

[bin]

Hi,

The main reason is the 2 channel bandwidth limitation of PI.

Please change the GPU memory to aviod the malloc error .

[kylemcdonald]

Thanks. I was able to get the JPEG setting working at full resolution. I can see that you answer this very basic question all the time, so thank you for your patience.

I thought that JPEG compression was performed by the IMX298. But when I set IMAGE_FORMAT fmt = {IMAGE_ENCODING_JPEG, 1}; and arducam_set_control(camera_instance, V4L2_CID_EXPOSURE, 1); so that the images are only 0.2MB, I can see that the framerate is still 4.4fps. So this must mean that the JPEG compression is performed by the arducam library.

When I modify the raw_callback or yuv_callback examples to use the full resolution I also get 4fps.

I read that CSI2 supports 1Gbit/s per lane, for 2Gbit/s total = 2,000,000,000 bits/s. A full frame of IMAGE_ENCODING_RAW is 20,407,296 bytes = 163,258,368 bits. 2,000,000,000 bits/s / 163,258,368 bits = 12.2 fps.

And on this page https://www.arducam.com/docs/cameras-for-raspberry-pi/mipi-camera-modules/ it says that the full resolution framerate is 10fps. This page says the full resolution framerate is 30fps https://www.arducam.com/docs/cameras-for-raspberry-pi/mipi-camera-modules/imx298-16mp-color-rolling-shutter-camera-for-raspberry-pi/

How can I achieve 10-12 fps?

[bin]

Hello,

Yes, you are right. Our configuration currently runs to 8 frames.  You can use arudcamstill to test. Due to different exposure times, there will be some delay. The mainly factor is the sensor’s configure.  Recently, I will try to optimize the sensor configuration

IF you use the encoder to get the jpeg image, there is a memory copy of the CPU fetch graph, and a lot of time will be lost.

 

[bin]

Hello,

I have optimized the imx298 configuration just now  and the max resolution can up to 12 fps.

 

[kylemcdonald]

Hi bin, I pulled the new code including the new library:

MD5(lib/libarducam_mipicamera.so)= ded62a83575b0a5f61ba13dadb597a0e

Then I did make clean; make arducamstill, but I still see 4fps:

I also checked with my own code and it still shows 4.4fps.

https://gist.github.com/kylemcdonald/eb5c72c19129039ee4f83472d4450298

[bin]

Hi, You should make install to install the new lib  at the MIPI_Camera path.

However, some people recently reported that the high frame rate is too fast, and our line will be a little disturbed. I still reduced the frame rate to 8 frames.

 

[kylemcdonald]

Of course, I’m surprised I forgot make install. Thanks.

I tried the latest code and now I get between 7-8 fps.

I also tried the previous version (after git reset --hard f337a4ddb7430e46e8db70f9cb01ba459c50fd72; make clean; make install) but I was unable to get 12 fps. Instead I get a Segmentation fault.

Would it be possible to expose the internal parameter that is allowing you to set the maximum framerate? I would like to be able to set this myself.

[bin]

Hello,

You can try to  reduce the value of the 0x0307 register to increase the clock.

 

[kylemcdonald]

Thanks! I will look into this. I haven’t set the registers on Raspberry Pi before so I will look for some examples first.

For anyone else coming across this thread, here are some more references that might be relevant:

https://www.raspberrypi.org/forums/viewtopic.php?t=254733

https://www.raspberrypi.org/forums/viewtopic.php?p=630909#p630909

[bin]

Great, thanks for your share.

Just use this API I attached for you to change the value of the register.

int arducam_write_sensor_reg(CAMERA_INSTANCE camera_instance, uint16_t address, uint16_t value);

[kylemcdonald]

Thanks! I used arducam_read_sensor_reg to get the current value, and I see it is 112.

Assuming this is directly connected to the clock speed, I tried to change it to 74 to get 12fps with arducam_write_sensor_reg(camera_instance, 0x0307, 74); but I don’t see any change.

I also tried changing it to a larger value like 224 but I did not see any change.

I also tried setting the register immediately after arducam_init_camera and immediately before arducam_capture but this showed no difference in results.

https://gist.github.com/kylemcdonald/eb5c72c19129039ee4f83472d4450298#file-capture_fast-cpp-L30

[bin]

Hello,

Oh I know, the sensor is very strange, you should change the clock before it’s streaming on register write 1.

I have help you add the mode. I have tested it on PI3 platform. Please try to download our new lib and make install. Then retry uisng mode 4 .

[kylemcdonald]

I pulled the new code (library) and ran make install then I replaced the call to arducam_set_resolution with arducam_set_mode(camera_instance, 4); immediately after calling arducam_init_camera.

From my tests, this new mode actually runs slower. When I modified raw_callback.c to test the speed: mode 3 runs around 7.8fps, and mode 4 runs at 4.8fps.

I checked the 0x0307 register and I noticed that it is 160 in mode 4 and 112 in mode 3.

I tried changing the register 0x0307 in mode 4, and it did not have any effect.

[bin]

Hello,

Strange, Have you tested it on PI3 platform?

 

[bin]

Hello,

Strange, Have you tested it on PI3 platform?

 

[kylemcdonald]

I don’t have a PI3 for testing, and need a PI4 for my application. If there are any updates to bring the PI4 up to the 12fps frame rate please let me know!

[bin]

Hi,

I am sure I have configure the sensor to 12 fps. I just download our new lib and test it again

[kylemcdonald]

I did a fresh clone and install:

$ git clone https://github.com/ArduCAM/MIPI_Camera.git
$ cd MIPI_Camera/RPI
$ make install
$ make arducamstill
$ ./arducamstill -t 0 -m 4

And I get the 4 fps result. When I use mode 3 I get 8 fps.

$ cat /etc/os-release
PRETTY_NAME=”Raspbian GNU/Linux 10 (buster)”
NAME=”Raspbian GNU/Linux”
VERSION_ID=”10″
VERSION=”10 (buster)”
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL=”http://www.raspbian.org/”
SUPPORT_URL=”http://www.raspbian.org/RaspbianForums”
BUG_REPORT_URL=”http://www.raspbian.org/RaspbianBugs”

$ hostnamectl
Static hostname: raspberrypi
Icon name: computer
Machine ID: 3001bc63adc342ff8a14dae4a06e58d6
Boot ID: 131ad9825a4d48dd913fdd4ed5dcf35e
Operating System: Raspbian GNU/Linux 10 (buster)
Kernel: Linux 4.19.97-v7l+
Architecture: arm

$ gcc -v
…
gcc version 8.3.0 (Raspbian 8.3.0-6+rpi1)

$ openssl md5 lib/libarducam_mipicamera.so
MD5(lib/libarducam_mipicamera.so)= 6046474cbf30d0fb45be1630a952b819

$ openssl md5 /usr/lib/libarducam_mipicamera.so
MD5(/usr/lib/libarducam_mipicamera.so)= 6046474cbf30d0fb45be1630a952b819

This is on a fresh install of the operating system. I don’t have any other peripherals installed. I haven’t overclocked or modified anything.

One difference I can see is that my Raspberry Pi is not plugged into a monitor. But I think this should make no difference.

[bin]

Hello,

I know the reason. The signal quality of the PI4’s mipi interface is relatively poor, the data error is very serious, and the frame loss phenomenon is serious.

So, on PI4, we can’t use the 12 fps configure mode .

 

[kylemcdonald]

Wow, that’s really too bad. Thank you for looking into this.

[bin]

You are welcome. It’s my honor to try my best to help you. Why not use Pi3?

[kylemcdonald]

I am working on an application where the image quality and framerate are both very important, but there will also be some image processing happening on the device. This means that every bit of extra CPU performance is important. We are also limited in our ability to add fans for cooling. So the 1.5GHz RPI4 is favored over the 1.4GHZ RPI3.

After some initial tests, I expect that the image processing combined with the JPEG encoding (before sending out over the network) will be the primary bottleneck instead of the 8fps capture rate. Hopefully I can figure out how to push the pixels back to the GPU for image processing, and maybe use the on-board JPEG encoder too. If I can do these things in separate threads, maybe I can get a pipelined version of this code to run at 8fps, and then I will revisit this issue.

Thanks again for your help!

[bin]

Hello,

Thank you very much for the detailed description, I know your needs. The on-board jpeg encoder of the Raspberry Pi is very slow. We also have a memory copy when taking pictures, which is very unsuitable for high frame rate occasions.I think if you want to meet your requirements, you need a camera that can directly output jpeg format, why not use a UVC camera?

For mipi camera output jpeg format, we are also trying to solve the CSI2 interface parsing jpeg problem.

[kylemcdonald]

I can’t find a UVC camera with a 16MP resolution and appropriate controls. If you know a product like this, please let me know!

My system needs to output JPEG over the network. But the JPEG image directly from the IMX298 (and from most UVC cameras) does not have a good “auto” algorithm, and no good controls for preprocessing. For example, it’s common for the gamma to be set incorrectly, or for the white balance to be inaccurate. Even with manual settings on the red and blue gain can’t get the same quality as processing the raw image manually. For example, here is the difference between the automatic white balance and exposure over JPEG compared to my custom raw processing code:

[bin]

Hello,

We are developing a higher resolution camera which supports 13 MP and jpeg output. But at present, the pi’s CSI interface does support receive jpeg format.

I don’t know if it can up to your aim if the resolution is 13 MP. If it is. We will try to develop it on our USB shield board recently.

 

[bin]

Hello,

We are developing a higher resolution camera which supports 13 MP and jpeg output. But at present, the pi’s CSI interface does support receive jpeg format.

I don’t know if it can up to your aim if the resolution is 13 MP. If it is. We will try to develop it on our USB shield board recently.

 

[kylemcdonald]

For my application, USB vs CSI is not important. Most important is resolution, image quality, and frame rate. Thanks for the news, I will watch for further developments 🙏

[bin]

Hi,

We will soon release a 13MP usb camera which supports jpeg 30fps output.

 

[Author]

Posts