Maximum Framerate IMX298 + RPI4
Published by kylemcdonald on
New Home › Forums › 1. Cameras Modules for Raspberry Pi › Arducam MIPI Camera Modules › Maximum Framerate IMX298 + RPI4
- This topic has 28 replies, 2 voices, and was last updated 7 months, 2 weeks ago by
bin.
- AuthorPosts
- May 14, 2020 at 9:49 pm #22645
kylemcdonald
ParticipantWhat is the maximum framerate for the IMX298 on RPI4 at full resolution?
I wrote some short code in C to test. I set auto exposure off, exposure time to 1, resolution to 4656×3496 and I used IMAGE_ENCODING_RAW_BAYER. I measured 4.4fps (not including the first frame which takes longer).
Is it possible to increase the speed? What is the primary bottleneck?
I also tried IMAGE_ENCODING_I420, but the speed is similar (even though the bandwidth is larger). I also tried IMAGE_ENCODING_JPEG, but it causes a crash:
`
usrvcsm: [vcsm_malloc_cache]: [5925] [mmal_vc_port buffer]: ioctl mem-alloc FAILED [-1] (hdl: 0)
usrvcsm: [vcsm_vc_hdl_from_hdl]: [5925]: invalid device or handle!
usrvcsm: [vcsm_lock]: [5925]: invalid device or invalid handle!
mmal: mmal_vc_shm_alloc: could not allocate 48944896 bytes of shared memory (handle 0) – mem (nil), vc_hdl 00000000
mmal: mmal_vc_port_payload_alloc: vc.ril.image_encode:out:0(JPEG): failed to allocate 48944896 bytes of shared memory
mmal: mmal_pool_initialise_buffer_headers: failed to allocate payload 0/1
mmal: Failed to create isp output pool.
`
- May 16, 2020 at 1:50 am #22689
bin
KeymasterHi,
The main reason is the 2 channel bandwidth limitation of PI.
Please change the GPU memory to aviod the malloc error .
- May 17, 2020 at 1:17 am #22695
kylemcdonald
ParticipantThanks. I was able to get the JPEG setting working at full resolution. I can see that you answer this very basic question all the time, so thank you for your patience.
I thought that JPEG compression was performed by the IMX298. But when I set
IMAGE_FORMAT fmt = {IMAGE_ENCODING_JPEG, 1};
andarducam_set_control(camera_instance, V4L2_CID_EXPOSURE, 1);
so that the images are only 0.2MB, I can see that the framerate is still 4.4fps. So this must mean that the JPEG compression is performed by the arducam library.When I modify the
raw_callback
oryuv_callback
examples to use the full resolution I also get 4fps.I read that CSI2 supports 1Gbit/s per lane, for 2Gbit/s total = 2,000,000,000 bits/s. A full frame of
IMAGE_ENCODING_RAW
is 20,407,296 bytes = 163,258,368 bits. 2,000,000,000 bits/s / 163,258,368 bits = 12.2 fps.And on this page https://www.arducam.com/docs/cameras-for-raspberry-pi/mipi-camera-modules/ it says that the full resolution framerate is 10fps. This page says the full resolution framerate is 30fps https://www.arducam.com/docs/cameras-for-raspberry-pi/mipi-camera-modules/imx298-16mp-color-rolling-shutter-camera-for-raspberry-pi/
How can I achieve 10-12 fps?
- May 17, 2020 at 8:12 pm #22716
bin
KeymasterHello,
Yes, you are right. Our configuration currently runs to 8 frames. You can use arudcamstill to test. Due to different exposure times, there will be some delay. The mainly factor is the sensor’s configure. Recently, I will try to optimize the sensor configuration
IF you use the encoder to get the jpeg image, there is a memory copy of the CPU fetch graph, and a lot of time will be lost.
- May 17, 2020 at 10:49 pm #22720
bin
KeymasterHello,
I have optimized the imx298 configuration just now and the max resolution can up to 12 fps.
- May 18, 2020 at 6:15 pm #22721
kylemcdonald
ParticipantHi bin, I pulled the new code including the new library:
MD5(lib/libarducam_mipicamera.so)= ded62a83575b0a5f61ba13dadb597a0e
Then I did
make clean; make arducamstill
, but I still see 4fps:I also checked with my own code and it still shows 4.4fps.
https://gist.github.com/kylemcdonald/eb5c72c19129039ee4f83472d4450298
- May 19, 2020 at 8:46 pm #22835
bin
KeymasterHi, You should make install to install the new lib at the MIPI_Camera path.
However, some people recently reported that the high frame rate is too fast, and our line will be a little disturbed. I still reduced the frame rate to 8 frames.
- May 19, 2020 at 9:27 pm #22837
kylemcdonald
ParticipantOf course, I’m surprised I forgot
make install
. Thanks.I tried the latest code and now I get between 7-8 fps.
I also tried the previous version (after
git reset --hard f337a4ddb7430e46e8db70f9cb01ba459c50fd72; make clean; make install
) but I was unable to get 12 fps. Instead I get aSegmentation fault
.Would it be possible to expose the internal parameter that is allowing you to set the maximum framerate? I would like to be able to set this myself.
- May 20, 2020 at 3:24 am #22860
bin
KeymasterHello,
You can try to reduce the value of the 0x0307 register to increase the clock.
- May 20, 2020 at 6:44 pm #22877
kylemcdonald
ParticipantThanks! I will look into this. I haven’t set the registers on Raspberry Pi before so I will look for some examples first.
For anyone else coming across this thread, here are some more references that might be relevant:
https://www.raspberrypi.org/forums/viewtopic.php?t=254733
https://www.raspberrypi.org/forums/viewtopic.php?p=630909#p630909
- May 21, 2020 at 4:00 am #22905
bin
KeymasterGreat, thanks for your share.
Just use this API I attached for you to change the value of the register.
int arducam_write_sensor_reg(CAMERA_INSTANCE camera_instance, uint16_t address, uint16_t value);
- May 21, 2020 at 5:33 pm #22921
kylemcdonald
ParticipantThanks! I used
arducam_read_sensor_reg
to get the current value, and I see it is112
.Assuming this is directly connected to the clock speed, I tried to change it to
74
to get 12fps witharducam_write_sensor_reg(camera_instance, 0x0307, 74);
but I don’t see any change.I also tried changing it to a larger value like
224
but I did not see any change.I also tried setting the register immediately after
arducam_init_camera
and immediately beforearducam_capture
but this showed no difference in results.https://gist.github.com/kylemcdonald/eb5c72c19129039ee4f83472d4450298#file-capture_fast-cpp-L30
- May 21, 2020 at 6:48 pm #22939
bin
KeymasterHello,
Oh I know, the sensor is very strange, you should change the clock before it’s streaming on register write 1.
I have help you add the mode. I have tested it on PI3 platform. Please try to download our new lib and make install. Then retry uisng mode 4 .
- May 22, 2020 at 3:31 pm #22996
kylemcdonald
ParticipantI pulled the new code (library) and ran
make install
then I replaced the call toarducam_set_resolution
witharducam_set_mode(camera_instance, 4);
immediately after callingarducam_init_camera
.From my tests, this new mode actually runs slower. When I modified
raw_callback.c
to test the speed: mode 3 runs around 7.8fps, and mode 4 runs at 4.8fps.I checked the 0x0307 register and I noticed that it is 160 in mode 4 and 112 in mode 3.
I tried changing the register 0x0307 in mode 4, and it did not have any effect.
- May 26, 2020 at 3:46 am #23086
bin
KeymasterHello,
Strange, Have you tested it on PI3 platform?
- May 26, 2020 at 3:46 am #23087
bin
KeymasterHello,
Strange, Have you tested it on PI3 platform?
- May 26, 2020 at 3:01 pm #23099
kylemcdonald
ParticipantI don’t have a PI3 for testing, and need a PI4 for my application. If there are any updates to bring the PI4 up to the 12fps frame rate please let me know!
- May 26, 2020 at 8:45 pm #23123
bin
KeymasterHi,
I am sure I have configure the sensor to 12 fps. I just download our new lib and test it again
- May 27, 2020 at 5:14 pm #23161
kylemcdonald
ParticipantI did a fresh clone and install:
$ git clone https://github.com/ArduCAM/MIPI_Camera.git
$ cd MIPI_Camera/RPI
$ make install
$ make arducamstill
$ ./arducamstill -t 0 -m 4And I get the 4 fps result. When I use mode 3 I get 8 fps.
$ cat /etc/os-release
PRETTY_NAME=”Raspbian GNU/Linux 10 (buster)”
NAME=”Raspbian GNU/Linux”
VERSION_ID=”10″
VERSION=”10 (buster)”
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL=”http://www.raspbian.org/”
SUPPORT_URL=”http://www.raspbian.org/RaspbianForums”
BUG_REPORT_URL=”http://www.raspbian.org/RaspbianBugs”$ hostnamectl
Static hostname: raspberrypi
Icon name: computer
Machine ID: 3001bc63adc342ff8a14dae4a06e58d6
Boot ID: 131ad9825a4d48dd913fdd4ed5dcf35e
Operating System: Raspbian GNU/Linux 10 (buster)
Kernel: Linux 4.19.97-v7l+
Architecture: arm$ gcc -v
…
gcc version 8.3.0 (Raspbian 8.3.0-6+rpi1)$ openssl md5 lib/libarducam_mipicamera.so
MD5(lib/libarducam_mipicamera.so)= 6046474cbf30d0fb45be1630a952b819$ openssl md5 /usr/lib/libarducam_mipicamera.so
MD5(/usr/lib/libarducam_mipicamera.so)= 6046474cbf30d0fb45be1630a952b819This is on a fresh install of the operating system. I don’t have any other peripherals installed. I haven’t overclocked or modified anything.
One difference I can see is that my Raspberry Pi is not plugged into a monitor. But I think this should make no difference.
- May 27, 2020 at 6:42 pm #23173
bin
KeymasterHello,
I know the reason. The signal quality of the PI4’s mipi interface is relatively poor, the data error is very serious, and the frame loss phenomenon is serious.
So, on PI4, we can’t use the 12 fps configure mode .
- May 27, 2020 at 6:50 pm #23174
kylemcdonald
ParticipantWow, that’s really too bad. Thank you for looking into this.
- May 28, 2020 at 6:17 pm #23230
bin
KeymasterYou are welcome. It’s my honor to try my best to help you. Why not use Pi3?
- May 28, 2020 at 11:16 pm #23234
kylemcdonald
ParticipantI am working on an application where the image quality and framerate are both very important, but there will also be some image processing happening on the device. This means that every bit of extra CPU performance is important. We are also limited in our ability to add fans for cooling. So the 1.5GHz RPI4 is favored over the 1.4GHZ RPI3.
After some initial tests, I expect that the image processing combined with the JPEG encoding (before sending out over the network) will be the primary bottleneck instead of the 8fps capture rate. Hopefully I can figure out how to push the pixels back to the GPU for image processing, and maybe use the on-board JPEG encoder too. If I can do these things in separate threads, maybe I can get a pipelined version of this code to run at 8fps, and then I will revisit this issue.
Thanks again for your help!
- May 31, 2020 at 7:54 pm #23310
bin
KeymasterHello,
Thank you very much for the detailed description, I know your needs. The on-board jpeg encoder of the Raspberry Pi is very slow. We also have a memory copy when taking pictures, which is very unsuitable for high frame rate occasions.I think if you want to meet your requirements, you need a camera that can directly output jpeg format, why not use a UVC camera?
For mipi camera output jpeg format, we are also trying to solve the CSI2 interface parsing jpeg problem.
- June 1, 2020 at 8:35 pm #23436
kylemcdonald
ParticipantI can’t find a UVC camera with a 16MP resolution and appropriate controls. If you know a product like this, please let me know!
My system needs to output JPEG over the network. But the JPEG image directly from the IMX298 (and from most UVC cameras) does not have a good “auto” algorithm, and no good controls for preprocessing. For example, it’s common for the gamma to be set incorrectly, or for the white balance to be inaccurate. Even with manual settings on the red and blue gain can’t get the same quality as processing the raw image manually. For example, here is the difference between the automatic white balance and exposure over JPEG compared to my custom raw processing code:
- June 3, 2020 at 7:29 am #23513
bin
KeymasterHello,
We are developing a higher resolution camera which supports 13 MP and jpeg output. But at present, the pi’s CSI interface does support receive jpeg format.
I don’t know if it can up to your aim if the resolution is 13 MP. If it is. We will try to develop it on our USB shield board recently.
- June 3, 2020 at 7:29 am #23514
bin
KeymasterHello,
We are developing a higher resolution camera which supports 13 MP and jpeg output. But at present, the pi’s CSI interface does support receive jpeg format.
I don’t know if it can up to your aim if the resolution is 13 MP. If it is. We will try to develop it on our USB shield board recently.
- June 3, 2020 at 11:42 am #23533
kylemcdonald
ParticipantFor my application, USB vs CSI is not important. Most important is resolution, image quality, and frame rate. Thanks for the news, I will watch for further developments 🙏
- June 4, 2020 at 7:11 am #23556
bin
KeymasterHi,
We will soon release a 13MP usb camera which supports jpeg 30fps output.
- AuthorPosts
- You must be logged in to reply to this topic.