
A Guide to Recording 660FPS Video on a $6 Raspberry Pi Camera - robertelder
http://blog.robertelder.org/recording-660-fps-on-raspberry-pi-camera/
======
robertelder
Hi Everyone,

I do have it already linked in the article, but I should point out that the
work I did is based upon the work of (at least) several other key individuals:

A lot of the low-level sensor interaction code is found in:

[https://github.com/Hermann-SW/fork-raspiraw](https://github.com/Hermann-
SW/fork-raspiraw)

Also, check out his web site with lots of other high speed camera hacking
stuff:

[https://stamm-wilbrandt.de/en/Raspberry_camera.html](https://stamm-
wilbrandt.de/en/Raspberry_camera.html)

The original fork of raspiraw is (I think) found here:

[https://github.com/6by9/raspiraw](https://github.com/6by9/raspiraw)

And the code for converting RAW images into standard formats uses dcraw:

[https://en.wikipedia.org/wiki/Dcraw](https://en.wikipedia.org/wiki/Dcraw)

I didn't personally write much code to put this together. I would say that my
contribution here was mainly to pull together some impressive work done by
others that wasn't getting much attention and market it well. Hopefully some
of you get ideas and push this even further.

~~~
geerlingguy
Thanks for posting your work! I’d like to see if I can get much more (if any)
out of a Pi 4 which has more memory bandwidth and up to 2-4x more RAM than the
3 B+ it seems you tested with! Have you tried a 4 yet?

~~~
robertelder
I haven't yet. It was actually quite a bit if work to get these videos
recorded and document/test the process. I suspect there is tons of low-hanging
fruit left in terms of improvements that could be made. Hopefully the steps I
documented will be reproducible enough that others can make improvements and
create even more impressive demos.

------
jschwartzi
This is amazing. These framerates are 1/10th of what a cursory google search
says I can get out of a top of the line high-speed camera, but for probably
1/100th the price. You could make so many awesome instructional videos with a
tool like this.

~~~
ktpsns
Which brings me to this Gedankenexperiment: If you buy/build 10 of the
raspberry pi based high speed cameras and run them in parallel and could make
sure each camera sees the same (we don't consider details here for the time
being), is it possible to merge the videos in order to reach the frame rate of
a "top of the line high-speed camera"? I could imagine two scenarios:

    
    
      1. Start recording with time delay. Probably requires 
         excellent clock synchronisation, a real time kernel, 
         reliable/comparable hardware (gatter timings differ at 
         small timescales). If this works, this would provide a 
         high quality equidistand framerate.
    
      or
    
      2. Start recording randomly, synchronize by time or
         with an video event, and merge the frames. This will
         result in a stochastic distribution of frame rates. The 
         usefulness of such a movie depends on what it is 
         supposed to be used for (thinking of scientific usage). 
            Will probably not give nice slow-mo movies. But
         here comes the catch: With interpolation, the shortest 
         reachable frame rate dictates the quality of this
         "superposition camera". How many cheap cameras do we
         need to archieve, on average, the top line high speed 
         camera?
    

I leave the questions open for friends of Gedankenexperiments ;-)

~~~
anfractuosity
That's a good question, I'm wondering though, if the shutter speed on
commercial high speed cameras is faster, but I could be wrong?

"A Phantom camera he uses goes up to a maximum of 1000 fps in 4K — an exposure
time of 1/2000 sec with a 180° shutter speed — and the image starts looking
pretty dark on the screen."

I'm just trying to find what shutter speed you can get with the Pi camera.

Edit: Just having a little play:

./raspiraw -eus 500 -md 7 -t 1000 -ts tstamps.csv -hd0 hd0.32k -h 64 --vinc 1F
--fps 660 -r "380A,0040;3802,78;3806,0603" -sr 1 -o /dev/shm/out.%04d.raw
2>/dev/null

500 microsecond exposure (1/2000 seconds) seems to run.

~~~
HermannSW
> I'm just trying to find what shutter speed you can get with the Pi camera.

You can reduce the shutter time nearly arbitrarily, as long as you increase
light intensity. I did place fast rotating propeller just above a 5000lm led,
and did two 9us long flashes. See this image on how bright the two blades of
propeller look, at both captured locations (the rest is dark, image was taken
inside a closed cardbox, without flash everything was just black):
[https://raw.githubusercontent.com/Hermann-
SW/Raspberry_v1_ca...](https://raw.githubusercontent.com/Hermann-
SW/Raspberry_v1_camera_global_external_shutter/master/res/two.HWsync.3.jpg)

------
canada_dry
> processor registers on the image sensor are set so that raw data is sent
> continuously (to be processed later)

Very slick.

Make me wonder what crazy high-speed imaging hacks might be possible with a
top-of-the-line sensor used in iPhone/Pixel phones.

~~~
Gcfcow
The Samsung Galaxy S9 & S10 can record short bursts of 960fps video at 720p
out of the box. I'd imagine a hack could extract similar numbers from the
pixel and iPhone.

------
londons_explore
The demo videos don't have a good black level.

Is this processing method perhaps not using a correct dark image, or not
correctly subtracting off the shaded calibration rows on the sensor?

If thats the case, it's probably screwing with the gamma conversion, making it
look even worse.

~~~
robertelder
Nope, I didn't do any black level/gamma adjustments. I just used the first
values that would produce what looked like a fairly high-quality result with
ffmpeg and dcraw. There is probably tons of low hanging fruit to improve these
results from someone who really knows what the're doing in terms of image
processing.

------
an4rchy
Wow. Never thought about this approach to leveraging RAM for cameras. Awesome
guide!

Can anybody chime in regarding the resolution limit (640x64) -- If this is for
memory reasons, could we pick alternate values such as 200x200 as long as the
total pixel count is the same?

~~~
makomk
I think you're stuck with the oddball aspect ratio. Part of the CMOS readout
process is done an entire line at a time, and there just isn't a way to
increase the speed this much without dropping most of the lines. Actual high-
speed cameras have similar limitations at the higher speeds.

------
jermaustin1
I read through, and I didn't find a reason that the resolution is 640x64@660
other than that is what was input into the raspiraw program.

Could this record at a higher resolution?

~~~
michaelt
Many image sensors have a 'windowed readout' function that lets you throw away
lots of pixels' data, in exchange for a higher framerate.

So for example, a sensor that 1920px × 1080px × 30fps has the ability to
output 62,208,000 px/second - and with the same ADC and MIPI bus, halving the
number of pixels can double the number of frames per second.

You'll note this article's demo, 640px × 65px × 660fps = 27,456,000 px/second.
So you might plausibly get it a bit larger - maybe 640×120 - but you're not
going to get HD resolution at this framerate using this hardware.

(Obviously, there's also questions like exposure time and light levels so it's
not quite as simple as I just made it sound)

~~~
gambiting
I'm mostly confused by the low vertical resolution. Would something like
320x240 work?

~~~
londons_explore
This depends on the design of the sensor.

Basic sensors effectively have one analogue to digital convertor (ADC) per
column of pixels. That would mean that if you wanted to reduce the horizontal
resolution, you would be disabling some ADC's, which in turn would reduce how
many pixels can be converted per second, giving you no benefit.

~~~
ZeikJT
Would it be possible to buy a second camera and wire up its ADCs to the
sensors on the first camera and then read the next set of vertical pixels?
Effectively doubling the vertical resolution?

------
tinix
if you put the sensor on a really fast step motor, you can pivot the sensor a
tiny bit in two dimensions, to get way more resolution out of it, at a loss of
framerate, and you'll need to do some post processing.

something like this: [https://chriseyrewalker.com/the-hi-res-mode-of-the-
olympus-o...](https://chriseyrewalker.com/the-hi-res-mode-of-the-olympus-om-d-
e-m1-mark-ii/)

~~~
Avery3R
The scrolljacking on that site makes it unusable

~~~
tripzilch
Literally unusable. I found that reader mode works fine.

------
bavell
Resolution: 640x64

~~~
msie
Then get 10 Pi's to get 640x640?

~~~
azhenley
6400x640?

~~~
jefft255
Nope, that’s a 100 pies...

------
Noctem
What was the lighting setup like for the demo videos? Did you do any
processing to reduce flicker?

I get a lot of flickering from CFL and incandescent bulbs in my 240 FPS
videos, though I’ve had better results with some LEDs. Considering how much
light I’ve needed for good videos at 240 FPS, I’d imagine the lighting
requirements at 660-1000 are fairly immense.

~~~
robertelder
Usually, I would try to wait until evening when the sunlight shines directly
into my apartment. Firlming under a 60 watt lamp was still quite dark. I
didn't do any experiments with post-processing to reduce flicker, but I know
there some advanced video editors do provide such features.

------
vortico
Are there any cameras at a higher price with better stats, but still lower
than $2k for a used Chronos 1.4GP/s? I wouldn't mind buying a $1k camera to
get 720p 1000+ fps.

~~~
makomk
There are a handful of flagship smartphones that cost under a grand and
technically support 960fps at 720p - the Samsung Galaxy S9, Note 9 and S10,
along with Sony's Xperia XZ series. They come with a bunch of caveats
including very limited recording time of less than half a second. A number of
other smartphones advertise 960fps but fake it using software interpolation.

The Chronos 1.4 is very aggressively priced for what it offers, even if it
does cost several grand.

------
alvern
1107 fps on the V2 IMX219 camera. I wonder how fast or how large of a frame it
could do on a Pi4 or odroid (article mentions the 640x64 frame is limited by
memory bandwidth).

~~~
HermannSW
You can do 640x64@665fps with v1 camera, and 640x128@667fps with v2 camera:
[https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=2125...](https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=212518&p=1310445#p1320034)

In tools directory you can use *_s tools to get double FOV vertically while
keeping the framerate. So you can get 640x128_s@665fps/640x256_s@667fps for
v1/v2 camera.

There real limit for v2 camera is 1007fps, with can be achieved with 640x75 or
640x150_s tools. You can reduce vertical resolution below 75, but you will not
get more than 1007fps (if you find out how to raise that limit, please shout!
Deal for raspiraw high framerate capturing basically is "halve vertical
resolution and get double framerate").

------
novaRom
Everything is doable if you have detailed description of the sensor. Do you
know any sensor to get a better resolution with comparable frame rate and
detailed specs?

Can RAM limits be improved with a simple compression?

~~~
penagwin
He did his testing on a raspberry pi 3 which means only 1GB of LPDDR2.

I'd love to see how the RAM limits improve with 4GB LPDDR4, especially with
usb 3.0 and real ethernet....

~~~
novaRom
Camera interface is not via USB on Raspberry Pi. There's a lot of redundancy
in high frame rate, so the best way to improve memory footprint is
compression.

~~~
zaarn
At high framerate, there isn't much time to do any type of fancy compression.
660fps means you have 1.5 milliseconds to process the image. The bandwidth
will roughly max out between 50 Megabytes/s or 432Mbit/s. LPDDR2 (on the RPi3)
can do 800 Transfers are second, at a maxmium transfer size of 4kb, maxing out
almost all of your memory bandwidth already.

So you literally do not have the memory bandwidth to do compression while also
capturing a good 640x64x32/660 Video.

If you have an RPi4, you get LPDDR4, the memory bandwidth now allows for
compression, _IF_ the CPU can handle that throughput (which it likely can't
since the RPi4 chokes on such large bandwidths easily).

~~~
rasz
Ram speed is not the issue, camera interface is limited to 120MB/s, ram
bandwidth is >2GB/s on pee3, 4GB/s on pee4.

~~~
zaarn
If you want to compress the frames, RAM bandwidth will be an issue.

~~~
rasz
Compressing 640x64@660 takes exactly the same amount of work as compressing
1280x720@30. You could even use hardware h.264 encoder as long as you can
debayer raw 10bit in software fast enough (or convince Broadcom employee 6by9
to make internal Videocore firmware debayering resolution agnostic, its
already fast enough keeping up with twice the amount of data when using camera
in 1920x1080@25 mode).

But you dont even need fancy compression, at those framerates things change
really slowly so simple delta encoding on raw data will do wonders.

~~~
zaarn
Compressing 660fps in real time is a much different task than 30 frames per
second, notably because your latency budget is a lot lower. At 30fps you have
33ms to load the frame, compress it and send it off to the destination, at
660fps you have 1.5ms for the same task, which does not necessarily get
exponentially faster with lowered resolution.

~~~
rasz
You have exactly as much time as at 30fps at 22x higher resolution. You dont
need to constantly reuse the same 52KB buffer. Data comes at a steady snail
35MB/s pace. Pee 3 should even be able to save it on USB2 HDD in real time
with no compression.

~~~
HermannSW
Really? USB2 is 300Mbps for PI3 ethernet over USB2, is it faster with hdd? I
ask because snail 35MB/s is 280Mbps.

~~~
rasz
replied here
[https://news.ycombinator.com/item?id=20658147](https://news.ycombinator.com/item?id=20658147)

last time I tried USB 2.0 hdd on pee3 did 35MB/s on the dot

------
sytelus
Very cool. Can this camera also be hacked for amplified sensor reading for
night vision? Infrared or UV photography? Long exposure photography?

~~~
HermannSW
Yes, there are NoIR versions of v1 camera, they just miss the infrared filter
normal cameras have (around 9$ on aliexpress.com). And there are 12$ versions
of v1 camera with (hardware) IR-cut filter, depending on photo sensor on
camera IR filter is enabled at day, and removed at night.

~~~
HermannSW
I was wrong, the 9$ are for NoIR camera with M12 lens mount and lens. I just
ordered 4 new NoIR cameras for 2.84$(!) including shipping!
[https://www.aliexpress.com/item/32946093276.html](https://www.aliexpress.com/item/32946093276.html)

------
person_of_color
Could you stream the frames over 10G LAN to record longer videos?

~~~
HermannSW
There is no Pi with 10G ethernet. New Pi4 has only Gigabit ethernet, Pi 3B+
has Gigabit Ethernet over USB 2.0 (maximum throughput 300 Mbps).

The idea is interesting, did some calculation. For v1 camera capturing 640xH
frames can be done at framerate "42755/H+13". Lets forget about the 13.
raspiraw transfers raw10 Bayer data, that is 640xHx1.25 bytes, with maximal
framerate that gives 42755x640x1.25=34204000 bytes/s=261Mbit/s(!).

So you definitely need Gigabit ethernet of the Pi4, not sure whether Pi4 can
actually stream out 261Mbit without losing stuff though ...

Just asked on Raspberry networking forum (261Mbps for v1, 464Mbps for v2):
[https://www.raspberrypi.org/forums/viewtopic.php?f=36&t=2482...](https://www.raspberrypi.org/forums/viewtopic.php?f=36&t=248254)

~~~
person_of_color
So it's feasible for Pi4?

~~~
HermannSW
No answer on the networking forum yet. I cannot try because I don't have a Pi4
yet. It seems possible. An easy test would be to test whether this command
will work on Pi4:

raspividyuv -md 7 -w 640 -h 480 -t 0 -fps 90 -o - | nc someIp somePort

and on someIp server service listeing on somePort just storing the data
received. Because yuv has 12bits/pixel which is a bit more than the
10bit/pixel of raw Bayer, the command produces a 316Mbps stream of data for v1
camera (732Mbps for v2 camera when recording with "-fps 180"). For testing the
service on someIp can be as easy as:

nc -l somePort > foobar

~~~
rasz
You should be fine with RPi3B+ internal gigabit NIC

[https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=2085...](https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=208512&start=50#p1293094)

pi@raspberrypi:~ $ sudo iperf3 -c XXX.XXX.X.XXX

[ 4] 0.00-1.00 sec 37.9 MBytes 318 Mbits/sec 0 257 KBytes

------
yeahdef
very impressive.

