I do have it already linked in the article, but I should point out that the work I did is based upon the work of (at least) several other key individuals:
A lot of the low-level sensor interaction code is found in:
Also, check out his web site with lots of other high speed camera hacking stuff:
The original fork of raspiraw is (I think) found here:
And the code for converting RAW images into standard formats uses dcraw:
I didn't personally write much code to put this together. I would say that my contribution here was mainly to pull together some impressive work done by others that wasn't getting much attention and market it well. Hopefully some of you get ideas and push this even further.
But 660fps is not the end for v1 camera, I added section "kFPS videos from kEPS multiple exposure frames" to my github "v1 camera global external shutter" repo:
In that section a 3000fps and a 9000fps animation are shown (and the tools needed to create them)!
The repo introduction now contains comparison of 20$ diy highspeed flashlight with commercially available one (and with the very costly one you cannot produce kEPS multiple exposures and kFPS framerate videos):
1. Start recording with time delay. Probably requires
excellent clock synchronisation, a real time kernel,
reliable/comparable hardware (gatter timings differ at
small timescales). If this works, this would provide a
high quality equidistand framerate.
2. Start recording randomly, synchronize by time or
with an video event, and merge the frames. This will
result in a stochastic distribution of frame rates. The
usefulness of such a movie depends on what it is
supposed to be used for (thinking of scientific usage).
Will probably not give nice slow-mo movies. But
here comes the catch: With interpolation, the shortest
reachable frame rate dictates the quality of this
"superposition camera". How many cheap cameras do we
need to archieve, on average, the top line high speed
"A Phantom camera he uses goes up to a maximum of 1000 fps in 4K — an exposure time of 1/2000 sec with a 180° shutter speed — and the image starts looking pretty dark on the screen."
I'm just trying to find what shutter speed you can get with the Pi camera.
Edit: Just having a little play:
./raspiraw -eus 500 -md 7 -t 1000 -ts tstamps.csv -hd0 hd0.32k -h 64 --vinc 1F --fps 660 -r "380A,0040;3802,78;3806,0603" -sr 1 -o /dev/shm/out.%04d.raw 2>/dev/null
500 microsecond exposure (1/2000 seconds) seems to run.
You can reduce the shutter time nearly arbitrarily, as long as you increase light intensity. I did place fast rotating propeller just above a 5000lm led, and did two 9us long flashes. See this image on how bright the two blades of propeller look, at both captured locations (the rest is dark, image was taken inside a closed cardbox, without flash everything was just black):
As long as your leads didn't have too much capacitance, it ought to just work.
The phase shifting you can do in software by (for example) restarting the camera hundreds of times till by chance it starts a frame with the phase offset you want, and then just run on from there.
I'm talking the actual quartz crystal here, not the output of some digital clocking chip (which you're right, wouldn't work).
There was a MIT student project group that took 16 v1 cameras(!) global external shutter frames and created an animated .gif from that!
"Ultra high-speed imaging, capturing matrix style bullet time"
Reproducing something like this using cheap sensors would require some hardware acumen, but isn't unfeasible. Most industrial cameras, even cheap ones, have external triggers/clocks with nanosecond latency.
The MIT group used a flash with 0.5us exposure time. I asked for details without answer yet, but assume that it is costly (Vela One does cost 940£) and one-shot, no fast repeating flashes (Vela One has only 50Hz maximal continuous strobe frequency).
I used cheap 2$ 5000lm LEDs with 7$ led driver to convert mains to 38V/1.5A DC. I was able to do global external shutter captures with that led with 4us shutter time, did not try below yet. Most interestingly these LEDs allowed for 8.33us flashes at 9KHz frequency and used that to capture 109m/s flying airgun pellet 10 times in a single multiple exposure frame (9000eps, exposures per second). PI pwm should easily allow for 90,000eps frames for capturing 900m/s 0.50bmg bullets multiple times per frame in flight ...
You can do the same with 6$ Raspberry v1 camera clone (not with v2), bill of materials as well as global external shutter setup, software and example captured frames can be found here:
And unlike v2 camera, the cheap v1 camera allows (besides high framerate capturing) for global external shutter capturing as well(!). I did capture an in flight 109m/s airgun pellet 10 times in a single multiple exposure frame!
Make me wonder what crazy high-speed imaging hacks might be possible with a top-of-the-line sensor used in iPhone/Pixel phones.
Is this processing method perhaps not using a correct dark image, or not correctly subtracting off the shaded calibration rows on the sensor?
If thats the case, it's probably screwing with the gamma conversion, making it look even worse.
Can anybody chime in regarding the resolution limit (640x64) -- If this is for memory reasons, could we pick alternate values such as 200x200 as long as the total pixel count is the same?
This is normal for high refresh rate cameras. They store straight to ram then dump to the disk (SSD) after. Especially the higher end cameras, they literally can't store data fast enough to go anywhere else.
Light exposure on the sensor, the faster the frame rate, the darker the image as its a fixed aperture. Photographic buffs could explain it better, but my take is the faster the exposure the bigger the aperture that is needed to let more light in to make the photo look well lit. I don't know if parts of the sensor also work faster due to its circuit design but I would imagine this is also a factor just like its possible to read more data form the outside of a spin disk than the tracks close to the centre.
Could this record at a higher resolution?
So for example, a sensor that 1920px × 1080px × 30fps has the ability to output 62,208,000 px/second - and with the same ADC and MIPI bus, halving the number of pixels can double the number of frames per second.
You'll note this article's demo, 640px × 65px × 660fps = 27,456,000 px/second. So you might plausibly get it a bit larger - maybe 640×120 - but you're not going to get HD resolution at this framerate using this hardware.
(Obviously, there's also questions like exposure time and light levels so it's not quite as simple as I just made it sound)
Basic sensors effectively have one analogue to digital convertor (ADC) per column of pixels. That would mean that if you wanted to reduce the horizontal resolution, you would be disabling some ADC's, which in turn would reduce how many pixels can be converted per second, giving you no benefit.
You can do the opposite and increase horizonatal resolution for line scanner type of application with still impressive framerates:
"v2 camera can do 3280x32@720fps and 3280x64@500fps with frame skip rate less than 1% (tools attached)."
You can do quite impressive gobal external shutter capturings (with v1 camera only!), eg. capturing a 109m/s
airgun pellet in flight on a single multiple exposure frame 10x:
Deal is basically "halve vertical resolution, and get roughly double framerate" (up to cap imit of around 665fps for v1 and exactly 1007fps for v2 camera).
In addition "tools/*_s" tools double vertical FOV while keeping the framerate.
Remember these sensors read a single row at a time. There's a Smarter Every Day video where they show how cellphones get those weird effects when recording plane propellers due to the way they scan images.
something like this: https://chriseyrewalker.com/the-hi-res-mode-of-the-olympus-o...
 Video from article: https://www.youtube.com/watch?v=-gMy8k4nHtw
I get a lot of flickering from CFL and incandescent bulbs in my 240 FPS videos, though I’ve had better results with some LEDs. Considering how much light I’ve needed for good videos at 240 FPS, I’d imagine the lighting requirements at 660-1000 are fairly immense.
The Chronos 1.4 is very aggressively priced for what it offers, even if it does cost several grand.
In tools directory you can use *_s tools to get double FOV vertically while keeping the framerate. So you can get 640x128_s@665fps/640x256_s@667fps for v1/v2 camera.
There real limit for v2 camera is 1007fps, with can be achieved with 640x75 or 640x150_s tools. You can reduce vertical resolution below 75, but you will not get more than 1007fps (if you find out how to raise that limit, please shout! Deal for raspiraw high framerate capturing basically is "halve vertical resolution and get double framerate").
Can RAM limits be improved with a simple compression?
I'd love to see how the RAM limits improve with 4GB LPDDR4, especially with usb 3.0 and real ethernet....
So you literally do not have the memory bandwidth to do compression while also capturing a good 640x64x32/660 Video.
If you have an RPi4, you get LPDDR4, the memory bandwidth now allows for compression, IF the CPU can handle that throughput (which it likely can't since the RPi4 chokes on such large bandwidths easily).
But you dont even need fancy compression, at those framerates things change really slowly so simple delta encoding on raw data will do wonders.
last time I tried USB 2.0 hdd on pee3 did 35MB/s on the dot
Instead, one needs to use a GPU, DSP, or other dedicated hardware to process/compress the data into a common video or image format.
And you can hack the camera by injecting I2C commands to achieve effects impossible with the normal Raspbian camera software, like taking odd/even frames with different shutter speed:
For reverse engineering camera I2C traffic, the v1 camera ov5647 datasheet is useful to have:
The first thing of note to me in  is the following:
> Digital video cameras perform a process called demosaicking before compression, which normally is lossy. While capturing images, digital video camera, uses only one of the three primary colors (red, green, blue) at each pixel and the remaining colors are interpolated through color demosaicking to reconstruct the full color image. A demosaicking algorithm is a digital image process used to reconstruct a full color image from the incomplete color samples output from an image sensor overlaid with a color filter array (CFA). The most commonly used CFA is Bayer Filter, where the output is an array of pixel values, each indicating a raw intensity of one of the three filter colors. The process of demosaicking, followed by lossy compression is irreversible and the video cannot be improved later on. Further, this process increases complexity, reduces compression ratio and burdens the camera I/O bandwidth.
> [...] if a lossless compression is performed first in the camera itself before demosaicking, then a sophisticated codec can be designed which is needed in applications where high visual quality has paramount importance. This motivated the use of lossless-compression first followed by demosaicking in medical videos where even the slightest degradation result is catastrophic situations. The algorithm uses a hybrid scheme of inter- and intraframe lossless compression to increase the compression and quality of the medical video data. The working of the encoding algorithm is briefed below and is given pictorially in Figure 1.
Skimming through the data sheet, I see a few things:
- Among the features on page 1 it mentions a 10-bit ADC, and “R, G, B primary color pigment mosaic filters on chip”.
- On page 10, there is a block diagram, figure 1.
- Starting at page 52 and throughout various pages of the remainder of the data sheet, there are figures that indicate to me that the output of the IMX 219 consists of mosaic pixel data. So if I am understanding correctly, demosaicing happens outside of the IMX 219.
So I am wondering, with the v2 camera module as a whole, is demosaicing performed on the module itself or in software on the Raspberry Pi? And if it happens on the module itself, can it optionally be disabled so that you can get the mosaic pixel array data?
Since the ADC is 10-bit, each pixel in the pixel array is presumably represented by 10 bits. The data sheet might say, I only skimmed through it on the initial read.
If so, then 640x64 pixels * 10 bits/pixel * 660 FPS ~= 270 Mbps ~= 34 MBps.
A benchmark from 2018  puts the Raspberry Pi model 3B+ at being capable of writing to the SD card at about 22 MBps.
So if we get the mosaic pixel array data, the most simple and naive thing we could do would be to “crop” the image to say 360 pixels wide in memory by skipping past the 140 first pixels of each line and copying the 360 next pixels of each of the 64 lines for a few frames, writing batches of 360x64 pixels of cropped frame data to a single file on the SD card.
If the mosaic pixel array is what we are given by default, or we can get the camera module to send it anyway, then with discarding data we have a starting point. At this point we are able to record for as long as the capacity of our SD card will allow us. 128 GB SD cards are not terribly expensive so it should be possible to record data for like a couple of minutes. Then we can later post process the data.
And if that is possible, then one can start to get really serious and try to device an efficient lossless compression algorithm specifically for our 640x64 pixels of video frames, and perhaps even optimized for different use-cases like horizontal motion only, vertical motion only, and motion in horizontal and vertical directions at the same time.
The idea is interesting, did some calculation.
For v1 camera capturing 640xH frames can be done at framerate "42755/H+13".
Lets forget about the 13.
raspiraw transfers raw10 Bayer data, that is 640xHx1.25 bytes, with maximal framerate that gives 42755x640x1.25=34204000 bytes/s=261Mbit/s(!).
So you definitely need Gigabit ethernet of the Pi4, not sure whether Pi4 can actually stream out 261Mbit without losing stuff though ...
Just asked on Raspberry networking forum (261Mbps for v1, 464Mbps for v2):
raspividyuv -md 7 -w 640 -h 480 -t 0 -fps 90 -o - | nc someIp somePort
and on someIp server service listeing on somePort just storing the data received. Because yuv has 12bits/pixel which is a bit more than the 10bit/pixel of raw Bayer, the command produces a 316Mbps stream of data for v1 camera (732Mbps for v2 camera when recording with "-fps 180"). For testing the service on someIp can be as easy as:
nc -l somePort > foobar
pi@raspberrypi:~ $ sudo iperf3 -c XXX.XXX.X.XXX
[ 4] 0.00-1.00 sec 37.9 MBytes 318 Mbits/sec 0 257 KBytes