
Rolling Shutter Simulation in C - ingve
http://nullprogram.com/blog/2017/07/02/
======
saurik
A couple friends of mine wrote an extremely mathematical analysis of the
rolling shutter effect, and even looked at some related questions like "given
a photo from a rolling shutter how can you tell how many propellor blades
actually exist and how fast it is turning?".

[http://danielwalsh.tumblr.com/post/54400376441/playing-
detec...](http://danielwalsh.tumblr.com/post/54400376441/playing-detective-
with-rolling-shutter-photos)

~~~
coldtea
> _and even looked at some related questions like "given a photo from a
> rolling shutter how can you tell how many propellor blades actually exist
> and how fast it is turning?"_

I've seen some code (don't remember the website now) that gets you the non-
rolled shuttered (sic) image from the rolling shutter one.

(It works to reconstruct a scene, but has issues at edges etc, so it's not
really usable to fix a photo/video 100% -- but there are commercial NLE
plugins that do reduce rolling shutter in post).

~~~
saurik
One concrete example: the most recent version of Autopano Video Pro does
rolling shutter compensation as part of its stitching of 360 video from a rig
of GoPro cameras.

------
Retr0spectrum
The author mentioned YUV4MPEG. It's a wonderfully simple video format, here's
a tiny C program to stream a video of Game of Life (With colours!):

    
    
        #define  F for(i=0;i<l*3;i++)
        main(n){int i,j,w=640,h=480,l
        =w*h,o[]={~w,-w,-w+1,-1,1,w-1
        ,w,w+1},b[l*5];F b[i]=rand();
        for(puts("YUV4MPEG2 W640 H48"
        "0 F30 C444");puts("FRAME");)
        {F{for(n=j=8;j;n-=b[i+o[--j]]
        &1);b[i+l]=(n^5&&!b[i]|n^6)-1
        ;}F putchar(b[i]=b[i+l]);}}
    

(You can pipe the output to mpv or a similar media player.)

~~~
sillysaurus3
Heh. I tried running that code through three different beautifiers and they
all failed spectacularly.

For those of us who don't enjoy visual self-flagellation, here's an attempt at
untying the above ball of yarn:

#define F for(i=0;i<l _3;i++)

    
    
      main(n)
      {
        int i,j,
            w=640, h=480,
            l=w*h,
            o[] = {
              ~w, -w, -w+1,
              -1, 1,
              w-1, w, w+1
            },
            b[l*5];
    
        F
          b[i]=rand();
    
        for( puts("YUV4MPEG2 W640 H48" "0 F30 C444");
             puts("FRAME");
            )
        {
          F
          {
            for( n=j=8; j; n-=b[ i + o[--j] ]&1 );
    
            b[i+l] = (n^5 && !b[i] | n^6) - 1;
          }
    
          F
            putchar( b[i]=b[i+l] );
        }
      }
    
    

EDIT: I had no idea you could italicize code blocks! This was an accident, but
I'm going to leave it because it looks pretty. Has this always been a feature?
I've never seen italicized monospace before. That first line is supposed to
read "for(i=0; i < l times three ..." but the asterisk is somehow causing HN
to italicize the code block. There's no matching closing asterisk, so this is
pretty amusing.

~~~
sillysaurus3
foo*bar

    
    
      baz
    

See? No italicization.

Now I'm curious what triggers it...

~~~
sillysaurus3
foo _bar

    
    
      baz*quux
    

Aha. HN's parser is matching with the asterisk in the code snippet.

~~~
dingo_bat
It even affects the reply button/text.

------
morecoffee
That's really cool. I wanted to see what the second order effect would be (as
the shutter speed changes, how does the final image change?)

Take a look: [http://antidom.com/fan.webm](http://antidom.com/fan.webm)

~~~
emmelaich
Nice. Reminds me of WW2 film of propeller planes. The propellors often looked
bent back.

------
sitkack
That video [1] has been motivating a lot of folks! Yesterday someone posted a
Rust [2] solution to do the same thing.

[1]
[https://www.youtube.com/watch?v=dNVtMmLlnoE&feature=youtu.be](https://www.youtube.com/watch?v=dNVtMmLlnoE&feature=youtu.be)

[2]
[https://www.reddit.com/r/rust/comments/6kl0vj/i_made_a_progr...](https://www.reddit.com/r/rust/comments/6kl0vj/i_made_a_program_in_rust_for_generating_rolling/)

------
rasz
Rolling shutter might sound very bad, but has some nice side effect
properties. Latency between light hitting the sensor and readout is minimized
(divided by rolling shutter readout block size/row) compared to global
shutter. This means your 1080p 60Hz camera can have 64KHz time resolution.

Cool research into RS effects: "Direct Semi-dense SLAM for Rolling Shutter
Cameras".
[https://www.youtube.com/watch?v=OLD1eeu1EUI](https://www.youtube.com/watch?v=OLD1eeu1EUI)
[https://github.com/jaehak/rrd_slam](https://github.com/jaehak/rrd_slam)

This only tries to reverse adverse effect of RS (at best as good as global
shutter), but in theory it would be possible to exploit RS effect to boost
accuracy above GS.

Afaik Jeri Ellsworth CastAR (shutdown, because Rubin invested only to sell to
google, and that didnt happen) IR tracker took advantage of cheap cellphone
camera rolling shutter to boost both spatial and temporal tracking resolution.

------
vortico
Nice! I'm curious if there's an open platform for writing video effects that
would be compatible with other video editing and conversion software.
Something that doesn't have any glaring inefficiencies and in C or C++.

~~~
kitotik
Openframeworks - [http://openframeworks.cc](http://openframeworks.cc)

It’s a c++ multimedia Swiss Army knife with a pretty strong community and repo
of addons.

Libcinder is another option, though only for Mac/Win

------
OskarS
That's a really clever trick, piping the video to PPM format and then doing
intermediary work using simple code. I had never thought of it. I bet you can
do fun stuff like pipe it through imagemagick for fun transformations.

------
seanalltogether
Can anyone explain why rolling shutter is still necessary? Wouldn't current
electronics have the memory bandwidth needed to just sample the entire cmos
sensor at once?

~~~
zeta0134
You've just hit the nail on the head yourself: bandwidth.

Sampling the entire CMOS sensor at once is not really possible when you stop
to consider the sheer magnitude of bits required to represent a single still
image. The CMOS sensor only has a finite number of pins connecting it to the
logic board responsible for polling its state, so it makes sense to use an
addressing system to read the sensor data. This addressing system uses a small
number of addressing bits (each pin is usually just one bit) to move a sliding
window over a much larger amount of data. Regular computer memory works the
same way. This enables that data to be passed through to the camera in small
pieces. While this can be made rather fast with good engineering, it is not
instant, and the non-instantaneous nature produces the rolling shutter effect.

It's the physical limitation on the number of pins that creates the need for
the rolling shutter in modern CMOS design. Creating speedier memory access
routines can reduce the effect of the shutter, but not eliminate it.

A possible alternative approach would be to build some memory into the image
sensor, so that you can ask it to "lock" all the bits in place, and then read
those bits out at your logic board's leisure. That should effectively
eliminate the rolling shutter effect, (More or less, assuming your "freeze"
signal arrives at all the pixels more or less at the same time) at the
engineering expense of needing to add per-pixel memory to your image sensor,
and additional electronics to perform the lock. I'm no expert on the subject,
but I would not be surprised if more modern image sensors have some feature
for this purpose, especially those used in high speed cameras.

~~~
cashsterling
Don't confuse the timing of light acquisition with read-out at the overall
focal plane level.

Numerous CMOS, CCD, and other types of focal plane / 2D detectors have global
shutters where all pixels 'integrate photons or energetic particles' at the
same time for a set amount of time.

Read-out is performed in a rolling fashion... the data generally has to be
serialized for analog to digital conversion (although not always) and/or
transmission over a communication protocol or write to file.

A global shutter detector at the hardware level has settings for framerate and
light integration time (among other things). Readout time is more or less
constant for a particular camera sensor. So if integration requires say 20ms,
and readout takes 30 ms... you're looking at about 20 FPS image acquisition
speed.

Good overview: [http://www.red.com/learn/red-101/global-rolling-
shutter](http://www.red.com/learn/red-101/global-rolling-shutter)

~~~
fest
IIRC a typical implementation of electronic global shutter for CMOS sensors
requires one extra transistor per each pixel to "lock" the charge. As for why
it's not being done outside specialised sensors: not many people are concerned
about this problem, they are more concerned about other parameters of image
sensor.

------
vanous
This is great. Searching for an example of the rolling shutters PPM image
transformation in Python, anyone?

