I've often wondered, if there are multiple different prints of an older film available, they could be averaged or something using modern ML or computational statistics. I get the sense that "unsupervised" methods are underutilized a bit in film and audio restoration but I know next to nothing about this area.
Yeah, there are tools that do this. I know nothing about this stuff, but a recent Tech Tangents video had him combining multiple degraded prints of Contact. It's the final segment and the tool is called Id-disc-stacker. You need a lot of time, space and processing power.
I've seen this done for vinyl discs. You play several copies of the same old disc. The noise defects appear in different places, and can be easily removed by merging the aligned the tracks. Even with just two discs, the output result is spectacular.
It was a project by some signal processing students in my lab. As far as I know, they didn't get to publish anything besides writing a technical report; and the code was a matlab thing that just ran on the computer of one of them... But their presentation was excellent! They brought their vinyls, the turntable, and did a live demo of the whole thing running.
I don't know, releasing a sellable product from an algorithm is a huge undertaking, probably harder and definitely less interesting than developing the algorithm itself. This was just a three-month work project by some smart students, who afterwards got lucrative jobs working in industry.
In more practical terms, people who actually listen to vinyls get to appreciate the "cracks", and they wouldn't really see the point in such a product. Major studios have surely developed similar tools internally. The method is probably a scientific curiosity, mostly of educative interest, like observing the helium in the sun atmosphere with a hand-made spectometer built with an old CD and a cereal box.
"This is an experiment [...] soundtrack was being read in the same way as the picture is – stopped 24 times per second? Would this be the ultimate flutter distortion?"
That bothered me like crazy - how did they scan the audio whilst the frame was physically stopped? Then I realized how it was done: the soundtrack of frame A is stored in frame B, where frame B is far enough away that it has constant velocity.
How does it have constant velocity? Is there a physical "buffer" for the stopped frame (ie slack in the reel)? Otherwise you can't stop one part of the reel while the other is still traveling.
So the frame being projected has to stop. Otherwise you'll get blurring unless your shutter is very fast, but then you'd burn out the film from the intensity of the light.
The film is stopped.
But, of course, you cant start-stop a 15 lb spinning reel 24 times a second millions -> itll all fall apart.
So the feed is a simple rotation at constant velocity (linear? Angular?). Then a piece of film ten to twenty (?) frames long gets put into a buffer (an empty box) and a mechanism akin to a sowing machine single steps through each individual frame.
Now the stepping mechanism only has to start-stop ten frames worth of film, a negligible amount of mass compared to the mechanism itself and the projector can last a long time of heavy use with simple maintenance.
That is pretty neat, is it on both sides of the lens (entry and exit)? The synchronisation mechanism between all these things also sounds very interesting.
I dunno! I know nothing about projectors except what I dug out when the sound issue started bothering me. I imagine the buffer on the exit is just slack.
The sync though is actually pretty easy, at least compared to computers and electronics - shafts and gears ensure synchronization with little effort (or you'd brake a shaft!)
Yes, if you look at a simple projector you can see it: there's a little half-loop of film before and after the shutter/lens assembly and you can see it vibrate as it buffers between the intermittent motion at the shutter and the smooth motion in the rest of the film path.
A little flutter was an unavoidable part of our audio for a very long time.
I think producers sometimes add a little flutter even to modern music productions to make them sound less digital. At least that is my assumption from the number plugins that are available for that purpose and from the obsession producers and mixing engineers have with vintage tape machines.
I never understand why playing at wrong FPS for these old-time footage is a thing. It was so obviously wrong.
(By that I meant in "professional" settings, like a commercial release. You see people stretching 4:3 videos to 16:9 all the time on YouTube, Tiktok etc. but at least Warner Bros. isn't going to do that. However the FPS incorrectness used to be prevalent even in the industry.)
> I never understand why playing at wrong FPS for these old-time footage is a thing. It was so obviously wrong.
Old-time footage isn't always at 24 fps, but once the film industry standardized hard on 24 fps, not standard film becomes very hard to use correctly, even in a projection environment --- projectors were engineered around 24 fps all the time, and likely can't be adjusted.
Tv of course was fixed at 60 fields per second (or close enough) or 50 in Europe. Even today, a US market tv is going to do terrible things with a 50 fps signal (although I gather EU market tvs work at 50 and 60 and probably 24). If your old-time film was shot at 22.3 fps, you don't have good options to show it, except on a computer with a VRR display, and even then, I don't know if the whole chain will actually be fine with it. Options are: pretend it is 24 fps and it plays a bit fast and you can use your 24fps pipeline; show some frames twice and have nasty judder; interpolate 24 fps from the data you have which is kind of terrible and only recently a real option.
Modern TVs in any market will take 24 (23.976) hz, 50 hz, and 60 (59.94) hz. It's been pretty common to support all common refresh rates since displays went all-digital.
The vast majority of midrange and better TVs can also spot 24 fps content being sent in a 60 hz signal and render it properly.
Most TVs however will not look good with 50 fps content in a 60 hz signal and you will want to set your output device to adjust its refresh rate based on the frame rate of the content being played. Set top streaming boxes actually worth using have this option, but it is usually disabled by default.
The way movies used to be shown on TV was the movie was projected on a screen and the TV camera was pointed at the screen. The "interpolation" was simply the analog aspect of the TV camera tube.
Low tech, simple, and effective.
Some of my old family movies were converted to VHS that way.
It could invert negative film to positive, and it could color correct scene by scene by looking at correction codes or counting frame numbers, then applying the correct color correction.
Later, broadcast TV started using flying spot scanners instead of just a video camera in front of a film projector. ²
And going the other way, before the introduction of videotape, the only way to store TV programs was to use a kinescope to produce a film recording of the broadcast:
Off topic, but what's the difference between the term jitter and judder?
I've tried to google it multiple times, but every time, I can only find various forum posts which are all over the place (if not straight up contradict to each other).
My understanding is that jitter refers to essentially stochastic variation of an ideally isochronous/periodic process or signal. You might say "the event is supposed to occur exactly every 1 second; in practice we observed jitter of +/- 50 milliseconds," referring to the RMS (1-standard-deviation estimate) of a sample of inter-event durations. Wikipedia has a nice article: https://en.wikipedia.org/wiki/Jitter
In the context of video, I think the most common use of "judder" is about the deterministic and periodic variation in timing that occurs when content of one frame rate is shown on a different-rate display. The most common situation is a "3:2 pulldown," where 24 frame-per-second film content is adapted to a 60 field-per-second or frame-per-second video signal. This is done by repeating one film frame for 3 video fields/frames (so 3/60 seconds), and then the next film frame for 2 video fields/frames, then the next one for 3, then 2, then 3, then 2, etc. (2/5 = 24/60 so it works out on average.) That repeating variation in frame duration or number of repetitions is seen as "judder." With a cinema projector or recent TV, you don't have this; the frames are shown 1/24 s apart and with an equal number of flashes each. But on an older 60 Hz TV, you'll have judder.
(I've also seen "judder" occasionally used to refer to the stuttery motion that comes from showing 24 fps content on a sample-and-hold display, like an OLED [without black-frame insertion] or a "good" LCD that just shows each frame with constant brightness for the entire 1/24 s and then almost-instantly switches to the next frame. But I don't think this is the correct usage.)
Adding on, jitter is something you can address with better precision (various mechanical and later electronic means were devised to ensure a precise time between frames during recording and projection to reduce the jitter between frames in a film) or in modern times, larger buffers (if you're streaming video and packet delays vary by 50 ms, a small buffer will allow for even playback). Jitter buffers improve perceived quality at the expense of delay and memory use, but for recorded video and audio, delay isn't a big deal.
On the other hand, judder is often due to a limitation of the system. Fixed frequency displays can't display off frequency content without compromises. And TV broadcasting was tied to fixed frequency displays. NTSC had no hope of 24fps support, sending two fields per frame at 48Hz or three fields per frame at 72Hz would both be way too far from the spec frequency of 60 Hz. I wonder if PAL@48Hz could have worked though --- 4% off spec is a lot, but also not that much.
And bringing it back to old time film that's not at 24 fps, that's definitely more hopeless.
ATSC and I presume DVB can broadcast content as 24fps, and the decoder will do the 3:2 pulldown as needed, but variable rate displays can show 24 fps (it's also usually not too hard to detect and invert 3:2 pulldown, at the cost of adding a few frames of delay). I don't think modern broadcasting does better for a 22fps old-timey film than analog broadcasting did though; it's still not something the system was built for, and you're still going to have off speed playback or a pattern of some frames shown more times than others (which is my essential definition of judder)
From context I would guess that judder is a global effect (e.g. when the film pauses and then suddenly jerks forward playing), and that jitter is more local (e.g. to do with the movements of the actual characters on screen moving with sudden velocity)
Well, tv stations these days speed up the playback of older shows by a few percent so they can fit more commercials in than the original creator provided time for. I’m so glad I haven’t payed for cable in a very long time.
If you're lucky they speed up the playback. If you're not, they cut to commercial on a schedule, damn the content. Doesn't matter if you miss the punchline of the joke that was being setup this whole time.
Of course, it's worse with online tv with ad breaks. The online ads cut in 2-5 seconds before or after the program break, so it interrupts the content, and if you skip back to try it again, they play another ad, also inserted incorrectly. To make it even worse, the ads are usually at a different framerate and resolution from the content, so you get a new resync and a new hdcp handshake, so there goes another few seconds of content; unless you set your player to fixed resolution and framerate which isn't ideal.
I'm in prime circumstances to be a couch potato in front of the tv, but it's so damn frustrating, it's not worth it. You can't even channel surf, because tuning a new channel takes too long, so you've got to plan your changes.
How do you suggest taking 24 FPS and turning it to 29.976 NTSC? A lot of times, they would just convert it 1:1, so the 24 frames would go by in a 24/~30th of a second, so it looked faster. Further, a lot of old, old film was actually meant to be played at slower FPSes than 24, I think 8mm was like 18 FPS.
Today, we do key-framing and other tricks to make transfers from different FPSes easy and imperceptible (plus with digital stuff we can playback at any FPS), but back in the 80's and 90's the tech was a lot less advanced to turn a film image into an analog or digital format, and in previous decades you didn't even have analog or digital, you just had to use 1:1 frame transfers to get onto a larger or smaller film format.
> How do you suggest taking 24 FPS and turning it to 29.976 NTSC?
3:2 pulldown, which has been done for decades.
You can also just duplicate frames (un-evenly). It will introduce jitter but most of people aren't very sensitive to that, and it's better than playing at wrong speed.
As professionals they should be shooting at 29.976 fps if that is the fps the end product is supposed to be. Messing up and using the wrong framerate when recording is for amateurs.
The end product is supposed to be 24fps for a cinema. But then a TV company wants to show the already-existing film. It needs to be solved.
In PAL countries, 24fps movies are often played a bit fast at 25fps on TV so that each frame fits nicely in the 50hz signal. It's not a "Squeeze more adverts in" thing, it's the best way to show the film, otherwise you'd have to introduce an extra frame every second, which would look jarring.
Undercranking was generally used for dramatic effect, but I definitely have read about it being used to allow longer takes with a fixed film reel length - if you undercrank by 10%, you get 10% more time before you have to stop and reload.
It's even better due to the way percentages work - if you undercrank by 10% you get a bit over 11% more time (after 100% of the original time, you still have 10% of the reel left, but you keep undercranking, so by the end of the 10% extra time, you still have 1% of the reel left, etc.).
I've noticed a lot of jitter in the video, even in "stabilized" versions. It seems like there was jitter in how the cels were placed on the backgrounds, in addition to jitter of the whole frame. It's not so easy to stabilize it out. It would be cool to clean that up too with modern techniques.
More speculatively it seems to me like AI is approaching the level of image understanding where it could draw new in between frames in a much, much smarter way than typical terrible frame interpolation. That would be a great project and potentially commercially valuable. Steamboat Willie is animated at 24 FPS but it's common for animated TV shows to be done at 8 FPS and they could really benefit.
It doesn't look to me like Steamboat Willie is animated at anything close to 24fps. I couldn't step frames, but just look at the video, the animation is jerky. (Not worse than animation usually is, but it's not 24fps.)
very interesting indeed, reminds me of some of the work done by Jamie Howarth and co with the 'Plangent Process' which uses the bias tone recorded by analogue tape machines to correct flutter on high-sample-rate transfers from tape...
In what sense? Higher resolution formats provide more true-to-life fidelity, but that's only more true-to-nonsense when the additional information is produced from extrapolation.
