First, why would you want lightfields? Computational photography -> variable boken (e.g. fix the bokeh for different depths, make it smoother, etc), depth info (extremely valuable for VFX), having some variability in your camera position - so being able to add camera shake more realistically. There's a bunch of other cool things you could do.
I've only taken a look at existing lightfield solutions briefly, but this looks cool. It's novel because it seems to only use the bare possible minimum number of ray samples to be called a lightfield, and makes up for it with heavy compute (even heavier than a typical lightfield).
This approach has interesting tradeoffs. A typical lightfield camera typically uses, say, 25 ray samples. This number of ray samples isn't enough to avoid compute, but cuts the resolution by a lot (25x in our case - instead of having 50 megapixels, you're at 2 megapixels). A typical implementation uses a micro-lens array in front of the camera sensor, and that means that there's wasted overhead due to tiling circles leaving a bunch of area empty.
Their proposed approach "only" leaves you with 9x less resolution (which is cool), which they try to reconstruct / upscale back to 2x less resolution (which is also cool and should be workable).
Judging from the samples, the iPhone with ML-based depth sensing and Lidar also seems to do a much better job. As will most of the latest ML models available.
Light fields will be extremely conducive for building ML models around. It'll be possible to generate incredibly rich and realistic 3D scenes with the additional information light fields convey. Possibly some downsampled number of rays to make compute easier.
Multiple view scene constructions will make for incredible filmmaking that can be tweaked in post. No shot, reverse-shot. No cheating the camera. (That's assuming we don't switch from optics to mocap + rendering outright.)
Luckily, or maybe unluckily, for me the camera ended up taking terrible pictures and no images of any consequence were taken with it.
The algorithm for processing plenoptic images is also well-known and this company is not the first to do it. Someone will come up with a free tool for dealing with these images.
Say you have the 255 shades of gray in RBG, and you want to spread them evenly over the distances of 1-5m. That would give you a 1-step increase in brightness for every 1.6cm or so, which happens to be pretty close to what I believe these folds‘ magnitude might be. I’m not entirely sure how prominent the difference would be to the naked eye. IIRC, the MPAA considers even 50 to be plenty.
I‘m leaving out lots of details (pun not intended, but appreciated): you’d spread your resolution logarithmically, for example, not linear. And, of course, you could work with more than the resolution of 255. But it’s a different domain and I believe some intuitions are off if you compare it with the x and y dimensions.
Using parallax to calculate depth undoubtedly has principal limitations in far away details, and mapping to an 8-bit depth buffer is another very reductive step in that regard. (regardless, I'd expect even the folds to show up, at least qualitatively, if I'd looked at an 8-bit rendering of a 3D scene's z-buffer; the gradient contour steps are clearly visible, and dense, yet fail to follow the folds, indicating that the underlying depth data simply doesn't track them at all)
Let's take the sleeves then -- clearly a large difference in relative depth, yet they blend into the rest of the garment. My impression is very much that of standard depth reconstruction "AI" that more or less guesses depths of a few image features, and does some DNN magic around where to blend smoothly and where to allow discontinuities, with the usual "weirdness" around features that don't quite fit the training sets.
Possibly all we can get out of this "parallax" method of depth reconstruction isn't a whole lot better than just single image deep learning, which would not surprise me, as it ultimately relies on the same approach for correctly recognizing and mapping image features across the 9 constituent images in the first place, vs. a true lightfield sensor that captures the actual direction of incoming light.
I think this design will require a lot more cpu for post processing though.
Recently my son was enthusiastic about anaglyph images viewed with red-cyan glasses. These are much better than what they had when I was a kid because the cyan filter passes both the green and blue channels. You can use whatever color you like as long as it isn't red or cyan.
I got enthusiastic too.
We found out that you can't (in general) make a good anaglyph image with a 2-d image and a height map because the left eye and the right eye can see around objects so the 2-d image usually is missing pixels that the left and right eye would see.
You could reproject an anime image if you had the cels; you could reproject the image for the left and right eyes by simulating the multiplane camera.
I am just starting to take 3-d photographs by moving my camera and that way I get all the pixels for the left and right eye. That lens might be better for taking portraits at very close range but I don't know if it would beat my current lenses. Certainly I could get a really good telephoto lens for that price today whereas I don't even know if I'll be taking 3-d photos a year from now when that lens would arrive.
You can use stereo pairs to calculate depth, and in this case, you only lose half your horizontal resolution (vs. losing significant horizontal and vertical resolution on the K lens)
Also, they effectively split the resolution by 3.x on each axis, because they are multiplexing a 3x3 image grid onto the same sensor. That means a $50k RED MONSTRO isn't "good enough" anymore to get 4K output.
The hardware tech is different too, Raytrix seems to be using a microlens setup (like most other lightfield cameras). A microlens setup also reduces resolution fwiw.
If I remember correctly, Lytro went bankrupt in the period between when Raytrix filed their patent and were granted it, so they "got lucky".
If I understand the K|Lens implementation correctly, they stuck a mirror-based kaleidoscope in the middle of the lens. I don't know if there's anything extra on top of that.
That said, I wonder what KLens has been doing from 2014 (when their patent EP 2 533 199 B1 was issued) until now.
The still of the motorcycle rider has obvious motion blur, which immediately raises the question of how that can work with a depth channel. Can you access the depth data in some other way than just a Zdepth channel? If not, there are some serious limitations to what you can do with it.
In the video example, the depth channel flickers a lot. This seems to indicate the depth is some kind of calculated estimate, and not very reliable.
I like it that they quote deep learning algorithms
Where all those lenses the same though?
With the Lytro camera I believe that could 'refocus' an image, but didn't think it could obtain 3D information? If that's the case, could anyone guess how 3D information is obtained from this camera.
It's just 9 images from 9 slightly different offsets along with some clever post-processing. (Although maybe the difference is one of degree rather than kind - the Lytro is "micro-lens array" so maybe the question becomes "how many images do you need to be a light field image?)
It explains the specifics of how light field cameras work quite well, but doesn't go too deep into light fields.
For more of an overview of light fields in general, I can recommend this video:
My answer to your question would be that light field cameras sample discrete perspectives from a 4D light field. These samples can either be 1) combined directly (requires a very high view density) 2) interpolated to try to recover the continuous 4D light field function (this is an active area of research), 3) downsampled to volumetric 3D (Google's approach with "Welcome to light field" etc.), or 4) downsampled to 2D + depth (a depth map)
Each of these use different methods.
Want to use an iPhone 12/13 to make models? https://youtu.be/0F3uFeqFOOw
German news article about it: https://www.golem.de/news/k-lens-one-erstes-lichtfeldobjekti...