
Sony Releases Stacked CMOS Sensor for Smartphones with 48 Effective Megapixels - okket
https://www.sony.net/SonyInfo/News/Press/201807/18-060E/index.html
======
foobar1962
While pixel density is good, there is no substitute for pixel size ie, larger
sensors. Hence the full-frame 35mm and larger sensors will always produce
better images. I have a Sony RX100 and its images are superb for a pocket
camera -- a combination of the 1 inch sensor and fine Weiss glass.

I studied photography before digital and the same was true then: larger
formats equalled higher quality particularly for landscape and scenes with
specular reflections. The reason was explained to me like this: there is a
tree with shiny leaves and these leaves have specular reflections (reflecting
the sun directly) so there are extremely small but very bright points of
light. These specular reflections hit the film/sensor and spread around a bit.
If the grains/pixels are small the highlights will spill into adjacent
grains/pixels and reduce sharpness.

~~~
WhitneyLand
Is it really true that larger pixels are better, or is it really only true
that sensing more photons is better?

Specifically, what is the reason multiple sensors with smaller pixels could
not perform as well or better if their combine total sensor area was larger?

The L16 is an example. It had issues, but they mostly seemed like
implementation problems rather than inherent limitations.

After all their challenges seeing that they still apparently raised 100MM in
capital, it would seem to suggest something better is possible.
[https://spot.light.co](https://spot.light.co)

Also they have bigger hardware that today’s smartphones would want, but in
principal I’ve never understood why smaller pixels are inherently worse as
opposed to total photons collected.

~~~
npunt
I forgot the specific term that describes this, but basically each pixel on a
sensor is going to have a threshold for whether it activates (much like a
neuron's action potential). Say its 20 photons. If it gets less than that, it
is unable to see anything at all. Meanwhile a larger pixel will see something
because it will be able to capture more than 20 photons in a given unit of
time. On the other hand, the aggregate of several smaller pixels across many
sensors will each fail to reach that threshold and thus see nothing.

The L16 has some low light and dynamic range problems, I believe owing to this
very situation.

~~~
dperfect
I believe the term you're looking for is reciprocity failure[1] (or
schwarzschild effect), though apparently it doesn't happen with digital
sensors - only film. Noise becomes an issue in similar situations, but it's
not affected by the number of photons hitting the sensor.

Correction: Shot noise[2] _is_ related to the number of photons hitting the
sensor.

[1]
[https://en.wikipedia.org/wiki/Reciprocity_(photography)#Reci...](https://en.wikipedia.org/wiki/Reciprocity_\(photography\)#Reciprocity_failure)

[2]
[https://en.wikipedia.org/wiki/Shot_noise](https://en.wikipedia.org/wiki/Shot_noise)

------
ivanech
The possibility of 4x more dynamic range (2 stops) is very exciting.
DPReview's writeup [1] mentions the possibility that each of the subpixels
could be set to different sensitivities and that each one's information could
contribute to the larger pixel's output, increasing its overall dynamic range
at the expense of resolution. This simple approach isn't practical with the
Bayer pattern because all adjacent pixels have different color filters.

EDIT: I actually found a little explainer on quad bayer on Sony's site [2] for
an industrial sensor from 2017. Instead of changing each subpixel's
sensitivity, they change how long the subpixel collects light.

[1] [https://www.dpreview.com/news/0249781150/sony-
imx586-smartph...](https://www.dpreview.com/news/0249781150/sony-
imx586-smartphone-sensor-comes-with-48mp-and-quad-bayer-design)

[2] [https://www.sony-
semicon.co.jp/products_en/new_pro/may_2017/...](https://www.sony-
semicon.co.jp/products_en/new_pro/may_2017/imx294cjk_e.html)

~~~
a012
It sounds similar to Foveon sensor

~~~
nomel
How so? Foveon used penetration depth to discern color, giving higher
resolution (and noise), which is very different than this.

------
userbinator
The fact that they have to explicitly state "effective" every time, along with
a footnote that vaguely explains "Based on image sensor effective pixel
specification method", strongly suggests that this _isn 't_ actually a 48MP
sensor.

...and indeed, the link in the other comment here
[https://news.ycombinator.com/item?id=17597939](https://news.ycombinator.com/item?id=17597939)
explains it best: "This requires processing to convert the 'Quad Bayer' data
into an approximation of what a 48MP Bayer sensor would have captured."

~~~
foobar1962
All single-sensors cameras are like this.

Each of the R-G-B sensitive photoreceptors in single-chip sensors are arranged
using a Bayer filter pattern which looks like a mosaic. The filter pattern is
50% green, 25% red and 25% blue.

The "pixels" of the final image needs to be calculated from this pattern
(called de-mosaicing or de-bayering). There is not a 1-to-1 correlation
between RGB pixels in the sensor and RBG pixels in the final image, and the
final image size may depend on different algorithms to remove artefacts caused
by the filter.

[https://en.wikipedia.org/wiki/Bayer_filter](https://en.wikipedia.org/wiki/Bayer_filter)

~~~
magicalhippo
Trying to capture some nice sunset photos while on a bike trip yesterday got
me thinking, with these high MP sensors could one use an augmented Bayer
filter to get higher dynamic range?

Instead of just the same RGB pixel filters all over, mix in pixel filters
which are darker (passes less light). Then use this info during de-Bayering to
construct a HDR image directly.

Maybe a very silly idea, but would have been fun to play around with (maybe
I'll write a simulator).

~~~
ipsum2
They do this on certain cameras, look up "Dual ISO". Effectively they make
every other line more sensitive, increasing dynamic range when they merge the
lines together. It's not a perfect solution, moire and other artifacts
increase, and you lose resolution. The trade-offs are often worth it,
especially for video.

~~~
magicalhippo
Thanks, that search term did indeed show some interesting hits.

------
hmd_imputer
I have already sold my Nikon camera and currently selling all my lenses to
permanently switch to Sony. As far as I know, Sony is the only company
currently doing real innovation in the field of image sensors.

~~~
renjimen
Are you talking about going from DSLR to mirrorless? If so, I'm also toying
with the same idea, but I'm unsure about when to switch.

DSLR tech has stagnated meaning they keep their value, so presently I can sell
with a small loss, while mirrorless tech is rapidly improving so today's
models will lose their value quickly - not a good investment. However, a lot
of people seem to be making the switch, which will drive down the value of
DSLRs over time, so switching too late means making a larger loss on my DSLR.

Not to mention the cost effectiveness of DSLRs vs. mirrorless cameras. From
what I've read, mirrorless are still behind.

I suppose I should wait for the release of a mirrorless model that rivals
similarly priced DSLRs at a time when the rate of improvement has lessened
somewhat to maintain its value.

~~~
pizza234
> while mirrorless tech is rapidly improving so today's models will lose their
> value quickly

It is improving very slowly, actually, by any standard of improvement. I have
an almost 6-years midrange MFT camera, which I wanted to replace with the
latest and greatest MFT, but things haven't changed in a significant way.

I've actually spent a holiday carrying around both the old camera and a
"latest and greatest" APS-C, and taking the same pictures with both. While
there was difference, it wasn't as large as one would expect from a ~5 years
timeframe. Of course, I must mention that I don't generally take pictures in
extremely challenging conditions (very low light, very fast focus speed
etc.etc.).

On top of that, the MFT market radically changed in the last years. Nowadays
the best sensors are in the top-of-line camers, which are very expensive (on
par with APS-C), and it seems they're not permeating down to the midrange
segment, as they used to do.

In other words, buying today a midrange MFT gives almost no noticeable
improvement compared to a 5 years old midrange MFT. Buying a top-of-line MFT
gives an advantage that is not as large as expected, while also being very
expensive.

I don't imply that using MFT doesn't make sense (it's my favourite format),
however, it's nowadays far from being revolutionary.

~~~
coldtea
> _It is improving very slowly, actually, by any standard of improvement. I
> have an almost 6-years midrange MFT camera, which I wanted to replace with
> the latest and greatest MFT, but things haven 't changed in a significant
> way._

Yeah, just 4K at 200mbps (GH5 and co) and 10bit color, 6MP snapshots from
video, 5 axis image stabilisation that's rock solid, sensor-shift 40+ MP
images, much better low light, and so on.

> _While there was difference, it wasn 't as large as one would expect from a
> ~5 years timeframe._

Analog cameras image quality didn't have ANY difference in a 5 or even 20
years timeframe, so there's that...

> _Nowadays the best sensors are in the top-of-line camers, which are very
> expensive (on par with APS-C), and it seems they 're not permeating down to
> the midrange segment, as they used to do._

Err, Panasonic for one has several cheap models with essentially the same
sensor as GH5.

> _I don 't imply that using MFT doesn't make sense (it's my favourite
> format), however, it's nowadays far from being revolutionary._

MFT does not exhaust mirroless (which is what the parent spoke about).

~~~
hydrox24
> Yeah, just 4K at 200mbps (GH5 and co) and 10bit color, 6MP snapshots from
> video

This is a good thing to point out, because video really has progressed
substantially in the last five years. But most people still can't make videos
like they can photos, and 10bit colour an high-bitrate 4K are both totally
useless to the average person, or even a enthusiast photographer.

~~~
coldtea
> _10bit colour an high-bitrate 4K are both totally useless to the average
> person_

Well, is clear 6400+ ISO (which we've got today) really useful to the average
person?

People have been snapping photos for 70+ years of film, where iso 400 was
really pushing it grain-wise...

------
jug
Hmm, I don’t think 48 MP is very suitable for smartphones. That sounds very
large and unwieldy with the often limited space (not only on phones but even
more so cloud backups). Who is asking for larger photos these days anyway?

The inproved DR is interesting though! Smartphones are currently very behind
APS-C and full frame sensors there since it’s traditionally been more or less
tied to sensor size.

~~~
frou_dh
Presumably the phone's image processing algorithms can make productive use of
extra sensor data without ultimately needing to persist a larger file to the
filesystem.

------
verytrivial
I would trade ALL the pixel above about 8MP for dozen or more stops of dynamic
range in the sensor instead of as a software hack.

------
lifeisstillgood
There was a point in the 90s when every upcycle in clock speed translates
directly to a payoff for software and users - that is moving to a new CPU
generation bought you faster response times or quicker databases

Somewhere a decade ago this kinda stopped for anything mere humans could
recognise - you could keep running your laptop or tower past the 18 month
upgrade cycle and not notice or care too much (pace gaming)

Cameras seem to be in the same position now - i click the button on my camera
phone and zoom in and am astounded - just astounded, at clarity.

I suspect the next driver will be like big data sucked up all those new cores,
something like 3d cameras will be where the new frontier for camera
improvements come from - not a demand for pictures that are better than most
humans can already tell

------
nashashmi
How is the 60 mp camera from Nokia different than this?

~~~
Synaesthesia
It’s a physically much larger sensor - this is the usual tiny sensor size
phones use.

------
_Codemonkeyism
More information:

[https://www.dpreview.com/news/0249781150/sony-
imx586-smartph...](https://www.dpreview.com/news/0249781150/sony-
imx586-smartphone-sensor-comes-with-48mp-and-quad-bayer-design)

------
Neil44
When was the last time you took a picture that didn’t come out how you wanted
it, and it was because you didn’t have enough megapixels?

~~~
kalleboo
If you read the description of the sensor, it actually sounds a lot more
interesting than just "moar pixels". It can use the adjacent same-colored
pixels at different sensitivities to either increase the dynamic range
(basically single-shot HDR), or join them together to increase low-light
sensitivity at a 1/4th the resolution.

------
haikuginger
I'm curious to see how the "Quad Bayer" mosaic works out. Other manufacturers
have tried novel filter patterns before, but nothing so far has really been
able to compete.

Essentially, almost all digital cameras today use a planar CMOS sensor with
alternating RGB-sensitive pixels, arrayed like so:

    
    
        RGRGRGRG
        GBGBGBGB
        RGRGRGRG
        GBGBGBGB
    

This pattern is not perfect, but is highly effective. Luma (color-independent)
resolution is essentially equivalent to the actual number of pixels, while
chroma (color-dependent) resolution is only slightly less - we essentially get
one "real" point of color information at each intersection of four color
pixels, because at each of those intersections we have one red, one blue, and
two green pixels.

In other words, "the luma information we gather for a given color on a pixel
of that color is immediately relevant to the effective pixel composed by it
and its adjacent neighbors at each of its four corners". In this 8x8 Bayer
pattern pixel grid with 64 real pixels, we get 49 effective chroma pixels; one
for each intersection of 4 physical pixels.

In comparison, here's the pattern for "Quad Bayer":

    
    
        RRGGRRGG
        RRGGRRGG
        GGBBGGBB
        GGBBGGBB
    

I'm concerned that chroma resolution and overall color accuracy will be much
lower with this pattern. Essentially, with the original Bayer demosaicing, you
only need to sample from the four color pixels adjacent to each corner in
order to get a bit of the three channels, and the pattern gives equal weight
to both red and blue, while providing extra accuracy in the green channel that
human vision is most sensitive to.

In comparison, as far as I can tell, a single "effective pixel" (one with
information on all three channels) using the Quad Bayer pattern has to be made
up of data from nine individual pixels. Additionally, when an effective pixel
is centered on an actual pixel with either red or blue filters, that color is
relatively dominant in the pixels considered - it'll be equally weighted with
the green channel, and the opposing color will only make up 1/9 of the total
signal composing that effective pixel. Effective pixels centered on green
pixels will give equal weight to red and blue, with slightly over 5/9 of the
weight given to the green channel.

Granted, the sensor should still be able to produce a full 48MP of luma
resolution, but chroma detail will be much more "smeared" because of the wide
area that has to be considered to get a full color pixel, and the more
substantial overlap of that full color pixel with other full color pixels.
Color accuracy will likely also be lower, because in effective pixels centered
on red and blue pixels, only a single pixel of the opposing color will be
used, which means that any noise in that channel will have an outsized impact
on the overall color.

What this boils down to is that, when used as a 48MP sensor, this sensor will
have entirely different imaging characteristics than a traditional Bayer
imager, and that those characteristics will be highly dependent on how the
output of this sensor is processed - which will be interesting in a world full
of software highly optimized to demosaic Bayer-pattern images.

What's slightly more interesting is the high-sensitivity 12MP mode.
Essentially, it's an attempt to reduce the impact of random noise in the image
by adding together four pixels of each channel to produce a "superpixel" less
impacted by noise overall. These superpixels can then be processed in a
standard Bayer pattern as a 12MP effective image.

Thinking about it overall, though, I become more and more confused. In both of
these modes, this pattern doesn't give us anything, really, that we can't
already do using a Bayer filter.

Let B represent a sensor using a standard Bayer filter pattern, and let Q
represent a sensor using this "Quad Bayer" pattern, where each of these
patterns have a red pixel in the top-left corner.

Let any given effective pixel be represented by a 3-tuple of the form (R, G,
B), where R, G, and B are the number of physical pixels sensitive to each of
the red, green, and blue channels which compose that effective pixel.

Let f(p, w, h, s, i) be a function returning a two-dimensional matrix of all
the effective pixels produced by a matrix of physical RGB pixels, laid out in
pattern p, with actual pixel width and height w and h, where an effective
pixel measures s actual pixels horizontally and vertically, and where an
offset of i actual pixels in either vertical or horizontal directions produces
the "next" pixel in that direction.

Thus, our standard Bayer pattern produces the following:

    
    
        f(B, 4, 4, 2, 1) =>
        (
            ((1,2,1),(1,2,1),(1,2,1)),
            ((1,2,1),(1,2,1),(1,2,1)),
            ((1,2,1),(1,2,1),(1,2,1))
        )
    

The 9-pixel-effective-pixel Quad Bayer pattern produces this:

    
    
        f(Q, 4, 4, 3, 1) =>
        (
            ((4,4,1),(2,5,2)),
            ((2,5,2),(1,4,4))
        )
    

Note that there are fewer effective pixels for the same total number of pixels
- that's okay, though, because the number of effective pixels approaches the
number of total pixels as the sensor scales in the X and Y dimensions - this
very small hypothetical sensor doesn't benefit from that scale yet.

You can also see that each effective pixel is composed of a larger number of
physical pixels - there's a tradeoff there, in that this means that overall,
noise should have a smaller impact on the value of a given pixel, but there's
a loss of resolution because those pixels are spread over a wider area.

This raises the question, "what if we do a 9-pixel effective pixel on a
standard Bayer pattern?" Well, we get this:

    
    
        f(B, 4, 4, 3, 1) =>
        (
            ((4,4,1),(2,5,2)),
            ((2,5,2),(1,4,4))
        )
    

Interestingly, while the exact arrangements of the different color channels
within the effective pixels are different, the total number of pixels of each
channel remains _completely identical_ , meaning that any given effective
pixel should have identical noise characteristics to the Quad Bayer pattern.
In fact, it's arguable that the Bayer pattern is better, because the color
physical pixels are more evenly distributed around the effective pixel.

What if we do the high-sensitivity superpixel sampling? For the Quad Bayer
pattern, it looks like this:

    
    
        f(Q, 8, 8, 4, 2) =>
        (
            ((4,8,4),(4,8,4),(4,8,4)),
            ((4,8,4),(4,8,4),(4,8,4)),
            ((4,8,4),(4,8,4),(4,8,4))
        )
    

And for the standard Bayer, like this:

    
    
        f(B, 8, 8, 4, 2) =>
        (
            ((4,8,4),(4,8,4),(4,8,4)),
            ((4,8,4),(4,8,4),(4,8,4)),
            ((4,8,4),(4,8,4),(4,8,4))
        )
    

Again, sampling in a similar pattern gives the same overall result. So, all
else being equal, I'm not sure it makes sense.

Of course, there is the possibility that all else is not equal. Having
multiple adjacent pixels of the same color could enable consolidating the
signals of those pixels together earlier on in the image processing pipeline
into an actual lower-resolution standard-Bayer signal. That could actually
have real benefits if that early-stage signal combination results in a greater
signal amplitude that drowns out noise.

Basically, this has all been kind of stream-of-consciousness and much longer
than I originally planned, but here's the Cliff Notes from what I can tell.

In comparison to a Bayer sensor of the same pixel resolution...

Pros:

\- (If implemented to take advantage, possibly) Ability to act as unified
large pixels, increasing SNR at lower resolution settings

Cons:

\- Less-fine maximum chroma resolution when all pixels are active

------
Aoyagi
Not that different from my ancient Nokia 808 PW or the Nokia 1020, and I
presume the low light performance will be worse as well.

~~~
coldtea
Why would a 2018 sensor from the low-light kind (Sony) have worse "low light
performance" than a 2013 one (of equal number of megapixels too)? Brand
loyalty?

~~~
Aoyagi
I'm guessing that from the pixel size (1.12µm vs 0.8 μm).

------
DrStalker
I'd be much happier with 5 high quality megapixels instead of pushing
megapixel count as an indicator of camera quality.

~~~
zokier
5MP is just very small these days. 3840x2880 is practically the minimum you
want to go (to be able to capture 4k shots) and that is already 11+MP.

For 4k video there is the additional challenge that if your sensor is not
exactly the right size then how are you going to sub-sample it? I bet it is
not a complete coincidence that the quoted 48MP figure is pretty close to 4x
above 11MP (=double the horizontal/vertical resolution); I imagine they are
planning to sample every other pixel over 8K area, which probably brings in
all sorts of benefits.

~~~
coldtea
> _5MP is just very small these days._

Very small for what? You can print 6MP in A3 size, and nobody would even
notice...

~~~
kalleboo
4K display. Who prints these days?

~~~
coldtea
> _Who prints these days?_

People who care about photography.

Who watches their photos on a 4K display and pixel-peeps that they're 20% less
that its full resolution? (4K is about 8MP -- but you need to go to 2MP
(square root of 8MP) to have half the visual resolution. 5-6MP to 8MP is much
closer that it looks on paper, as the change is not linear (pixels cover an
area, not a line).

------
mtgx
Android discussion on this yesterday, and why it may not be as good as it
seems. As others have said, you can lose a lot with that pixel density:

[https://www.reddit.com/r/Android/comments/915a1r/sony_releas...](https://www.reddit.com/r/Android/comments/915a1r/sony_releases_stacked_cmos_image_sensor_with_48/)

------
nottorp
Do the megapixels matter if they're paired with crap lenses? Don't think so.

~~~
chaoticmass
Does the THD matter on an amp if the speakers are crap? Do good brakes matter
if the tires are bald?

------
amarant
whats really interesting here is the 4 times greater dynamic range, that could
actually make a visible difference to your pictures! increasing the pixelcount
is largely irrelevant for smartphones, and has been for a while.

