
Sony Builds AI into a CMOS Image Sensor - anandaverma18
https://spectrum.ieee.org/view-from-the-valley/sensors/imagers/sony-builds-ai-into-a-cmos-image-sensor
======
rapjr9
This could improve privacy if all the processing remains internal, however
there are several ways it could reduce privacy and create problems depending
on what you do with the AI output. For example, the AI output could deny
access to buildings if it detects a person is a child (good for industrial
settings, not so good if you're a short adult) or it could deny access if it
detects a person is white or female (bad for society). The video data might
all be processed internally, but if the chip also has a video output the AI
output could simply be used to decide what video to save. Or the AI output
could trigger recording from an external video camera. So overall, the privacy
aspects of this sensor seem pretty weak; it would have to be used in a
properly designed system or else this privacy protection could be easily
bypassed.

I've worked with similar systems that used local AI to process speech (to
determine things like turn taking in conversations) and there was a claim that
the system enhanced privacy because no speech was ever recorded, but in truth
it would have been easy to compromise that privacy protection. If the ability
to record or export video is not part of the sensor design, then it would be
difficult for anyone to alter the chip to record video, but how do you verify
that? The chip could have a secret "test" mode where it exports video and AI
parameters for troubleshooting. You'd have to trust Sony, which might be
reasonable in some circumstances, but not in others. The same is currently
true for a variety of phones with "smart" features associated with the camera.
It may seem like the phone will only unlock for you when it sees you, but what
if it also secretly unlocks when presented with a specific QR code? How would
you ever know?

~~~
jcims
Yep, I think the world would benefit much more if they just put the ability to
do digital signatures in the CMOS. People want to do touchups, they just add a
link to the original signed image in the EXIF do folks can take a look for
themselves.

Definitely not going to stop state actors, but might help with some of the
noise from doctored images/video.

~~~
enigmo
unlikely to be useful unless you're shooting in RAW and need evidence that a
specific sensor took the shot. phones and cameras do a lot of post processing
before an image hits the disk and the original is discarded.

~~~
jcims
Eh, the purpose would be to demonstrate that a camera took the shot. The fact
that it id’s which one specifically isn’t particularly material but might be a
little useful in some circumstances.

You wouldn’t need to publish the unprocessed photo, just keep it in escrow.

~~~
londons_explore
Nothing stops you editing the photo, printing it out, and taking a photo of
the printout.

The new photo would still get signed, and look original.

And if you did this in the right conditions, it wouldn't be possible to tell
it was a photo of a print.

~~~
quonn
Since all the metadata (exposure time, shutter speed and so on) in the photo
is signed as well, it would be possible to tell.

------
unnouinceput
Quote 1: "They can then select only relevant images, or parts of images, to
send on to cloud-based systems or local hubs. "

Quote 2: "..enhanced privacy.."

Syntax error.

------
supernova87a
I have a beginner's question about sensors and chips, for the experts out
there:

When chip hardware gets faster, or you read about such developments as
cramming more and more functionality onto the sensor, does that mean the
package could draw less and less power and do the same functions as previously
for less energy?

Suppose this chip's function were (as in the article) to image a scene and
decide whether to send the image over wireless to a central monitoring
station.

As the chips get more capable, does that mean less power is used compared to
before, to do the same function? Or are there overheads that dominate the
power consumption for such under-utilization of a chip? Is there a rapid
falloff in benefit of the "advanced-ness" of a chip for such applications
where you really don't need such sophistication, and rather have energy-
conserving, simple design? Will this really lead to months more lifetime for a
remote sensor powered by battery?

(sorry if some of my nomenclature is imprecise, but hopefully you get the idea
of the question)

~~~
alok-g
If comparing for the _same_ functionality, power consumption will typically
reduce unless it is dominated ny other components. Both positive and negative
examples below to explain:

* An audio amplifier: Power consumption is ultimately dictated by the energy conversion to sound and the conversion efficiency of the speaker. The system will consume at least that much irrespective of the technology developments. In practice however, the efficiency of the amplifier also comes into play. The static power drawn by an efficient amplifier would be much less than the above basics for creating sound, and so going to higher technology nodes would make much less percentage difference.

* Computations in a digital circuit: Power consumption in these is sometimes dominated by communication (interconnect; bus) like between processor and memory. If the bus is internal to the chip, advancing to a new technology node would typically result in power reduction.

* If the power is dominated by computations alone, there would surely be power reduction, again assuming the functionality is held constant. Circuits have static leakage power too, which happens irrespective of the computations, however, it is still usually lesser than dynamic power by design.

Typically however, more and more functionality is frequently added like in
this case of computer vision added to the sensors, which results in an overall
increased power consumption to an extent that the customers will accept (e.g.,
battery life asked for).

Back to this specific case of computer vision on the "edge":

Having the same functionality achieved by having computer vision on the sensor
itself vs. the cloud, can result in orders of magnitude savings in power.
There is a lot of power associated with transferring the data back and forth.
Even if computations are handled by a processor on the same board as the
sensor, significant power is spent as compared to when the communication lines
are avoided.

Related:
[https://news.ycombinator.com/item?id=23202839](https://news.ycombinator.com/item?id=23202839)

~~~
supernova87a
Thanks!

------
spiritplumber
Reminds me a little bit of the chip in the game boy camera - it tried to do
edge detection on chip, like vertebrate eyes do.

[https://gbdev.gg8.se/wiki/articles/Gameboy_Camera](https://gbdev.gg8.se/wiki/articles/Gameboy_Camera)

~~~
userbinator
Here's the actual description of how it works:

[https://gbdev.gg8.se/wiki/articles/Mitsubishi_M64282FP#About...](https://gbdev.gg8.se/wiki/articles/Mitsubishi_M64282FP#About_the_image_processing_functions)

It's just a very restricted convolution.

------
alok-g
>> Hanson says that while other organizations have similar technology in
development, Sony is the first to ship devices to customers.

Following are some links to such similar technology developments for those
interested. Of these, [4-5] cover a lot more.

PS: I was the technology lead for [1] below, which I believe is a pioneering
work which kickstarted this.

[1] Qualcomm Wants Your Smartphone to Have Energy-Efficient Eyes
[https://www.technologyreview.com/2017/03/29/243161/qualcomm-...](https://www.technologyreview.com/2017/03/29/243161/qualcomm-
wants-your-smartphone-to-have-energy-efficient-eyes/)

[2] emza Visual Sense - IoT Visual Sensors [https://www.emza-
vs.com/](https://www.emza-vs.com/)

[3] Why the Future of Machine Learning is Tiny
[https://www.oreilly.com/library/view/tinyml/9781492052036/](https://www.oreilly.com/library/view/tinyml/9781492052036/)

[4] The tinyML Summit
[https://www.tinyml.org/summit/](https://www.tinyml.org/summit/)

[5] TinyML: Machine Learning with TensorFlow Lite on Arduino and Ultra-Low-
Power Microcontrollers [https://www.amazon.com/TinyML-Learning-TensorFlow-
Ultra-Low-...](https://www.amazon.com/TinyML-Learning-TensorFlow-Ultra-Low-
Power-Microcontrollers-ebook/dp/B082TY3SX7/)

------
fmakunbound
Hi, AI nub here. What’s the AI part of this?

~~~
tanilama
Nothing AI about this. This is just a neural network model on chip.

AI is the keyword to get clicks.

~~~
kolinko
You’re confusing AI with machine learning.

------
centimeter
The proposed use cases in the article have zero benefits over a normal CMOS
sensor and a normal processor.

~~~
andromeduck
Aside from power, packaging and off package bandwidth you mean.

