
Pixel Visual Core: Google’s first custom-designed co-processor - dazbradbury
https://www.blog.google/products/pixel/pixel-visual-core-image-processing-and-machine-learning-pixel-2/
======
dharma1
In addition to still photos, would love to see exposure fusion (or HDR+ as
Google calls it) in mobile video too. RED Epic and Scarlet cameras do this
with 2 exposures per frame and call it HDRx, I've found it adds about 4 stops
of dynamic range in grading. RED doesn't do any fancy anti-ghosting stuff
though like HDR+. Magic Lantern firmware for Canon dSLR's also does something
similar with dual ISO frames.

This together with 10bit video profile for h265 (think phone hardware is
capable now but haven't seen any phones with 10bit video yet) would make
mobile video a whole lot better, right now it frankly sucks for anything with
high contrast or dynamic range.

------
sandGorgon
This is pretty much Tensorflow on metal. Considering that tensorflow works
pretty well on most mobile phone GPU, im betting this feature can be turned on
for most phones if Google wants.

I would love to see the apples to apples comparison of the Pixel tensorflow
computation running on a vanilla snapdragon GPU vs this custom core.

Also will be interesting to know why halide and not opencv.

~~~
blt
Halide and OpenCV are not comparable. Halide is mainly about decoupling the
description of a low-level image processing operation from implementation
details regarding tiling, storage of intermediate results, etc. OpenCV is a
library implementing high level algorithms like multi-view 3D reconstruction
that Halide does not even attempt to address.

~~~
sandGorgon
My mistake, I phrased my question incorrectly. Why not opencv with a halide
backend. The tensorflow whitepaper compares the TF ir to the halide ir. In
fact, the work in tensorflow core is around XLA
[https://www.tensorflow.org/performance/xla/](https://www.tensorflow.org/performance/xla/)
(which brings compatibility across various architecture)

It seems that this is more an attempt at locking the tensors to target custom
metal, so that it can't be copied and made to work on other phones.

For examples, the Google Camera NX version is a port of the HDR+ feature on
compatible phones.. Even the cheap Xiaomi. They actually give results
comparable to Nexus.

However, by locking the model to a very custom image processor (and custom IR)
, this can't be ported. This looks to me as a version of walled garden more
than a performance win.

~~~
trishume
Halide is much more flexible. OpenCV is a fixed high level toolbox, combining
things requires lots of intermediate buffers. Halide allows compiling
arbitrary kernels in a fused way with extremely good performance. Sure you
could implement OpenCV functions using Halide, but better to expose the lower
level more flexible API. Source: I’ve rewritten parts of an OpenCV app in
Halide for performance.

Also your argument that it’s not about perf makes no sense to me. Sure other
phones have HDR, I bet its way slower and more energy intensive than the Pixel
2. Except maybe the iPhone, which may also have custom silicon. If they had
custom HDR tech in software they wanted to keep they could just keep the
source closed, or patent it.

~~~
sandGorgon
That bet is the problem. Tensorflow on mobile already targets GPU and is going
to be even more so with XLA. This debate is going to be hearsay unless we have
an apples to apples comparison of the base tensorflow code running on the
halide ir targeted to the tpu vs a vanilla ARM GPU... Even using opencv.

I find it telling that they compared cpu vs tpu... Not GPU vs tpu.

~~~
joshuamorton
I'm really confused why you keep bringing up opencv. What does opencv have to
do with the (presumably) TF ML model running on the IPU that you want to
benchmark against the TF ML model running on a GPU?

~~~
sandGorgon
Because halide is positioned as a high performance improvement over opencv.
While I believe it is just an architectural decision to enable a non-portable
ML model.

~~~
trishume
But Halide does have higher performance, is portable to many platforms as well
as GPUs, CPUs (w/ parallel SIMD), and isn’t used for ML models...

------
jrs95
Maybe Google should start designing it's own CPUs/SoCs as well, since Qualcomm
doesn't really seem capable of keeping up with what Apple is doing. Snapdragon
835 came out 6 months after the A10, and scored about 2/3 as well at single
core, and less than 10% on Geekbench 4. A11 scored twice what the 835 did on
single core, and over 50% more on multi core, and the 835 is going to be the
best chip available in the U.S. for Android until sometime next year. Plus,
you've got Samsung getting exclusive launch deals for the newest Snapdragon
chips now, so other manufacturers are just now launching their flagship
devices with hardware that is increasingly far behind what Apple is putting
out. This gap could end up being as bad as Intel and AMD pre-Ryzen if
something isn't done about it. Imagine what the high end laptop market would
have looked like if only Macs had Intel chips!

~~~
awalton
The problem is that basically Qualcomm is strangling the entire cellphone
space. So what if you built a completely new, much faster, much better-in-
every-way SoC... if you still have to pay Qualcomm for the _entire chipset_
because they refuse to sell you just the modem without you paying the full
bill.

Multiple designers _could_ run circles around Qualcomm, but the marginal cost
of doing so combined with the number of CPUs you'd have to mint to do so makes
it a losing proposition. Apple's in the unique position of being able to
leverage Intel to bend Qualcomm over a barrel, and Qualcomm fires back with a
gazillion lawsuits and New York Times articles that cry "woe is me, Apple is
eating my lunch!"

On a different topic, benchmark navalgazing is probably not going to help
anyone. Every company that's been designing CPUs from the mid 80s on has been
writing benchmark defeating code, and I'm 100% certain PA SEMI/Apple is no
different; they certainly have the silicon space to waste writing Application-
specific-accelerators for acing benchmarking tasks, especially on a platform
with as much opacity as Apple holds over the iPhone.

~~~
dragonwriter
> The problem is that basically Qualcomm is strangling the entire cellphone
> space. So what if you built a completely new, much faster, much better-in-
> every-way SoC... if you still have to pay Qualcomm for the entire chipset
> because they refuse to sell you just the modem without you paying the full
> bill.

If Qualcomm is that big of a drag on Android (or not-Apple, but it amounts to
the same thing) smartphones, maybe the solution is for Google to buy them
outright and rearrange their practices.

Sure, it'd be expensive, but not anything Google couldn't afford.

------
GuB-42
So what is that IPU exactly? Lots of buzzwords but few actual details about
the architecture.

Is it like a integer GPU? The massive amount of ALUs suggests something like
that.

But later, I see "Notably, because Pixel Visual Core is programmable, we’re
already preparing the next set of applications.". So, is it FPGA-like?

~~~
sp332
It's an Image Processing Unit. Architecturally I'd expect it to look like a
GPU, maybe with something like the new matrix-multiply unit from their TPU.
[https://www.slideshare.net/AntoniosKatsarakis/tensor-
process...](https://www.slideshare.net/AntoniosKatsarakis/tensor-processing-
unit-tpu)

Edit: wrong name

~~~
godelski
So I'm seeing this and another comment downvoted. Is that because you're
wrong? If so, can one of the people downvoting (or someone else) expand and
actually answer the original question?

------
mholt
Cool, but am I the only one who thinks the photos on the right (toward the
bottom) are a little overexposed / bland / washed-out?

~~~
abainbridge
Yeah, agreed. I screen captured the unprocessed image from the webpage (which
has been jpeg'd and therefore lost lots of the shadow detail). I loaded it
into a paint package and did a simple (and computationally cheap) histogram
adjustment. IMO, the results were at least as good as theirs. And these are
the results they cherry picked. Not impressive.

I guess most people won't load the image into a paint package to fix it, so
their aim is just to automatically do a job nearly as good as a trivial change
a human would make. Still, I was hoping for more.

~~~
sdrothrock
I don't think the Pixel Visual Core is supposed to do anything new or
revolutionary -- the primary purpose is to do these old things FASTER for the
user.

~~~
abainbridge
But a simple histogram stretch can be done with very little processing power
by modern standards. 50 ms on one core of the application processor would be
enough.

I think they just chose bad examples. The results from this Google Research
blog post are much much more impressive:

[https://research.googleblog.com/2017/04/experimental-
nightti...](https://research.googleblog.com/2017/04/experimental-nighttime-
photography-with.html)

I wish my smartphone could do _that_ automatically.

~~~
dharma1
You end up with a lot more noise with simple histogram adjustment from a
single image, plus the dynamic compression to a final 8bit image won't look as
good for many use cases.

Exposure fusion from multiple images has much higher usable dynamic range and
lower noise.

------
bitL
I wish my Nikon pro camera had a similar chip inside. Now we have two worlds -
awesome pro cameras with incapable post-processing chips (I mean beyond RAW
and initial processing) and horrible mobile cameras with outstanding post-
processing chips.

~~~
ggus
Also the double lens, one for wide angle, one for tele, snapping at the same
time and combining the images into one.

It will be a looooong time until we see that on a pro camera with
interchangeable lenses, if ever.

~~~
ksk
We already have zoom lenses that can do wide angle and take portrait shots.
Having multiple full frame sensors with multiple lenses doesn't make much
sense to me TBH. What is the problem you're trying to solve here?

------
ttoinou
The left/right comparison for HDR are pointless : they should show the same
image with same contrast/exposure with the two techniques at 100% pixel. The
difference between without HDR and with HDR should be less noise (otherwise if
it's just about having more exposure in the darkest parts of the image, then a
little bit of image processing can do the trick)

~~~
alkonaut
Hopefully the idea is that it looks at the scene, decides which exposures are
needed to capture the DR (for example 1/100s ISO800 for the foreground and
1/200s at ISO100 for the background) and then quickly captures exactly those
two exposures in rapid succession. Then the software would try to compensate
movement between the frames and stack these exposures into one.

So long as it involves multiple pictures and the inputs (exposure params)are
computed are taken done when the photo is captured, then it does things you
can't do with image processing in any single image.

~~~
ttoinou
Yes you can, there are multiple inputs BUT the effect is about extending
dynamic range only (not create an artistic effect between the inputs).

You can always extend numerically the DR of any set of values but you will
stumble upon some noise eventually.. This HDR+ module is supposed to have less
noise in the shadows OR have better sun effects / beautiful highlights
(candles for example) / good looking clouds. A good test would be shooting in
strong direct sunlight and look at the shadows.

~~~
alkonaut
Right, the key "artistic" thing is identifying foreground/background, faces,
interesting elements etc., and simply try to balance the exposure to create a
pleasing end result. I have no idea what this thing does, or even if it uses
multiple exposures (or even multiple cameras), but I was simply thinking that
facial recognition/focus and automatic toning has been around for ages, it has
to do something that requires _fast_ decision (at trigger time) about exposure
params.

I couldn't see whether that was done though, it could be done with most of the
same effect by simply exposing e.g. +2EV and -2EV and simply deciding
afterwards how to blend those images into a HDR given the image content
(faces, ...)

------
ggambetta
> HDR+ addresses this problem by taking a burst of shots with short exposure
> times, aligning them algorithmically, and replacing each pixel with the
> average color at that position across all the shots.

This is almost exactly how you'd make super-resolution pictures (adding a
nearest-point scaling up step at some point! Yet their super-resolution app
(Photoscan) has some sort of artificial upper resolution limit. If they do
plan to expose this capability to 3rd parties, I hope someone makes a proper
scan/super-resolution app! Also tried Office Lens, also left disappointed :(

~~~
TheAdamist
Super resolution would want to utilize the OIS to artificially shift the lens
around instead of cancelling movements. I dont think they are exposing
anything like that.

~~~
londons_explore
Turns out one doesn't have to. The users hand movements are at least one pixel
as long as the phone is handheld. Just use natural movement to do
superresolution.

Everything you need is available in the camera2 RAW API.

Using the users hand movements help cancel out various types of per-physical-
pixel noise too.

------
simonh
It seems odd to me that Samsung isn't doing things like this. After all they
have both the chip design talent in house, they design their own Eynos chips,
and even have the fabs in house as well. They're not as well regarded for
software, but they do at least have some in house dev capability. A Samsung
Galaxy with that sort of differentiation would, like this phone, make even a
die hard iPhone user like me take serious notice.

Or is Samsung actually right not to go down that path and the economics of
Android handsets just don't support the significant extra spend and
commensurate price bumps that sort of investment would require? The Pixels
look like great devices, but without dramatically upping their distribution
and marketing game is this really a vanity project for Google? I hope not, I'm
crossing my fingers that with the talent they acquired from HTC, Google is
really intent on making a splash in the market with the Pixel and not just in
the tech press.

~~~
ktta
I'm not sure if they have the software dev capability. Sure, they make nice
UIs/apps on top of Android, but what about core computer science and domain
specific skills? The new OS they tried to make was apparently full of kernel
level bugs.

Google has ML experts, compiler experts, etc. who probably contributed a lot
to this. I guess it is a perk of keeping domain experts in the company who are
there to advise when time comes.

------
bearjaws
HDR+ is one of my favorite features from my 6p.

I am routinely blown away by how well it performs at all lighting. Everyone
i've shown pictures from low light scenarios is always impressed.

HDR+ is a big reason I bought the Pixel 2, I know its going to keep getting
better.

------
tfolbrecht
Consider how inexpensive and high quality modern image sensors are. With this
SoC, you can use so many amazing image processing algorithms. ML based object
masking, deconvolution, focus stacking, exposure stacking, sensor fusion like
FLIR, frame stitching multiple cameras, color compensating in camera flash.
For serious heavy lifting you can send these bundles of images to the cloud
for processing. All done without any more user input than tapping the take
photo button.

It's going to seem like magic to the user.

~~~
tfolbrecht
A tag-a-long comment, this SoC + imaging system is going to make it much
easier to write the software because you know all of the particulars of the
optical system, sensor, position data, etc etc. It's not like implementing
this stuff on the application layer which is nightmare mode difficult. Cameras
are so fast, their controls are so fast, this is like some super human time
dilation photography and photogrammetry.

I've held out for so long, it's finally time to replace my HTC One m7.

------
phn
Even with all the fancy ML stuff, the HDR "glow" is still clearly visible on
most of these photos.

On the plus side, the co-processor seems programmable, which means quality can
improve without the need to upgrade the hardware.

------
MBCook
It seems odd to me that they’ve made this special part of the chip but it
won’t be enabled until a future software update that will come out 1+ months
after release.

The basic idea of using special hardware makes sense. Apple has done the same
thing, and as you get more and more data off the sensor this seems like the
only way to do analysis/adjustment in real time without wasting a ton of
power.

------
ramshanker
It looks like they are releasing these bits and pieces before the Finale.
Custom ARM core, SOC in entirety.

~~~
Aaargh20318
I'm not so sure about that. I don't doubt they have the skill to do it, but I
doubt they can make the numbers work. Apple can make custom CPU cores because
they sell a shitload of phones. Google does not.

~~~
gehsty
Maybe they would license it to manufacturers - use our core design AND our
operating system gratis, as long as you send users to our services?

Googles ad revenue is so important that they should do absolutely anything and
everything to protect it, spending billions on R&D to keep android at the
forefront is essential, and at the moment they are only really competitive on
the software side (A11 smokes latest snapdragon chips, as does the A10....),
offering a custom chip would give android the vertical integration that Apple
leverage to be so dominant.

~~~
gsnedders
It also gives leverage over Qualcomm etc. wrt availability of security
updates.

------
without_words
Is the HDR+ stuff running on the PVC already, or is it still running on the
application processor until they enable it?

They say it uses Tensorflow, but is it a limited subset of Tensorflow designed
only for image processing, like running a pretrained model on a TPU?

------
kbumsik
Wow. It feels like the new era of custom-designed SoCs. I should've studied
HDL like Verilog while I was in EE major in the school.

------
sfifs
Can't see any pics on Firefox Focus browser on mobile. Are they using some
weird Desktop only CSS?

~~~
sundvor
Works fine on my own Focus browser (Android). Weird.

~~~
zspitzer
Focus on Android currently uses WebView (aka chromium/chrome)

------
jhallenworld
Looks like Movidius Myriad... I'm surprised Google would make their own chip
for this.

------
amelius
> The camera on the new Pixel 2 is packed full of great hardware, software and
> machine learning (ML)

Is that hardware only accessible by the camera? If so, isn't that wasteful?

~~~
lorenzhs
I think the following bit at the very bottom of the article suggests that it
isn't:

" _HDR+ will be the first application to run on Pixel Visual Core. Notably,
because Pixel Visual Core is programmable, we’re already preparing the next
set of applications. The great thing is that as we port more machine learning
and imaging applications to use Pixel Visual Core, Pixel 2 will continuously
improve. So keep an eye out!_ "

------
agumonkey
Anyone feels it's tech for the sake of tech ? I don't even appreciate the HDR
processed shades. There's a lot more detail that's true, but 1) it's fake IMO
2) pics don't have to be perfect, it's just a moment of life, the details are
precise enough.

