
On-Device Supermarket Product Recognition - theafh
https://ai.googleblog.com/2020/07/on-device-supermarket-product.html
======
swimfar
I would love something like this paired with augmented reality to
show/hide/highlight only the products that meet my search criteria. Try going
to a big supermarket to look for whole-milk yogurt, for example. It's a pain
trying to filter them out of the rest of the 100 varieties.

Another thing I would like to do is compare products by price/quanity. Maybe
I'm trying to get the most protein/$, or oz/$, etc. You could also hook it
into some database to only show you ethically caught fish, organic projects,
fair trade products, local products, etc.

I've thought about this kind of technology a lot. Cool to see that it's
actually becoming possible.

~~~
azalemeth
> It's a pain trying to filter them out of the rest of the 100 varieties.

Arguably, shops _specifically design_ themselves to do things like this. When
I was growing up, it used to drive my dad mad that the local supermarket would
move almost literally everything around seemingly at random "to prevent
shoplifting". He was a pilot, and usually had little time -- he liked to go
into supermarket, grab x, pay, and leave. Suddenly, that stoped working, as
finding x became increasingly difficult. It really annoyed him. For a long
time the floor manager said that they moved their stock around to prevent
shoplifting – until one day the store manager, when asked the same question,
said it was "to promote footfall". Suddenly the real reason became clear.

>Another thing I would like to do is compare products by price/quanity. Maybe
I'm trying to get the most protein/$, or oz/$, etc. You could also hook it
into some database to only show you ethically caught fish, organic projects,
fair trade products, local products, etc.

In the UK at least, all of this information is required to be printed on the
label: product per unit money on the shop label and misery per unit product on
the product's label.

Are you asking for a filter / search engine for that on top? Either way, I
don't think your average supermarket would let that stay un-advertised for too
long. Products at eyeball level pay a premium to be there.

~~~
squeaky-clean
I don't know if it's legally required, but that's also the standard here in
the US. One problem though is the units may not always be equal. I was buying
canned coconut milk this weekend and among all the different choices were
$/pound, $/gram, and $/liter. It's easier to do the math from scratch yourself
at that point.

------
cs702
This is only the beginning. At the moment, mobile devices must run sub-SOTA
models, such as a MobileNetV3[a] for visual recognition, to provide an
acceptable user experience. As mobile devices continue to get more powerful
and start incorporating dedicated AI hardware for multiply-add-accumulate ops,
I expect to see devices that can run today's SOTA models locally within a
couple of years. Exciting stuff!

[a] [https://ai.googleblog.com/2019/11/introducing-next-
generatio...](https://ai.googleblog.com/2019/11/introducing-next-generation-
on-device.html)

------
awinter-py
interesting that one of the uses of NN here is to shrink the amount of storage
required per product in the product recognition DB -- compression is a neat
application of NN that generally follows from recognition / synthesis tricks,
and will matter a lot on mobile.

I'm wondering if 'on device' is an exaggeration. The post is saying 64 bytes
per product, and a DB of millions of products, but the video is showing
product details, prices, and a box image. I wonder if once it identifies the
product it then loads metadata from ze cloud?

~~~
anoncareer0212
'it had to load an image from the cloud' sounds much more like a nitpick than
'it recognized the product locally' sounds like an exaggeration

~~~
awinter-py
it's a spectrum, right? 'on-device' is a big claim with different implications
if you're thinking about this as an NN problem or a privacy problem. Loading
an image from the cloud = sending telemetry on your supermarket shopping to a
giant ads company.

------
apapli
This is interesting - I couldn’t determine how they train them system to
recognise all the products though.

Flavour variants, special promotional packaging etc make training hard due to
the sheer size of images that need to be in a training dataset to yield
accurate results.

How is this achieved?

~~~
joshvm
This is likely done using OCR:

> OCR is executed on the ROI for each camera frame in order to extract
> additional information, such as packet size, product flavor variant, etc.

You could probably also do some very simple statistics like a colour histogram
on the box, once you had high certainty of the product line.

------
riyadparvez
I am interested into why they haven't released this feature in their main
Google Lens app. My hunch is they released a separate app for experimental and
beta testing. Once these features get robust and reliable, they will probably
move these features to the main Google Lens app.

------
ganzuul
This is one of those obvious killer-apps that with a moment's thought was
obviously computationally prohibitive or much too complex for a sole
developer. If this is in the pipelines then we can soon also expect a bunch of
semantic apps.

------
tolidano
Didn’t Amazon do this with the Fire Phone and FireFly in 2015? What I need is
to comparison shop this item at 5 other grocery stores and FreshDirect and
Amazon Fresh and have it tell me if I’m getting a good deal.

------
nmstoker
Would be handy if they applied this to an internal shop level Google Maps

Would cut the need to speak to shop staff, who in many places in larger cities
don't have the language skills to know what you're looking for.

------
dillonmckay
License this to Instacart so I can get the proper size package I ordered.

------
mjs33
Does it take into account flavors? Yoghurt packages and Oreos come to mind.
Lots of different flavors and the packaging is subtly different

------
greesil
Interesting how it isn't just one giant neural net mashed together, and seems
to leverage "hand-crafted" features.

------
lifeisstillgood
An RFID scanner will do a lot of this surely ? The issues are presumably
access to the TFID number database, but for me the big open question is that
as we progress to an RFID chip in everything world, that access becomes a
public good - a complete supply chain record will be the next "food labelling"
fight

~~~
ladberg
That basically already exists with barcodes.

~~~
lifeisstillgood
1\. Barcode UPN lists are closely guarded secrets. Knowing that 123456 is a
lot of hummus is valuable to a lot of retailers.

2\. The supply chain for who made the hummus, who grew the chickpeas, were
they sprayed with what. That basically does not exist, let alone in public.

------
polskibus
Is this app open source?

------
arkanciscan
People who can't recognize products on a shelf often can't use a touchscreen
either. I live with a Boomer and he can't even use a Chromecast because he
can't see his screen, let alone use screen readers or voice commands.

~~~
pkaye
Its another option for the visually impaired.

------
armitron
Privacy nightmare not to mention solution looking for a problem.

~~~
gundmc
What specific aspects of the design do you consider to a "privacy nightmare"?

Typically on-device computing is lauded from a privacy perspective when
compared to cloud equivalents.

~~~
ffpip
Doing it on-device, and storing it on-device are 2 different things.

Shopping data is the most valuable. It is the end result of advertising.

What ad caused you to but that specific product? Can Google show a competitors
ad to you?

