
Show HN: I built an AI that recognizes food - subcosmos
https://www.infino.me/hungrybot
======
AJRF
Something like this would be a fantastic addition to MyFitnessPal.

I find myself sometimes not logging food on that app because I couldn't be
bothered to search for it.

On the other hand if the product has a barcode I always scan it in because
that is convenient.

~~~
jcoffland
If you're eatting a lot of food with barcodes on it then your doing it wrong.

~~~
SolaceQuantum
I don't understand this sentiment. My grocery store allows us to print
barcodes that describe the weight and price of any produce or bulk order we
purchase. This ranges from quinoa, oats, nuts, seeds, tea, along with the
usual fruits and veggies.

Also, don't bulk packages of vegetables tend to have barcodes on them
inherently?

~~~
jcoffland
But those barcodes wont show up in a public database. Also bulk foods are not
fresh. Fresh food from local farmers does not usually have a barcode on it.
This is not just snobbery.

------
nathan_f77
I would pay something like $5 per day for this if it was a real person who
looks at my photos, categorizes them accurately, and enters them into
MyFitnessPal.

I won't trust an AI until it has a massive database and a million users. Until
then, it's just not going to work well enough. You also need to convince users
why they should use your app, instead of MyFitnessPal (which already has a
massive database, including barcodes.)

~~~
subcosmos
I have considered something like this as a business model, although I was
thinking of nutrition coaching instead of mere logging itself, but thats
actually a really great idea.

~~~
jqueryin
There's nothing stopping you from having a human aspect do the training and
verification behind the scenes.

It's how companies such as Expensify were able to handle their receipt OCR
gracefully for such a wide variety of invoices and receipts.

I'd err on the side of saying this is a more palpable sales approach to
selling it as AI. Letting your customers know that while AI is running the
show, things are being monitored closely and fixed by real people when
misclassifications arise.

~~~
subcosmos
Ive been running the underlying nonprofit (infino.me) for about 3 years with
this ambition to study gene-exercise-nutrition interactions, with very little
interest in making any money at it. But this thought of having a paid service
for people who don't want to go through the asspain of logging .... this does
have me intrigued. Everyone tells me they want an effortless meal logging app
... and photo snaps might be the closest we can get short of having something
like google glass persistently monitor intake.

------
subcosmos
Built using those newfangled convolutional neural network things. Trained
caffe on a set of about a million food related images. Can't believe how
accurate the classifier is...

UI needs work. Would love feedback on how to make food logging more intuitive.

~~~
sandGorgon
why caffe ? why not something like h2o or tensorflow ? just curious.

~~~
cemsazara
Is not Caffe actually GoogLeNet? So technically it is Google architecture.

------
koolba
Someone should combine this AI with something like Google glass (so you don't
have to pull out your phone), a basic calorie/nutrition filter, and a shock
bracelet[1].

That way when I look at a Krispy Kreme donut I get immediate feedback!

[1]: [http://pavlok.com/hello.php](http://pavlok.com/hello.php)

~~~
internaut
I'm dying here. Sign me up for your startup.

Let's call it: "American Dystopia"

~~~
subcosmos
There's a trump joke in there somewhere.

~~~
internaut
Electrification was called "the greatest engineering achievement of the 20th
Century" by the National Academy of Engineering. -
[https://en.wikipedia.org/wiki/Electrification](https://en.wikipedia.org/wiki/Electrification)

With today's technology we can do it again.

------
solaxun
I've been thinking of this exact thing for a few years now, good to see others
are interested too. I think there is HUGE potential for an app that can
accurately (within a reasonable error range) count macro-nutrients through
pictures of food, but I think that is WAY more difficult than you would
imagine. How can you train a classifier to tell how much butter is in a dish?
How can you account for food that is literally underneath other food?

Maybe a combination of spectrometer, weight, and photos would be a future
solution. I certainly don't have an idea of how it would be implemented, but I
think using data from all 3 of those sources could be a good step.

~~~
subcosmos
Another angle is post-consumption. Ive always wondered if precisely monitoring
body temperature might tell you something about consumption, since there is
this 'thermic effect of food'. Would be hard to get right though ...

In the lab, we can use doubly-labelled water to precisely measure calorie
intake and burn, but its so damned expensive. If only it were commercially
viable.

~~~
hokkos
About post-consumption could a turd-shot help to compute the calories in/out ?

~~~
subcosmos
That could be called hungryPOT ...

ah, good ole toilet humor

------
holografix
This has HUGE potential. Some of the comments here disregard the fact that AI
can do a __better job __than humans. An expert chef looking at dishes will
make an educated guess on the ingredients and quantities that went into a dish
and will have a better idea of the caloric content. The AI could learn a
similar skill, identifying whether a dish contains pastry, buttery sauces,
fatty meat etc etc.

------
nickpsecurity
You have a business opportunity here if you train that on fresh vegetables.
Many people have trouble with these issues:

1\. Knowing what the hell a (example) rhubarb is or looks like.

2\. Knowing if it looks fresh or about to go bad.

Being able to aim a camera slowly across an isle with it getting highlighted
would solve first problem. Spinning vegetable around in front of the camera
would could solve second. So, try that and make it a paid app.

~~~
subcosmos
You know, its funny. being a former fat guy, Im not entirely familiar with all
of my vegetables. I took this app to a farmers market recently and it was
actually helping me identify a few things ...

Had never seen a beet before un-cut. Ashamed of my ignorance ;)

~~~
nickpsecurity
I live in a semi-rural area. Gives me a bit more experience. I still don't
know what all these things are in the supermarket, though. So, I can't judge.
;)

That said, you should try training it on different kinds of produce that look
very similar. One example I recall is Zucchini vs Cucumber as the stem at one
side is the giveaway. Another is Nappa Cabbage vs Romaine Lettuce. Only two
examples I could think of off top of head that would confuse people.

------
zeveb
Pretty nifty, but please consider allowing creation of an account with a
simple email address, or even anonymously. I really don't want to have to
connect my Facebook or other social-networking information to your app in
order to use it.

Thanks for allowing one to try it out without logging in via another service;
it'd be great to be able to actually have an account.

~~~
subcosmos
Email accounts are definitely on the long todo list.

------
aembleton
This is not compatible with my device which is an LeTV X500, running Android
5.0.2. Is there any reason that it might not work? I'm in the UK, if that
might be the reason.

~~~
rymate1234
This happening for me as well with a Moto X Play running Android 6.0.2

Looking forward to trying the app once I can install it though, looks neat

~~~
subcosmos
Found my bug! Try again. Way more droids are supported now.

------
gchristen
Hey Subcomos, great work! We've been working on something pretty similar, an
app called Logameal (only available in the UK). We have run into (are running
into) a lot of those UI issues too and would be glad to help. Let me know if
you'd like to chat.

~~~
subcosmos
Your UI looks great!

------
aantix
I remember the latest Android camera app being able to extrapolate distances
in the app (thus simulating photos with large apertures/shallow depth of
field).

I wonder if you could use this visual distance algorithm to somehow judge
portion sizes/plate sizes?

~~~
subcosmos
Haven't thought of that! What I have considered is letting users put something
like a quarter or ID card in the field of view for that purpose.

Aside from that though, I figured Id do portion estimation down the road
should this preliminary version take off.

~~~
aantix
Good thinking. Assuming you don't have the portion size data, you may be able
to run your data set through Mechanical Turk or hire an oDesk worker, and you
could build a training set..

------
micro_cam
Nice work! Another big user group of stuff like this is diabetics who need
carb estimates to adjust insulin doses.

~~~
subcosmos
I earned my PhD in diabetes research at the Mayo clinic. My whole goal for
this project is to cure that damned illness.

Aim high ...

------
franciscop
Isn't it strange that in [1] the beginning of the "Normal" BMI has more risk
of death than the beginning of the "Obese"? I'd even venture to say that the
"Overweight" range has, in average, lower Mortality Risk than the "Normal".

PS, another plus for the Metric System. The Imperial System form includes
Feet+Inches, while the metric one only needs Centimeters for the height.

[1] [https://www.infino.me/join/](https://www.infino.me/join/)

~~~
bjterry
The finding that people with an "Overweight" BMI have a lower rate of all-
cause mortality is consistent and has been called the "obesity paradox." A
very interesting study came out this month that addressed it.[0] Basically,
they did a prospective study of 55,000 (mostly elderly women) where they
measured their body fat percentage directly using DEXA scans. It turns out
that when you control for body fat percentage, it's actually better for all
cause mortality to have the highest BMI category and the lowest body fat
percentage. Of course, the measure "BMI after controlling for body fat
percentage" is kind of unintuitive, since BMI is supposed to be a crappy proxy
for body fat percentage. But basically, as long as your body fat percentage is
low, being bigger is healthier than being smaller.

0:
[http://annals.org/article.aspx?articleid=2499472](http://annals.org/article.aspx?articleid=2499472)
(this paper used to be available, because I read the whole thing, but I guess
it's paywalled now)

~~~
subcosmos
Thanks for posting this! I wonder if part of this paradox is also explainable
because people with high muscle mass will tip towards the high end of the
scale.

Very cool!

------
virtualritz
First thing I tried: logging /two/ Granny Smith apples manually.

I had do the entire process twice because there is no (obvious) way to change
the serving size to 2.

~~~
subcosmos
If you click the 'no barcode' button to pull down a fatsecret nutrition
record, there is a servings pulldown that is displayed. Its kinda hard to see.
Working on the CSS

------
zaroth
Please do blog about the technical details, if you could post a walkthrough of
setting up and training the net it would be great!

~~~
subcosmos
Someday! Right now I need to put the fire out in my server room. Didn't quite
prepare for HN traffic ...

------
mark_l_watson
That is very cool subcosmos! I am also very interested in both
cooking+nutrition and AI (see for example my web site cookingspace.com). My
contact information is in my HN profile - shoot me an email if you want to
talk about AI, food technology and business, etc.

------
Zolomon
This should be in use at those self-registry machines at food markets. I
imagine that you place the food item on the scanner and it will automatically
register it or ask for human input when something unrecognized is found.

Would decrease cheating perhaps?

------
sandGorgon
Where did you get your training data set? I'm trying to build a network to
recognize faces from ID cards and match them to faces/selfies. Onenof the hard
things has been to get a training data set.

~~~
zodPod
This sounds like it could be used for the wrong purpose for sure.. Are you
working for some nosy employer?

~~~
sandGorgon
Employer? No - we are automating paper applications for financial
institutions. This includes a photo ID and a photo.

That is checked manually today.

~~~
zodPod
Haha nice! I was assuming someone was going to do some web screening before
hiring people.

------
devstack
That's very interesting indeed. Are you using any API?

~~~
subcosmos
Fatsecret API for nutritional information and Foursquare for checkins, but
installed and trained the neural network on my AWS instance for the prediction
stuff.

~~~
tdkl
Is it possible to implement the myfitnesspal API in the way user can scan the
food and it gets entered in their food log on MFP ?

~~~
subcosmos
I've been emailing myfitnesspal for years for access to their API .... emails
go ignored.

------
mrfusion
I think something like this could get really good at estimating calories too.
I'm saying within +\\- 50 calories.

~~~
subcosmos
I've heard that Google is working on that problem. They have way more
resources than my one-man part time nonprofit operation, so I look forward to
how awesome their thing will be when they finish it.

Will probably opensource this, since I want to grow up a nice research
community around quantified self fanatics.

------
npalli
The classification looks good, congrats!. Where did you get the million
labeled images to train your classifier?

~~~
subcosmos
Wordnet! Got a list of URLs from there and then hit flikr with some python.

------
natch
Is there a writeup about how you approached this? All I see are a couple of
app store links.

~~~
subcosmos
Ive been meaning to make a blog for some time now, since my nonprofit also has
some preliminary research results with the genetic basis of obesity. Just
havent had the time really. It will happen though. These neural networks are
amazing and worth sharing.

------
doppenhe
This is great. Where are you hosting your caffe model? Just AWS with API
Gateway?

~~~
subcosmos
Its on the instance right now, but I could certainly post it to github at some
point. I think I need to retrain it though and swap a few categories out.
Theres a few animals in there ;)

Also, been needing to train on non-food items to weed out abuse.

------
KaoruAoiShiho
How much hardware and how much time was used in training?

~~~
subcosmos
Took a single AWS instance about a month. Not too bad.

------
amelius
How does it measure the glucose content of that sauce?

~~~
zo1
Probably gets it from FatSecret?

~~~
subcosmos
It does! They've been gracious enough to let me use their data since I'm
running a nonprofit experiment.

------
johnPullman
Another app that is similar to this is called tuingle
[http://tuingle.com](http://tuingle.com)

------
misiti3780
where did you get the annotated training data ?

~~~
subcosmos
wordnet:
[https://wordnet.princeton.edu/wordnet/](https://wordnet.princeton.edu/wordnet/)

~~~
aaronfc
I can't find anything related to images... Could you expand on this?

~~~
subcosmos
Oh dear, I meant Imagenet! [http://image-net.org/](http://image-net.org/)

~~~
aaronfc
Awesome! Thanks! :D

------
halonewstza
halonewstza.blogspot.com

