
Launch HN: Mirror (YC W17) – turn a selfie into emoji that look like you - sergefaguet
Hey HN!<p>We are launching an app (Mirror) that uses a single selfie to create ~250 emoji&#x2F;stickers that look like you and your friends.<p>We previously started a large online travel company in Russia (Ostrovok, &gt;$500m in sales this year) and while there got very interested in AI as a tool to solve business objectives like predicting cancelations or conversion rates to inform marketing spend. And when we started something new, we wanted a product that had extremely high-frequency usage to build feedback loops into + something that a lot of humans really want to do, but that AI can do better.<p>Emoji keyboards sound like trivial toys, but this is actually a path to teaching AI one of the most complex human social behaviors. Offline, our faces communicate the majority of information that passes between us; far more than words do. In fact there is a whole piece of our brain (fusiform gyrus) dedicated to facial analysis and if lesioned humans lose ability to read faces while losing nothing else. Basically humans pay closer attention to faces than to any other object in the universe – from the day we are born.<p>AI needs data that informs it whether its decisions are right. And emoji are by far the largest structured source of facial data + usage in the world – &gt;15 billion sent a day. Getting that ocean of data will help train AI that can solve all kinds of future problems – from VR characters that look like you in the gameworld, to personalized advertising that has your face in it the way you like, and a lot of other things. Everyone asks us why after starting a big “serious&quot; company, we decided to work with emoji, and this is the reason.<p>On to the actual product. We think custom emoji apps like Bitmoji are a good step towards bringing our real, unique faces into messaging. We take this further and let you:<p>- Get your emoji in 3 seconds with one tap.<p>- Not spend mental energy wondering what your head shape&#x2F;nose shape etc. are<p>- Add your friends&#x2F;family (soon your pets and more) to your own emoji in seconds. Manually constructing 10+ friends is just too hard.<p>- Use all text-containing emoji in 8 languages (will expand to &gt;25 shortly). We really have no idea why Bitmoji and other such apps tend to be English-only.<p>- Get emoji that become more and more recognizable&#x2F;liked with time by learning from data (explicit choices + implicit metrics). For example we see that people want larger-than proportional eyes and darker skin. Except in Japan where it seems users want lighter skin. Really excited to find more like this with usage data.<p>Google Allo (and some others) tried doing this but got very little done. The reason is that while the idea is extremely obvious, the tech is quite hard. Here&#x27;s why:<p>- You have to get a lot of facial features right. Some of these (e.g. hair, facial hair) are very diverse across age&#x2F;sex&#x2F;ethnicity. Some (e.g. head shape) are 3d but need to be inferred from shadows in a 2d image. Some (e.g. lower eyelids or wrinkles) can be detected but have a tradeoff of authenticity vs beauty. Some (colors of skin, hair, lips, eyes, eyebrows) are highly dependent on lighting conditions. All are must-haves. And this is still not enough – you have to detect and place wrinkles, birthmarks, sunspots etc. too<p>- You have to compensate for a lot of variability in selfie-taking – lighting&#x2F;shadows, turn&#x2F;pitch&#x2F;tilt, camera distance, device camera type, background, whether the user is smiling or not etc.<p>- You have to use one photo to infer and reconstruct how that person looks with different emotions.<p>- There’s a whole dedicated part of the brain for this task, with a lot of neural machinery (fusiform gyrus). Humans can’t formally explain how they recognize other humans, but are very demanding to result quality. Humans take our exceptional skill at face recognition for granted, although for us a face is THE most well-perceived object in the universe. Think about it: can you instantly recognize a piece of a random wall that looks similar to every other wall, and that you stared at for an hour 10 years ago? You can do this with a face.<p>All the above makes training neural nets that solve recreating (rather than simply recognizing) faces much harder.<p>It took us close to a year and a large engineering team + multiple people with many years of experience with faces to get this to beta.<p>We know there is still a long way to go to make the product great. Right now we are working on several things to make it better:<p>- improving the tech from just “personalized” to “recognizable by friends” – both by utilizing feedback loops and by looking at new developments in AI. Particularly excited about GANs although we don&#x27;t see a pure-GAN approach as viable (longer discussion).<p>- adding new features – e.g. auto-detection of groups of people, pets, clothes, backgrounds. doing a lot of this automatically based on your Photo Stream content and on-device ML technologies.<p>- expanding our content library to ~10,000 emoji in 25 languages.<p>You can get the app here: <a href="https:&#x2F;&#x2F;www.mirror-ai.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.mirror-ai.com&#x2F;</a><p>Let us know what you think – we really appreciate all your advice!
======
jasonkester
I installed this and pointed it at my six year old son. It came back with a
big image of an angry girl shouting "SHUT THE FUCK UP!"

Not really what I was expecting. Glad I wasn't showing him the screen at the
time, at least. But you might want to fix that.

~~~
sergefaguet
ugh. thanks for telling about this. we don't detect the gender of kids too
well, but we do detect age and should just turn off the more explicit
stickers.

thank you, very useful. and sorry you had that experience.

~~~
thisisit
Can you please remove the profanity ones from the first page?

~~~
sergefaguet
yep, doing this.

------
djrogers
Well that was a bit of a waste of time - the stickers (they’re not emoji,
they’re stickers) look nothing like me - got the color of my hair totally
wrong, the shape of the face is nothing like mine, and the facial hair is
completely wrong.

To top all of that off, I was immediately presented with profanity thats
completely out of line with the store rating.

Nice try, but bitmoji was more fun and way more accurate.

~~~
sergefaguet
thanks for your feedback – we know it doesn't work for some people and are
working to make it better. we'll get there :)

regarding profanity – thinking how to both let people who want to use it use
it (as many do) while not annoying anyone with it.

thanks for trying the product, sorry it didn't work for you

~~~
torgoguys
Um, yeah the profanity was completely unexpected. I happen to be one of those
weird people that basically doesn't swear and presenting "me" with a big ole
(non abbreviated) STFU word bubble feels bad. Uninstalled.

If you're going to have them, put them in a "you gotta look for them corner"
of the app, not front and center.

------
y0ghur7_xxx
I tried it, and I know there is a lot of work behind this, but you asked for
an opinion and here it is: I look nothing like the emojis, I could not
identify myself with them.

~~~
pacaro
Seconded. It got my hair color wrong (I have light mousey brown hair, with a
sprinkle of gray, the app creates an avatar with dark brown hair), and it got
my eye color wrong (it chose brown over my blue)

While the idea is admirable, there is very little scope for error in a space
as personal as thjs

~~~
sergefaguet
agreed re scope of error – this is the key challenge given the complexity of
face perception.

this is a beta, we will improve quickly.

------
throwaway0071
I don't get it. The technology looks super interesting but the end result
maybe is more targeted at teenagers?

 _> AI needs data that informs it whether its decisions are right_

Well, aren't the emojis all out there for you to analyze? Isn't there a lot of
videos with real faces for you to analyze?

 _> VR characters that look like you in the gameworld_

XBox does this today, doesn't it?

 _> personalized advertising that has your face in it the way you like_

I couldn't care less about advertising. Maybe you're pitching to investors?

 _> Everyone asks us why after starting a big “serious" company, we decided to
work with emoji, and this is the reason._

What's the reason? It's not clear.

As an user of this, it looks like a gimmick (like Tom The Talking Cat).

~~~
sergefaguet
hey! let me illustrate with an (oversimplified) example:

– we tweak eyes to be larger-than-real and observe the reaction by letting
users pick from several options or just observing usage frequency. that way we
find the optimal point and then slice it down by age, sex, ethnic etc. groups;
as well as to personal preference. \- do that with a large amount of data +
genetic algos that do much less-obvious things than eye size. \- when a user
wants a World of Warcraft orc avatar, we can use the data to make the orc look
both like an orc and like he wants to see himself.

you can't do this with generic emoji – you actually have to keep tweaking and
seeing user reactions to understand how people truly want to see themselves
and others. and then you can use this to other contexts.

thanks for the feedback, i agree this is not explained clearly enough in the
post.

~~~
throwaway0071
Thanks for explaining it in more detail. This is very interesting (and if it
doesn't pan out, you got the hottest skills in the market right now). All the
best.

~~~
sergefaguet
thank you! really appreciated :)

------
nkkollaw
Meh.

Your payoff says I could turn a selfie into an emoji that looks like me, but I
got stickers instead of emoji, and they looked nothing like me.

~~~
sergefaguet
we'll make it better. matter of time + data. right now it works for some
people, but not for others – just tough to get really good results because of
how complex human facial perception is.

thanks for your feedback!

------
jamesmcintyre
Just tried it out, only one attempt and got good results!

Couple of ideas, you may already have considered:

1) Use a color-correction algo to help with tinted hair. Then use something
like MASK-RCNN to get pixels of hair and get dominate cluster of color. 2) If
lighting/shadows/environment causes so much trouble maybe have user
rotate/tilt their head in circle-motion (like Apple has users do to train
FaceID) then these run ~30 images through your algos and average the results?
Maybe have torch on during capture to make the data more homogenous. (Or do
once with torch, once without and train two different networks for both types
of captures.) 3) If user manually overrides hair color show them your algos
next 5 best predictions as other color choices so you can more easily retrain
if user selects one of these other predictions. Maybe have them manually
adjust bounding polygon of hair detection and even adjust image color
correction sliders to show "what my hair actually looks like" (so you can
train color correction).

Just some ideas. Cool product! Good luck!

~~~
sergefaguet
thank you! passing to the ML team!

------
bshlgrs
This app sounds really cool!

I installed it but wasn't too impressed with the quality of the emoji. (This
could be because I look kind of weird; eg I have pink hair.) It's really easy
to take photos and see how the emoji look; I encourage other people to try
that out.

I'm excited to see whether this technology ends up working well in future.

~~~
sergefaguet
hey!

yea we don't have pink hair – basically it was really hard to get the color
palettes working right in all kinds of different lighting conditions, so we
didn't include rare/custom hair.

we'll get better at it :) thanks for using the product!!

------
caio1982
I have a nice lumberjack-kind-of-beard and my avatar showed up with a designer
chinstrap and glossy lips...

~~~
sergefaguet
yeah – it's definitely not perfect. we are quickly making it better. in
particular beards are very limited because there are no good existing
datasets/trained networks for them, so we have to do everything ourselves.

thanks for mentioning, we'll improve!

------
shostack
This appears to be a keyboard? What other data does your app have access to in
that regard? Do you get to read the contents of SMS or email or anything else
with that? Also, what data or images of mine are stored on your servers?

~~~
sergefaguet
we don't log your messages – in theory keyboard developers could do this, but
Apple would wipe the floor with us if we did. we do plan to have a bag-of-
words RNN that runs locally on the device to suggest emoji to you depending on
context of your SMS etc without sending anything to a server.

we do send your images to our servers at this point, and we do store them +
standard usage data about how you interact with app flows (conversions,
retention etc).

in the future we'll try to have ML on the device and not even send your images
to the cloud. both better for privacy and saves us money.

thanks for bringing this up. i have zero sense of privacy for myself so it
helps when others point out how they feel about this.

~~~
shostack
Thanks for the detailed response. In the interest of total transparency, I
have to say I'd be a little concerned with installing a keyboard app that
sends photos of my face to servers in Russia, particularly with everything
going on right now.

I'm not sure what could be done to alleviate that concern, and perhaps I'm
being paranoid, but for now, that has stopped me from using the app. Obviously
a 10 year old wouldn't have that concern, and perhaps that is a different
problem.

I hope you understand--nothing personal.

~~~
sergefaguet
they aren't in Russia, they are in AWS :)

if we kept it in Russia latency would suck + we'd have to comply with local
regulations we don't want to deal with.

------
kylehotchkiss
Cool app! read my facial shape, eyes, hair color, and style from an indoor
selfie. Wish I could have just uploaded a photo to start. People might have a
better photo of themselves in their photo stream.

~~~
sergefaguet
we initially let people do that, but they started uploading much worse photos,
group photos etc. – the selfies help consistency. glad you liked it :)

------
logiclion
Just wanted to say, this is a fantastic app in my opinion. I took one photo
and it was already more accurate & personally identifiable to some friends as
a 'better' Bitmoji.

I'm impressed

~~~
sergefaguet
really glad to hear!!! hoping we can get everyone to have this experience :)

thanks for giving us a try! if at some point you have other feedback pls let
us know.

------
yalooze
Is the app only available in the States? I'm in the UK and when I click the
download link on my iPhone I get this:
[https://imgur.com/zHytq4u](https://imgur.com/zHytq4u) (url:
[https://itunes.apple.com/us/app/mirror-emoji-
keyboard/id1199...](https://itunes.apple.com/us/app/mirror-emoji-
keyboard/id1199126741?ls=1&mt=8))

Maybe clarify that, or open it up to the rest of us :)

~~~
sergeytimoshin
Could you please try to open this link in Safari? I guess Chrome failed to
redirect to App Store.

~~~
yalooze
Safari on iOS works correctly and opens the App Store.

~~~
sergefaguet
thanks for finding this, we'll fix it!

------
sarreph
Excuse my (potential) ignornace, but isn’t this just Bitmoji (which has huge
market share), just with slightly reduced setup from face scanning?

~~~
sergefaguet
not quite. let me point several things out:

\- bitmoji takes a long time and a lot of mental effort to set up. i have no
idea what my face shape is and had to ask my ex-gf whether my eyes are green
or brown.

\- with bitmoji you cannot add other people to your emoji because even if you
DO go through mental effort and time to create yourself, you won't do a bunch
of other people. and often you really want this – e.g. i want to send an image
of YOU as Captain Obvious if the situation calls for it, not of ME.

\- over time this will get better and better in terms of quality, while
Bitmoji will not.

\- bitmoji has no market share in any other language than English, and little
market share in non-snapchat messengers.

~~~
whamlastxmas
My experience trying it just now with my girlfriend: I think we both like
bitmoji better. With Mirror my girlfriend took her selfie probably 5 or 6
times to get one she was happy with, and had to try her hair in different
styles to see what she liked. She also asked if there was more customization
options like bitmoji. I don't think people really mind the setup time of
bitmoji and possibly actually like playing with it.

Anyway, it's a cool app. I'm not trying to be a downer, just wanted to give
you honest feedback.

~~~
sergefaguet
thanks for sharing! i wonder if there are just different demographics here
where some people want to customize while others don't.

i remember that at our hotel booking company, only 10-15% of people used any
filters/sorts for hotels, and >80% just picked from the top list of what we
gave them. to me this is shocking given a hotel is a big and rare purchase.
but it makes me feel the majority of the world want the fewest taps and the
fewest mental barriers.

plus we kind of feel like we want to force ourselves to do a good job out of
automatic creation, and a lot of manual customization is defocusing.

------
garethsprice
Cool idea! I'd allow more options to customize - I didn't feel that the emoji
accurately looked like me and wouldn't feel comfortable sending this to
people. However some slight tweaks (manually changing the hair color, beard
style, etc) would go a long way to giving me a sense of ownership over the
emoji.

~~~
sergefaguet
thanks for the feedback! yep, we are planning to add exactly the tweaks you
mentioned. plus eye color.

will do this soon :)

------
aiyah4la
As a male with long hair past my shoulders, there seem to be no such hair
option when creating the profile

~~~
sergefaguet
hmm, obvious mistake. will fix.

------
alexdowad
Tried the app on my Nexus. It didn't work, at all. When I open the app and try
to take a selfie, the image is completely blurred out and it can't detect my
face. (The built-in Camera app works just fine, though.)

~~~
sergefaguet
huh. this definitely shouldn't be happening. does it replicate every time? any
chance you can try a couple times and post the exact time you are doing this?
will try to hunt down where it went wrong.

thanks for testing

------
augustocallejas
This is great. I had an idea similar to this, with the added concept of being
able to generate animated emojis using video and either keeping the audio of
the user or generating synthetic audio from text.

~~~
sergefaguet
yeah we decided against that because it would interfere with sharing frequency
– people are much more willing to share images than voice/video. plus
animation makes content volumes much harder to scale.

------
gnode
I wonder if there's any potential for this to be used for 3D avatars in VR. I
imagine it would be fairly easy to do. Not much of a market yet, but perhaps
in the future.

~~~
sergefaguet
definitely. we think doing this with emoji first is much better than going
after VR first. this is where the market (and the data) is today. and that
will help build the applications of tomorrow like VR.

------
bovermyer
My vanity made me click on this. My curiosity about using AI with human
interaction made me install the app.

Very interesting problem set. I look forward to seeing what you do with it.

~~~
sergefaguet
vanity... definitely my favorite sin :)

it does tap into a lot of human emotions. one of the reasons we decided to
pursue this area.

------
willart4food
I just installed and tried it:

\- very easy to install and use \- took a pic (I am not that good looking) and
my emoji looks great! \- I sent it to a couple of friends who liked it

~~~
sergefaguet
awesome :) really happy you liked it!! thanks a lot for the feedback!

------
scotttrinh
I went through the trouble of actually logging into my Hacker News account
just to say that the generated version of me looks exactly like me. O_o

~~~
sergefaguet
awesome :) really happy to hear this!!

------
mherdeg
huh what does 'emoji' mean in this context -- are you using some combination
of Unicode characters?

~~~
sergefaguet
no. a more accurate description is definitely "stickers." but everyone knows
and searches "emoji" much more. this is more of a marketing naming decision, i
agree that it isn't accurate.

~~~
mherdeg
Ah gotcha. So is the idea to signal to future partner companies that the
product is like a 3rd party version of "Animoji", using cool pictures to
collect a big corpus of face feature data for future products?

~~~
sergefaguet
if you mean about the word "emoji" it is more to pick up traffic from people
searching for emoji keyboards and the like.

if you mean in general, right now we are focused on building a great, highly
engaging product that people enjoy. and once that is there and at large scale,
we'll start thinking about other applications.

impossible to do anything in life well without obsessive focus...

------
soupflavor
Quite intriguing on a number of levels!

~~~
sergefaguet
thanks :) we obviously think so too

------
Jemaclus
My emoji turned out fine, but full access to keyboard data is a non-starter
for me. Sorry.

~~~
sergefaguet
well you can use it without the keyboard, or with an iMessage app if you are
on iOS.

unfortunately full access is the only way to make image pasting work. Bitmoji
and all other apps that utilize this mechanic do the same thing.

For what it's worth, Apple would brutalize us if we were offloading customer
keystrokes to our servers or something like that.

------
kartikkumar
What's the business model?

~~~
warent
Be bought by another company ;)

~~~
codegladiator
Looks like something facebook would buy. Do you consider them as competition ?

~~~
sergefaguet
Facebook are clearly a very smart and capable team that is very interested in
this exact area. So yes, they are competition.

But it is a classic case of that for a large company it is hard to execute on
many priorities at once and dedicate a lot of organizational resources and
attention. Plus they seem to have bet on going directly to VR.

We are laser-focused on doing this particular usecase extremely well, have a
large/growing/strong engineering team, and have business leaders that don't
have any other priorities beyond making this product excellent.

