
Computer, Respond to This Email - dpf
http://googleresearch.blogspot.com/2015/11/computer-respond-to-this-email.html
======
supersan
I'm often amazed how Google still gives it staff this unique hacker-like
approach to their million dollar projects.

For example, the stuff in Google labs (gmail) has had some silly things like
don't hit send when you're drunk but sometimes very useful features which may
be considered unorthodox like "Gmail telling you when it _thinks_ you wanted
to attach a file but forgot".

It's something I wouldn't generally associate with a very corporate design but
here they are wanting to add another silly feature and who knows how it will
turn out? Maybe it will be super useful and then all the other companies will
start to copy it.

But the thing is they're inventing new ways that really don't fit your product
development roadmaps. I really like that about them.

~~~
mmanfrin
I had a similar thought with Apple -- El Cap has a feature where if you shake
your mouse around, the cursor will momentarily grow in size so you can find
where it is.

That does not feel like a top-down design idea, but a bottom-up feature
designed by an engineer who was sick of those moments where he/she had lost
the cursor.

~~~
ErrantX
Oh wow! That explains wierd artifacts I've been seeing since El Capitan, when
using my trackball mouse. The cursor randomly grows in size, usually when I
move it between screens quickly. I expect it's some "random jitter" in my
using the trackball that causes it to happen.

~~~
elwell
Case in point why the OS is a terrible place to allow a hacky ethos.

~~~
kuschku
Yes. Popping up a notification the first few times (or the first time after a
few months of not using it) would be neat.

------
imh
I appreciate the privacy standards they used (no humans reading your email to
develop this), but am concerned that it's not enough. As I understand with
language models, overfitting takes the form of returning a sequence of words
seen in the training set. If this is overfitting in any part of the response
space, this could happen. Out of a million emails, how many suggested
responses are going to substantively resemble another response the original
author wouldn't want read by others?

~~~
Zelphyr
Much of this strikes me as a "just because you can, doesn't mean you should"
issue. Google clearly loves machine learning and doing cool things but I think
lately they've been taking it too far.

For example; after purchasing a book on Amazon recently I happened to do a
Google search on that book and the first thing I see is, "Your book is
scheduled to be delivered on..." Aside from the creepy factor I'm left
wondering what purpose this serves? I just ordered the book. I KNOW it's on
its way.

Turns out they just mined my emails from Gmail to provide it in search
results.

I'm sure some developer or product manager thought it would be a cool thing to
do without giving any consideration to usefulness much less user privacy. I
really don't feel like Google needs to know what I'm buying thankyouverymuch.
Gmail account: closed.

~~~
DaveWalk
> a "just because you can, doesn't mean you should" issue

To me, this phrase is the essence of much of Google's features. To my
discredit, I chuckled when I read the blog's phrase "we've used...deep neural
networks to improve...YouTube thumbnails." I am certain this was no easy task,
and a resulting technical breakthrough. But doesn't it sound kind of petty?

Of course, what's petty for one is essential for another. I wish every e-mail
client had that "undo send" feature, which was just GMail whimsy years ago. Is
the line between petty and essential always going to be blurred?

~~~
jsolson
Improving YouTube thumbnails can be a huge usability win Consider a series of
lectures or DIY videos with a common setting. Pulling out a frame that
captures something unique about the video (be it the DIY item being worked on
in close-up or an important theorem on a title slide or blackboard) makes it
easier for users to separate content and find specific items.

------
blixt
This kind of stuff is really cool. I imagine the future of Google being the
same old search box, but instead of entering a search query, you engage in a
conversation with Google so you can delve deeply into a narrow topic and get
back tailored responses to your question (as opposed to opening 10 tabs of
Stack Overflow links that are maybe related to your question).

I worry a bit about the long-term risks of this kind of training (query /
reply). While this is obviously very far from being "high resolution" enough
to single out very specific information, at some point these kinds of tools
(and assistant AIs that can answer questions) will out of necessity be able to
converse around very domain specific topics. At this point, how do they know
what data is private and what is not? It could be tempting to train these AIs
on chat logs or e-mail conversations, but that'd give them knowledge of very
private information which they might leak to others. Even if they're limited
to data that is accessible anonymously, they'd be extremely good at picking up
information that wasn't intended for the public. For example, if you could
describe a person called /u/andreasblixt on reddit, and leave it up to the bot
to put the pieces together that this person is also on Facebook, Twitter,
etc... and that obscure forum from 10 years ago. Food for thought.

A final thought on this... When these assistant AIs will inevitably have to
know something about you. For example, your preferred schedule, food
restrictions, preferred airlines, name, family, friends, phone numbers, where
you were last night, who you've talked to, etc. Even if we all get our own
namespaced AI assistant (i.e., the trained neural network that contains
private information is stored and encrypted for your access only), that
assistant's "brain" may very well become a prized target because if you get
access to it you can interrogate it for information (you most likely can't
just access the information in any meaningful way – you'd literally have to
give it queries to make it return semantic output with the information you're
trying to access).

Anyway, automatic e-mail responses, yay!

~~~
Natanael_L
Essentially integrating Wolfram Alpha and IBM Watson into Google?

------
icey
I just started reading Avogadro Corp ([http://www.amazon.com/Avogadro-Corp-
Singularity-Closer-Appea...](http://www.amazon.com/Avogadro-Corp-Singularity-
Closer-Appears-ebook/dp/B006ACIMQQ)) this weekend, and this reminds me quite a
lot of the emergent AI that figures heavily in the story (ELOPe). A quick
synopsis: developers build a system to "improve" responses from emails. The
system at some point is given the ability to send emails on its own, and a
poorly issued directive. It's been an engaging read so far, and fairly
hilarious since the corporation in the book is very obviously based on Google.

~~~
toomuchtodo
You'll love the ending. I highly recommend you read the sequel as well:
[http://www.amazon.com/gp/product/B007FZVI2M/](http://www.amazon.com/gp/product/B007FZVI2M/)

~~~
icey
I started reading the sequel this weekend, based on your recommendation. So
far it's been excellent. Thank you for the suggestion!

~~~
toomuchtodo
You're welcome!

------
rcthompson
Hey, remember that one time when this feature was Gmail's April Fools Day
joke[1]? Kind of funny how it's now a serious and actually useful-looking
product.

[1]
[https://www.gmail.com/mail/help/autopilot/](https://www.gmail.com/mail/help/autopilot/)

~~~
vram22
Another Google April Fools Day joke, some years ago, was Google Paper, or some
such name. IIRC it was funny.

~~~
rcthompson
Well, they kind of made that one for real too, didn't they?
[http://www.google.com/cloudprint/learn/index.html](http://www.google.com/cloudprint/learn/index.html)

~~~
kuschku
Great Scott, that site looks like it’s from 1985!
[https://www.google.com/cloudprint/gadget.html?user=0](https://www.google.com/cloudprint/gadget.html?user=0)

------
dikaiosune
While the technology is very cool, this pushes me that much closer to a
"dumber" email service. It's always a conflict for me as I love shiny new
things, but I'd rather we let AI loose on someone else's email, especially
when the AI's revenue stream is advertising (a business predicated on knowing
as much as possible about your audience).

~~~
bobmichael
I don't think I understand your reasoning. You're already tolerating an AI
processing your email to produce the ads, so assuming that, why does a cool
new feature push you closer towards a "dumber" service?

~~~
dikaiosune
Because I'm lazy and switching would be time consuming. Each new privacy-
related "innovation" motivates me a little more. You're right that the
practical effect of this new service is negligible on a technical level, but
it still reminds me that I should find an alternative I find acceptable.

~~~
TeMPOraL
So basically, because pretty much every useful data-and-automation-based thing
- especially on the social, not individual level - can be considered an
erosion of privacy, should we just stop progress? That's the sentiment I feel
radiating from yours and others' complaints about privacy here.

~~~
dikaiosune
Certainly not. I quite enjoy the features I get from having my data run
through a lot of systems. However, you don't need to be a Luddite to recognize
that sometimes some users of data (like sometimes Google) aren't as
responsible or considerate of the privacy implications of their actions as
they should be.

When their primary financial incentive is user engagement to sell ads, I have
a hard time believing that the latter doesn't influence the use of my data as
much as the former.

I still use a variety of Google products all the time, but when they're
training machines to understand the meaning of my emails, I begin to think
perhaps that's beyond my comfort zone. I think that machine learning has
tremendous things to offer society, but I'm not sure I'm super excited about a
company using smartish machines to pull meaning and context from my emails so
that we can save some percentage of users 10 seconds when they reply to
something. I get that in most cases this isn't ever going to have a practical
effect, but I would be a lot more comfortable if our society's data stewards
were less blase about that side of things.

~~~
TeMPOraL
I see your point better now, thanks for clarifying.

> _When their primary financial incentive is user engagement to sell ads, I
> have a hard time believing that the latter doesn 't influence the use of my
> data as much as the former._

I don't agree with it in case of Google. I may be mistaken, but my impression
was that they sort of separate their products into two groups - the ad-related
are earning the money, and then the money is spent on funding something else
(like GMail), with little direct connection between the two groups. That is,
products like GMail certainly help Google earn more money on ads, but are not
themselves optimized for ad-related purposes.

> _I think that machine learning has tremendous things to offer society, but I
> 'm not sure I'm super excited about a company using smartish machines to
> pull meaning and context from my emails so that we can save some percentage
> of users 10 seconds when they reply to something._

I think you're seriously underestimating the potential for productivity gains
here. Mobile use case is perfect for replying to e-mails, but mobile
experience totally sucks. This service, if it works as advertised, will be
probably saving not 10 seconds, but something like one minute _per e-mail_.
That + friction reduction have enabling properties that make people do things
they didn't before (e.g. I often don't reply to e-mails on a phone only
because it's too slow, opting to browse Facebook instead), and aggregated that
can help liberate _a lot_ of time.

------
davedx
Meanwhile, Twitter change their favourites from stars to hearts.

------
deckar01
I kept imagining it would generate responses based on MY past emails until the
end of the article. Example "When does your flight leave for Paris?", I was
imagining it would pull the info from Google Now to reply "Friday at 9am", but
I don't think it is built for this type of specific question.

~~~
benplumley
Knowing Google, they'll be working on that by the time they release this.

------
mattgmg
I love you.

 __* This is an automated response __ _

------
hokkos
Does the responses generated from one user response leaks to another one ?
Because this could be a major privacy flaw.

------
keshav57
> The solution was provided by Sujith Ravi, whose team developed a great
> machine learning system for mapping natural language responses to semantic
> intents. This was instrumental in several phases of the project, and was
> critical to solving the "response diversity problem": by knowing how
> semantically similar two responses are, we can suggest responses that are
> different not only in wording, but in their underlying meaning.

Does anyone know where I can find out more about this? A paper or something
maybe?

------
OkGoDoIt
This looks more powerful than iOS's context sensitive keyboard, but that also
looked impressive when demoed. It will be interesting to try this out with
real-world emails and see how well it works in practice.

This gets really exciting when implemented on a smartwatch. Single tap
responses that are actually useful would make smartwatches significantly more
powerful. Speech recognition is great but it's not always appropriate for
social environments.

------
alphydan
> Another bizarre feature of our early prototype was its propensity to respond
> with “I love you” to seemingly anything

Is it trained mostly on personal gmail accounts?

~~~
x5n1
I love you! We should go out some time.

~~~
ewzimm
Would like to see the Maeby Funke addon where "Marry Me!" is always the first
response.

~~~
jethro_tell
Baby Sit me!

------
denniskane
I have a concern that is somewhat related to this issue. I've been wanting to
play around with doing NLP in the client using the SpeechRecognition API
available in Chrome. It typically gives pretty good results for simple
recognition purposes, but there is not yet any way to specify an arbitrary
grammar for the backend service to use. This is a big deal when someone wants
to be able to recognize a name spelled like "Bobbie" rather than "Bobby". The
W3C Web Speech spec currently allows for setting arbitrary grammars, but it
isn't implemented. I'm just saying it would be cool if some of you AI geniuses
at Google could help me out on that front.

You can play around with the dumb little AI app that I've been working on that
does a little bit of NLP in the browser. Check out this site with Chrome:
[https://yc-prototype.appspot.com](https://yc-prototype.appspot.com). His name
is Bertie. You should see his face on the bottom of the page. The site works
like an OS in your browser.

------
novalis78
Interesting. I used a sequence to sequence methodology about 10 years ago in a
chatbot that won in a self-learning competition ... of course never had the
data available that google has.

A bot like that is a great tool to experiment with NLP and ML ideas. This is
really an intriguing area of research.

[https://thinkingai.wordpress.com/](https://thinkingai.wordpress.com/)

------
julianozen
Where's the "sorry, I won't be able to make it." in the server dropping email.

------
tabrischen
When you receive a call today, you are already prompted with auto response
messages such as 'I'm busy, I'll call you back.' This seems like the natural
next step.

~~~
alttab
Imagine the day when it suggests an insult for a co-worker based on his or her
perceived inferiority.

------
drzaiusapelord
> But replying to email on mobile is a real pain, even for short replies.

Imo, this has already been solved. I just whisper replies into my watch for
both email and SMS. Google's voice recognition is finally good enough for
this. I can't think of anything more convenient.

I'm not sure if AI-ish replies are something I'm interested in. Throwing
complex fuzzy logic at what should be a hardware/interface problem seems
shortsighted. I prefer a much higher level of granular control, especially if
these messages are work related.

------
jes5199
Did they really write this whole article about machine-generated text but not
actually include an example of the actual reply the system generated?

~~~
gosub
There are three example responses at the bottom of each picture, above the
navigation buttons.

------
jasonmorton
This is almost exactly the singularity-igniting innovation described in the
sci fi book "Avogadro Corp" by William Hertling.

------
rasz_pl
What could possibly go wrong?

dudette:are we still on for dinner with my mom?

dude:sure, cant wait to see mrs.Plinketon again.

dudette gets suggested reply message generated from one of dude's earlier
emails to his friends: "mrs.Plinketon has great tits!"

It is amazing to realize this type of technology can be bootstrapped only once
(?)- as soon as you start getting automated replies in the system quality of
learning on past data will drop, at some point snake will be eating his own
automated reply tail.

I imagine this system would benefit greatly from optional additional user
voice input, something as simple as 1-3 spoken words fed back into Reply
network to help with the reply intent decoding. You get email, NN generates 3
replies it thinks you would like, you have different idea for a reply, click
microphone icon and say "no, maybe next week", Reply network rebuilds
suggested answers with your input shaping reply vector.

------
mapleoin
Finally! This will dispel all those rumours that Google reads all your email.

~~~
shpx
Why do you care if a computer reads through your email? Its not like Larry
Page is looking through your search history. AI seems to just be data
expressed in some way, ei me and you are a physical expression of years of
gigabit streams of data. This will only become more strongly felt as we get
closer to serious AI.

Sadly I think its something you either get used to or go live in the woods.

~~~
pluma
Same argument as the NSA: "It's not surveillance until a human looks at the
data"

~~~
marvy
Interesting point, but there's a potentially important, albeit somewhat subtle
difference. Google collects data which they claim nobody will ever look at.
The reason they collect lots of data is so that machines can look at all of it
now, while the NSA collects lots of data because they think that a person
might need to look at some of it in the future.

~~~
pluma
If anything, that makes Google's bulk data collection more suspect than the
NSA's. Not only do they collect it en masse, they also process it.

(Note that I personally think Google's collection is more ethically defensible
than the NSA's, if only because they at least seek implied consent whereas the
NSA doesn't even have to pretend)

------
barnacs
Privacy issues aside, please don't. These kinds of features are outright
harmful to our society.

Human communication and our social skills in general have already detoriated a
lot by replacing face to face conversations with audio only, then text.

This is just further taking away the incentive to sneak in something personal,
something human into our written communication every now and then.

IMO, the very reason someone takes the time to type out these kinds of
questions as an email message is to make it more personal. Otherwise, they
could just use an rsvp/calendar system or a bug tracker for the particular
queries in the article, which already enable you to send such generic replies
with 1 or 2 taps at most.

Can we stop misusing technology to dumb down our everyday lives and stop
transforming our society into a bunch of isolated individuals living on the
same planet?

------
yeukhon
Minus all other concerns, for the operation / infrastructure / devops, the
auto-replying about server downtime or inquiry about performance, those kinds
of stuff is something I am actively thinking about and starting to learn how
to implement in my organization.

------
Animats
Slowly, the personal assistants will get smarter and do more of your job. Then
you get laid off.

------
MikeTV
There's a quip at the bottom: "This blog post may or may not have actually
been written by a neural network."

Has there been any progress in AI-driven/assisted creative writing (other than
sportscasts)?

------
sagarjauhari
Similar to the quick text feature when not taking a phone call, this could
come in handy - especially on phones.

But does that mean I'll have to re-enable Inbox to try it. Duh :(

------
sandworm101
"...In developing Smart Reply we adhered to the same rigorous user privacy
standards we’ve always held -- in other words, no humans reading your email."

That the humans at Google may read my email is the least of my privacy
concerns. That the robots will pass them along to other robots at places other
than google is the real worry. But as no such agency exists, why worry?

------
WWKong
The real potential here is eliminating human interaction. On the sending end
machine should auto compose, auto target a mailing list, send it out. On
receiving end machine should scan incoming email, compose a response, send it
out. That would cover 95% of current email traffic.

------
scottm30
I'd love to see this extended in future to include:

\- using mannerisms I have used with the sender in past emails

\- scan my calendar schedule and have an idea of my availability and use that
in the response

The product in its current form sounds great and I'm keen to give it a try.

------
killion
I'm not sure I trust that this will work that well. Because I still get other
peoples flights added to my calendar automatically if they forward their
itinerary to me.

------
JabavuAdams
Are these kinds of projects only available in Mountain View?

------
rdl
This would seem to be an area where senders could alter behavior to make
things work better with this...maybe being explicit about the question and
options.

------
msoad
It's not available for Google Apps users or am I missing something? It doesn't
show any replies to me...

~~~
rgawdzik
"available later this week", they should of released this blog post when it
was available.

------
zhanwei
next step, check your calendar for scheduling conflict and suggest the right
response, i.e., "i'll be there"/"i can't make it".

further extension, pull out quick answers from other emails, i.e., "what's the
sales number this month?" "100M" from another email

------
ambicapter
Sounds like the system has no way to take into account context. Each message
is encoded into a 'thought' vector, but is it difficult to come up with a
'thought' that changes meaning depending on prior messages? I imagine it is
such a concern that leads one to look into Hierarchical Temporal Memory and
similar techniques.

------
hmate9
Maybe shortly we'll just have neural networks talking to each other on behalf
of us.

~~~
ocdtrekkie
Everyone has "people" now. Having "people" is no longer for the rich. But
they're also not really people.

------
ksk
I guess I've already automatically consented to Google using my emails to
train their algorithms. How does filing bugs work in the scenario where the
machine learning can't figure something out? Is a copy of someones personal
email attached to the bug report?

------
shrikrishna
In a parodical reality with mistaken pressings of the "two buttons to reply"
feature

> Obama: Hey Hillary, what do you think, should we bomb <X>?

> Hillary: Sure! Count me in!

> Hillary: Hey! Don't consider that last email! Damn auto reply...

That aside, as a fellow nerd, neural net FTW!

------
pjmlp
Waiting for the news about social and legal problems that will happen from
typing on the wrong answer.

~~~
willhinsa
You can anyways undo the sent message! A great feature of Inbox.

~~~
pjmlp
No you cannot, it is not part of the email protocol.

Once it is sent, it is gone.

I always have fun with those "Please delete messages" from Outlook sent to my
email client.

~~~
BrandonY
Gmail does not literally unsend SMTP messages. To give you time to undo, Gmail
delays sending the message for a few seconds. So if you don't select "Undo"
within the time limit, your message will be sent.

How to use:
[https://support.google.com/mail/answer/1284885?hl=en](https://support.google.com/mail/answer/1284885?hl=en)

~~~
thrownaway2424
Note these instructions are for how to enable this on Gmail, the original.
Inbox does this by default.

------
Sevzinn
Senior research scientist? So, a forgetful research scientist?

