
Graph-Powered Machine Learning at Google - tim_sw
https://research.googleblog.com/2016/10/graph-powered-machine-learning-at-google.html
======
bnchrch
While there is a lot to get excited about with ML both as a consumer and as a
software developer I can't help feeling a pang of sadness.

Big data and in this case the relationship (graph) between big data points are
whats needed to make great ML/AI products. By nature the only companies that
will ever have access to this data in a meaningful way are going to be larger
companies: Google, Amazon, Apple, ect. Because of this I worry that small
upstarts may never be able to compete on these type of products in a viable
way as the requirements to build these features are so easily defensible by
the larger incumbents.

I hope this is not the case but I'm getting less and less optimistic when I
see articles like this.

~~~
rspeer
The graph algorithm they're describing is basically Manaal Faruqui's
"retrofitting", although they don't cite him. I will make the charitable
assumption that they came up with it independently (and managed to miss the
excitement about it at NAACL 2015).

Here's why not to be sad: Retrofitting and its variants are quite effective,
and surprisingly, they're not really that computationally intensive. I use an
extension of retrofitting to build ConceptNet Numberbatch [1], which is built
from open data and is the best-performing semantic model on many word-
relatedness benchmarks.

The entirety of ConceptNet, including Numberbatch, builds in 5 hours on a
desktop computer.

Big companies have resources, but they also have inertia. Some problems are
solved by throwing all the resources of Google at the problem, but that
doesn't mean it's the _only_ way to solve the problem.

[1] [https://github.com/commonsense/conceptnet-
numberbatch](https://github.com/commonsense/conceptnet-numberbatch)

~~~
igravious
I can't find a single comment thread on HN where ConceptNet is discussed.
Bummer. See here:
[https://hn.algolia.com/?query=conceptnet&sort=byPopularity&p...](https://hn.algolia.com/?query=conceptnet&sort=byPopularity&prefix&page=0&dateRange=all&type=story)

Short of signing up to the mailing list on Google Groups is there any info out
there on the net that tells me:

a) How ConceptNet relates to efforts like Cyc's OpenCyc and the big G's
Knowledge Graph?

b) How to add domain specific concepts, attributes, and relations to
ConceptNet

Both these things are the first that pop into my mind but the website:
[http://conceptnet5.media.mit.edu/](http://conceptnet5.media.mit.edu/) doesn't
seem to cover them.

~~~
rspeer
Hm, I know I've brought it up before.

In relation to OpenCyc: ConceptNet doesn't have CycL or any sort of logical
predicates -- the assumption is that you are just doing fuzzy machine learning
over ConceptNet. I'm pretty sure you need the rest of Cyc to do anything
interesting with CycL, though.

ConceptNet takes advantage of linked open data when it can, so it _does_
contain attempted translations of the facts in OpenCyc.

Compared to the Google Knowledge Graph: well, you can't really use the
Knowledge Graph without being at Google, or making a research agreement with
them, right? But from what I've seen of it, and of its predecessor Freebase, I
think it focuses a lot on named entities: particularly things you can look up
on Wikipedia or things you can buy. Which is fine information. I'd say it's a
different segment of linked data than "what words mean", which is ConceptNet's
focus. So maybe think of it as more like a bigger WordNet than a smaller
Knowledge Graph.

How to add domain-specific concepts to ConceptNet: the unsatisfying answer is
that you can get the code for ConceptNet and alter the build process to
include new data sources (try it on the 5.5 branch if you attempt this). And
the startuppy sellout answer is that doing this automatically is one of the
things my company Luminoso is for
([http://www.luminoso.com](http://www.luminoso.com)).

Thanks for asking, and I'll try to clarify this kind of stuff when I deploy
the new site.

~~~
igravious
Thanks for the response. I'll keep an eye on your progress, dig deeper into
what you already do, and have a play with your tools.

------
nicolewhite
I'm a big fan of the smart replies in Allo and Inbox so this was a fun read. I
did something similar in grad school where I and some other students manually
labeled a handful of sentences and then used graph-based semi-supervised
learning to label the rest for the purposes of using it as a training dataset.
It would be neat to hear what technology they used for the graph-based
learning; perhaps Cayley? We used Python's igraph at the time but it was
pretty slow. It would also be interesting to try this in Neo4j.

------
BinRoo
The algorithm is well known. The main contribution is that they were able to
scale up the algorithm to a huge graph. Google doesn't go into much detail
about that because it's proprietary. So whats the point of this blog post? Oh
yea, free Allo/Inbox advertisement.

------
harigov
This is really cool. The underlying graph optimization algorithm seems similar
to how graphical models are trained. Is that correct? Can someone please help
me understand the difference?

~~~
fritzo
Yes, label propagation is very similar to belief propagation on a Gaussian
Markov random field (a particularly simple type of graphical model). To be
precise this is 'inference' on the graphical model, rather than 'training' or
'learning'. See the author's 2016 JMLR paper for more details:
[http://www.jmlr.org/proceedings/papers/v51/ravi16.pdf](http://www.jmlr.org/proceedings/papers/v51/ravi16.pdf)

------
mark_l_watson
Great stuff. I have been using machine learning since the late 1980s. Semi
supervised learning is new to me, and it is exciting to be able to mix labeled
and unlabeled data.

------
zitterbewegung
Anyone know if they are going to release code in Tensorflow that works with
public data? I have been using the inbox reply feature in inbox and its very
useful.

~~~
akssri
These algorithms can't really make much use of Tensorflow.

------
onetwo12
I see this as a knn approach in which the distance is a function of the
strength of the edge between the vertices.

In the retrofitting paper cited in the comments there is a process of
smoothing, that is feeding back the message or information to update the
states of the graph (example in the modern ai book). It doesn't seem anything
new.

------
sandGorgon
is there something in Expander that makes this happen... or can something like
Apache Spark be used as well ?

------
b3n
As a software engineer who knows nothing about ML, where's the best place to
learn?

------
prohor
Is there implementation released maybe? Similar like they did with TensorFlow.

------
yalogin
Unrelated but none of the google blog links open up on safari in iOS. The link
opens but text doesn't show.

~~~
lis
This is probably due to your ad blocker. Same here. The page doesn't work if
you disable the tracking. After disabling Focus it's working.

~~~
protomikron
That this is the case and explanation is unacceptable (not that I blame the
parent, but I would like to add the statement, as we might otherwise forget,
that information could be available without the tracking-mafia).

