
Word2vec: Fish + Music = Bass - tobr
https://graceavery.com/word2vec-fish-music-bass/
======
xg15
So. Could anyone explain _why_ it is doing that?

My (very rough) intuition to what w2v does is that it associates each word
with a point in a space, then moves the points around so that their distances
roughly correspond to how often they are used together with similar other
words. (I.e., two words that both are often preceded by "It happened in" would
be put close to each other, while words occurring into very different
sentences would be placed further apart.)

I can see how this could give some semantic meaning to _clusters_ of words -
however, I'm still baffled as to why it gives meaning to _directions_ within
the vector space.

I also wonder how consistent it is. Seems to me it would be interesting to
investigate the directions further. E.g., are all of them meaningful? What
happens if I slightly rotate or scale a direction? How much transformation can
I apply before it changes its meaning?

Are there more complex shapes in the space that also have meaning?

~~~
GistNoesis
The "trick" is that word vectors are just projections of sparse tensor.

For example the word "queen" can be represented by its attributes.
Representing in triplet form from wikidata Q16511993 (queen, subclass
,monarch),(queen, subclass, female),... A king would be (king, subclass
,monarch),(king, subclass, male),

Transitioning from queen to king is just changing an attribute in this triplet
space, i.e removing a triplet and adding another. All those operations are
linear. And then you take the projections.

This is kind of similar to the kernel trick of SVMs. You do the calculations
in a lower dimensional space (word vectors) but implicitly you are doing them
in the higher dimensional space (graphs of relations, aka triplets lists).

~~~
joycian
The sparse tensor perspective intrigues me, could you perhaps provide me with
some references?

~~~
mlevental
that person's description is unnecessarily jargony. sparse tensor just means
vector that's mostly zero and has some ones. take your corpus of concepts and
number them 0 to n-1. then each word you want to turn into a vec turns into a
vector that has ones for concepts it's associated with and 0 elsewhere.
learning that association is what word2vec does.

~~~
rytill
Can you explain attention in this same way?

~~~
mlevental
nah i don't know much about LSTM but this looks like a fine tutorial

[https://towardsdatascience.com/intuitive-understanding-of-
at...](https://towardsdatascience.com/intuitive-understanding-of-attention-
mechanism-in-deep-learning-6c9482aecf4f)

------
markchen90
These analogies need to be taken with a grain of salt. For instance, king -
man + woman actually returns king (since woman - man ~= 0). It's only queen
when you ban the words king, man, or woman (since the embeddings of queen and
king are close).

See Ian's tweet
[https://twitter.com/goodfellow_ian/status/113352818965167718...](https://twitter.com/goodfellow_ian/status/1133528189651677184)
or the linked paper
[https://arxiv.org/pdf/1905.09866.pdf](https://arxiv.org/pdf/1905.09866.pdf)
for more information.

------
userbinator
The bass/bass confusion seems to have permeated Chinese machine translation,
as a search for 低音魚 (literally "low frequency fish") will show.

Another amusing one there that shows a similar ambiguity is: pig : oink ::
mouse : keypress

------
mkl
The typo in "pig : oink :: Fred Flinstone : wassup" bothered me, but even
"Flintstone" doesn't produce the "Yabba dabba doo" that Woody Woodpecker (!)
produces: pig : oink :: Fred Flintstone : mumble mumble.

If you're wanting to try it, you need about 6GB of RAM free.

------
personjerry
I'm trying to replicate this on my Macbook and it's causing a lot of lag, and
taking a long time. Anyone else have this issue, or might be able to help me
debug this? I'm assuming I just don't have enough RAM (I only have 8GB here)?

~~~
mkl
It took several minutes to load off my old SSD, and took 6GB just to load the
data. Trying out a bunch of word combinations pushed it up to ~9GB I think. So
yeah, it seems like you just don't have enough RAM.

