
The tensor renaissance in data science - datascientist
http://radar.oreilly.com/2015/05/the-tensor-renaissance-in-data-science.html
======
chestervonwinch
Every time I get hear about tensors:

"It's just a multidimensional array!"

Ok, simple enough. But then I go to wikipedia get hit with differential
geometry, manifolds, multi-linear algebra, etc...

I mean these are all topics I've been meaning to learn for a while, but each
one seems like it would take a year to properly introduce myself to (for which
I never seem to have the time).

~~~
krastanov
Tensors are most definitely __not __just multidimensional arrays. (this is a
pet peeve of mine)

Tensors have geometric meaning just like vectors (just like vectors are not
just single-axis arrays). It would be too long to delve into the geometry
(especially if it is abstract "geometry" like the geometrical space spanned by
the songs in the Pandora database or the space spanned by the faces in a face-
recognition software). Instead I will try to give an example:

You all know what rotating a vector means. All the components change in
certain way, but the geometrical meaning of this object is preserved in some
sense. This is not true for some arbitrary list of numbers. Well, a tensor is
just a more complicated geometrical structure that still has very specific
rules governing what its components can be and how they can change.

And a counterexample: the Christoffel Symbol is a multi-dimensional array that
is not a tensor.

P.S. It is true that it is a very common abuse of nomenclature to call all
multidimensional arrays "tensors".

~~~
chestervonwinch
It's true though that vectors are basically 1D lists if we're speaking about
elements from the vector space R^n. Is it true that tensors are
multidimensional arrays if they come from the space R^{m1 x m2 x ... x mn} ?

I'm also confused: do you think about a tensor as a vector from vector space?
as a object that maps between vector spaces? or something completely
different?

I realize there's a LOT of detail here - just trying to flesh out my
500,000,000 mile view a little more.

~~~
im3w1l
They use tensors here to mean simply multidimensional array. And as this
discussion shows, that choice of words is unfortunate and confusing.

~~~
rspeer
What I'm seeing is that physicists and geometers disagree with programmers
about what "tensor" means -- not in that their views are inconsistent, but in
that the programmers' view is too limited for the things that the physicists
and geometers want to do with tensors.

They _also_ disagree about what "vector" and "matrix" mean for the same
reason, but they can't really put that horse back in the barn. But "tensors"
remain unfamiliar enough that this disagreement can come up whenever anyone
says "tensor".

Here's a possible truce:

\- Programmers should default to talking about "arrays"

\- Geometers and physicists should default to talking about "vectors",
"matrices", and "tensors"

\- We all generally recognize that some (vectors|matrices|tensors) are arrays,
and some arrays are (vectors|matrices|tensors), and it's not the worst thing
ever if someone uses one to mean the other

\- We all generally recognize that you can sometimes do _damn cool things_ by
using an array as a tensor or a tensor as an array, so there isn't that much
benefit in shouting "NO THEY'RE TOTALLY DIFFERENT".

~~~
tel
Eh, I think as you stack more and more "structure" atop something it becomes
less and less appropriate to confuse it with its representation. There is
afterall already a completely appropriate, cross-disciplinary term for what
"programmers" here call "tensors": multidimensional arrays, or even nd-arrays.

No mathematician will bat an eye if you say that a tensor can be represented
by an nd-array---but if you equate them, then you're clearly missing
_something_.

~~~
dvanduzer
So... Do "programmers" think of "matrix" and "tensor" as "synonymous"?

edit: that was actually a serious question, because i really don't understand
this idea that programmers have a common notion of what a matrix is.

~~~
tel
I didn't mention the term "matrix". I think it has a much more complex story
with respect to common use by "programmers" along with vector. Since linear
algebra is such a commonly taught subject there are lots of people with
practical working knowledge of matrices. If you don't have the geometric or
abstract algebraic perspective here then matrix is almost certainly "2d-array"
at first blush, but you'll also admit that there's a sort of non-trivial
relationship between these "matrices" and size-conformant "vectors"
(1d-arrays) and another one between two size-conformant matrices. This
distinction is made from time to time [0][1].

Finally, of course, a matrix is a special case of a tensor. Tensor is such a
broader topic that it becomes more and more difficult to talk about it without
its geometric underpinnings. This is what I was writing about above.

[0] Is a black-and-white image a matrix? I'd imagine a lot of people would
have a bit of a tough time saying yes directly---and for good reason. But
there's also a reasonable argument for pretending like it is, and even
reasonable arguments for turning it into one!

[1] Numpy distinguishes between nd-arrays and matrices exactly right in that
matrices are 2d-arrays only and are imbued with multiplication as composition
of maps. In practice, many people I know who are Numpy users just think of
matrix as a convenient way to overload *, though.

------
dxbydt
Luckily, I was at mlconf last week where Dr.Anandkumar spoke - they called her
the "tensor lady" :) She's using tensors in machine learning for a bunch of
things -

Latent Variable Models: Training LVM's using local search methods like EM,
gradient descent, variational bayes etc. have a bunch of problems - they get
stuck on local minima, the algorithms are hard to parallelize with poor
convergence. In these cases, tensors yield guaranteed learning using
embarassingly parallel algorithms, so faster convergence & can be run on
Spark.

Also saw a demo on training 2-layer nets for GMM using tensors, and they
learnt the weights rather fast. So using tensors in deep learning shows
promise, though the techniques are in their infancy.

One of the challenges the professor mentioned was the availability of open
source libraries to do tensor decomposition, which the above methods require.

It was a very successful talk -
[https://twitter.com/cdubhland/status/594220061025435649](https://twitter.com/cdubhland/status/594220061025435649)

Tensor slides: [http://www.slideshare.net/SessionsEvents/animashree-
anandkum...](http://www.slideshare.net/SessionsEvents/animashree-anandkumar-
electrical-engineering-and-cs-dept-uc-irvine-at-mlconf-sea-50115)

~~~
tachim
This isn't quite right. Moment methods (that rely on tensor decompositions)
have a few problems:

(i) they have convergence bounds, but in practice need more data than we have
available

(ii) they don't do as well as EM usually, but using them to initialize
parameters for EM sometimes does better than EM with random initialization
schemes

(iii) it turns out variational methods can also be embarrassingly parallelized
without losing much accuracy in practice

(iv) right now moment methods don't work for arbitrary graphical models

~~~
dxbydt
I believe while you are right in general, she is looking at a class of
problems for which tensors handily triumph other methods. You might be
interested in these papers -

[http://newport.eecs.uci.edu/anandkumar/pubs/powerdynamics.pd...](http://newport.eecs.uci.edu/anandkumar/pubs/powerdynamics.pdf)

[http://newport.eecs.uci.edu/anandkumar/pubs/ProvableNN_spars...](http://newport.eecs.uci.edu/anandkumar/pubs/ProvableNN_sparse.pdf)

------
clebio
Does using tensors for storage yield some benefit over the 'panels' metaphor
that Python Pandas affords? It seems like the real power of tensors is in the
calculus that can be run over them, rather than just them by themselves.

If I want to store lists of numbers, arrays and (linked) lists are basically
equivalent. It's when I want to calculate cross- and dot-products that vectors
prove especially useful.

The discussions I've seen recently in data science about tensors seems to only
be about the storage aspect. And, I suppose, if your libraries _don't_ provide
such a storage mechanism, then yes, you'll benefit from adding that
capability. But I keep waiting for more from the ML community about leveraging
differential geometry full-on. I assume it's out there somewhere, in some
shops, but I've not found a lot of discussion about it.

~~~
tel
There's actually quite a lot of this under the phrase "information geometry".
Offhand, I feel like a lot of this work is coming out of Japan.

[http://www.amazon.com/Information-Translations-
Mathematical-...](http://www.amazon.com/Information-Translations-Mathematical-
Monographs-
Tanslations/dp/0821843028/ref=pd_sim_b_2?ie=UTF8&refRID=0E92317F4Y8R9KQWT7JM)

[http://www.amazon.com/Algebraic-Statistical-Monographs-
Compu...](http://www.amazon.com/Algebraic-Statistical-Monographs-
Computational-Mathematics/dp/0521864674)

~~~
clebio
Oh, awesome, thanks for those links!

~~~
tel
No problem!

There's also one more book I was trying to find which basically recasts normal
linear regression in the geometric projection terms that most people know but
that are never emphasized when learning it. The nice part of that method is
that then you can extend the geometric interpretation to manifolds and get, I
believe, GLM very naturally.

~~~
clebio
Thanks again. I'd found this text[1], but it's a bit pricey! [1]:
[http://www.amazon.com/gp/product/0412398605/](http://www.amazon.com/gp/product/0412398605/)

------
riemannzeta
Although not a small amount of effort, it really helps to learn geometric
algebra notation:

[http://www.amazon.com/Geometric-Algebra-Physicists-Chris-
Dor...](http://www.amazon.com/Geometric-Algebra-Physicists-Chris-
Doran/dp/0521715954)

Once you get an intuitive grasp of bivectors, it's not hard to "see" tensors
as a linear map of bivector into bivector.

~~~
cschwan
This is a really great book! It covers a lot of physics and gives plenty of
insight on the relation of geometry and physics. It even made me program this
library: [https://github.com/cschwan/hep-ga](https://github.com/cschwan/hep-
ga) (shameless self-advertisement).

------
jheriko
He keeps saying tensor when he means pseudotensor. It's an important
distinction

Tensors and tensor fields can be even more powerful by expressing non trivial
relationships between components that remain unchanged under transformation
into other spaces... Although whether that applies here I do not really know.

------
animakumar
It is great to see all the discussion. We also have source code for topic
modeling using tensor methods. Look forward to feedback and further code
development [http://t.co/VvWyTmZLps](http://t.co/VvWyTmZLps)

------
nether
Coworker drives a Toyota Matrix, has a license plate frame saying, "My other
car is a tensor."

