
Limits on Matrix Multiplication - beefman
https://rjlipton.wordpress.com/2018/08/30/limits-on-matrix-multiplication/
======
costrouc
I didn't see in the paper why these methods are not used in practice. There
are parallel implementations of strassens algorithm out there (I have written
one). Yes they are faster but they do not guarantee stability. Also they are
significantly harder to parallelize than block cyclic MM.

~~~
throwawaymath
_> There are parallel implementations of strassens algorithm out there_

Do you have one off the top of your head you could link the code too? Doesn't
have to be your implementation specifically. I just want to read the code for
a parallel implementation.

~~~
jhanschoo
Parallel Strassens sounds like just a log2 n sequence of map-reduce jobs

------
salty_biscuits
I hope it is 2. That would be surprising.

~~~
dan-robertson
I’m not sure it would be surprising if it’s 2. Many people believe it is 2, or
at least can be less than 2+epsilon for any epsilon > 0.

I would however be impressed if it were constructively proven to be 2

~~~
coherentpony
And the constant wasn't so huge as to be impractical.

------
foxes
What is the connection with the continued fraction?

------
std_throwaway
It seems that not very many mathematicians are working on optimizing this
problem but matrix multiplication used every day by many people and companies.

Did we already exhaust economics of scale on this one?

~~~
dan-robertson
Well most matrix multiplications tend to be small matrices (eg 4x4) and the
time complexity of multiplying 4x4 matrices is O(1). These _are_ actively
optimised in the practical, hand-written assembly and new hardware, sense.

If one has to multiply very large matrices then the goal is usually not so
much to optimise matrix multiplication in general but to optimise the
multiplication of the subset of matrices one is interested in. In these
situations one cares about eg density and symmetry and choice of basis. There
is a lot of information in a large random matrix and there is typically less
information in an interesting large matrix. Therefore the goal is to skip the
work required by all the entropy one does not have

~~~
taeric
This is no longer there case, given deep learning, though. Right? Effectively,
the training process can be seen as a giant matrix multiplication in one step.

~~~
dan-robertson
If deep learning we’re just a big matrix multiplication then it would not be
deep, it would be linear. The (eg sigmoid) functions mixed between the layers
make it deep. I’m not really sure what it is about the training that you think
is just a matrix multiplication.

It’s also worth noting that eg a convolution can be written with matrix
multiplication (or rather some tensor products and contraction but there would
be much redundancy from the fact that distant points cannot influence each
other and the same thing is done to each point in the convolution. This is
much less general than matrix multiplication

~~~
stochastic_monk
If floating point values were real numbers, simple matrix multiplications
would not produce nonlinearities.

However, researchers at OpenAI demonstrated that the use of subnormal floating
point values and their discontinuity [0] provides sufficient nonlinearity to
learn nonlinear associations.

[0] [https://blog.openai.com/nonlinear-computation-in-linear-
netw...](https://blog.openai.com/nonlinear-computation-in-linear-networks/)

~~~
crankylinuxuser
Then, would it be better to store rationals as 2 ints with understood
division, and irrationals as 2 ints as understood powers?

Why are we working with these numbers so imprecisely, and with such
discontinuity? When I've done math work in school, I always worked with the
primitives without approximating.. 2^.5 was handled as such unless asked for
an approximate answer.

I also ran across this in AutoCAD. Because they force arbitrary precision, one
cannot define a sqrt length. 1.414 doesn't reach all the way on a r-triangle
with 1,1 .

------
kibwen
The OP's WordPress install appears to be infected with malicious adware; every
time I load the site I get quickly redirected to a random one of a number of
scummy pages which hijack the back button, have autoplaying sound, display
false error messages, and attempt to ask for browser-level permissions.

~~~
fermienrico
Why does the Ad industry suck so much?

Put some nicely formatted text such as Google ad results and clearly mention
it as an ad. WHY the fuck do ad companies have to do insanely annoying
borderline unethical things? Why is this industry so rotten to the core?

~~~
nabla9
Because suckers who are not using not adblockers are the only market segment
for these ads.

In this time and age, every computer literate person install adblocking in
every system or asks someone to help to do it and almost nevers sees these
ads. What is left is people who are more likely to fall for scams.

