Don't read that. That's one of the most horrible papers I know, a hodgepodge of mathematical well-known concepts thrown together with some vague ideas how they connect. The mathematical parts are explained better in any undergraduate book. I'm not sure why the author felt the need to expound on what a manifold is when that has been done better in hundreds of other texts - literally.
And it lacks a definite conclusion: They don't prove anything, don't make any particular experiment, but just loosely talk about how these ideas might be relevant to machine learning.
I'm surprised that such highly cited researcher have produced such a paper. I would be embarrassed to be on it - and I'm embarrassed on behalf of the ML community that they are citing it.
Great selection of works, but I am missing a lot of references from topology in ML, with the article only assuming a very cursory perspective in terms of 'topology captures connectivity and/or continuity.'
Some works from my colleagues and me go a little bit deeper (no pun intended), for instance:
One common theme I see in the paper(e.g. in protein folding) is:
"Identify what properties are important (geometry, algebra, topo) and which one is an useful prior and then "use" the guide to select an initial struct. This is probably harder than it sounds(unlike bayesian priors which are more forgiving for one to select, but quite like them in that they both require special assumptions)."
I wonder: could one use it to bring together certain multimodal data and a proposed network for a task? Like could one bring in sensor, map topology, urban topology, pictures which have certain properties and that help me use this guide to make a statement like : "Street data could be embedded with Sensor data to do ABC kind of inference using XYZ NNetwork structure because this paper suggests that is a reasonable thing to do"?
All machine learning is just embedding of various forms. If you have a way to translate disparate types of data into a common space, in ways that preserve inductive bias and information content, you can then combine them for downstream tasks.
I am 100% convinced that these kind of approaches will be what delivers ML research from the current resource-hungry and ungeneralizable status quo. Low-dimensional Euclidean geometry is special. Higher-dimensional Euclidean spaces are less special. Most real-life data is high-dimensional, not at all smooth, and possessing a structure you cannot call Euclidean with a straight face. Look at what works with tabular data (which is probably most of what practitioners work with in the wild). It's gradient boosted trees, not neural networks.
There is a fundamental mismatch between the data we usually work with and the spaces we shove it into. Tools from algebraic topology and geometry are old hat in physics. If anything, they should be even more useful in ML.
Is geometric, topological, and algebraic ML/data analysis actually used in the industry? It is certainly beautiful math. However, during grad school I met a few pure math PhD students who were saying that after finishing their PhD they will just go into industry to do topological data analysis (this was about 10 years ago and ML wasn't yet as hyped up). However, I have never heard of anybody actually having success on that plan.
As someone who did an applied math PhD before drifting towards ML, it's worth pointing out that these applied math groups typically talk about applications, but the real question is whether they are actually used for the stated application in practice due to outperforming methods that use less pretty math. Typically (in every case i have seen) the answer is "no", and the mathematicians don't even really care about solving the applied problems nor fully understand what it would mean to do so. It's just a source of grant-justifiable abstract problems.
Thanks. That's certainly very interesting. Albeit it seems to me that the number of jobs doing geometric and topological ML/AI work in the drug or protein design space would be quite limited, because any discovery ultimately has to be validated through a wet lab process (or perhaps phase 1-3 clinical trials for drugs) which is expensive and time-consuming. However, I'm very uninformed and perhaps there is indeed a sizable job market here.
I think the job market in general for this kind of stuff is "small"; but you can find jobs. Look at Isomoprhic Labs for example. There are new AI/ML companies that have emerged in recent years, helped by success of things like AlphaFold. I think your question is really: does this research actually creates tangible results? If it did, it would be able to create more jobs to support it by virtue of being economically successfully and therefore growing?
Hyperbolic embeddings have been an interest of mine ever since the Max Nickel paper. Would love to connect directly to discuss this topic if you're open. here's my email: https://photos.app.goo.gl/1khCwXBsVBuEP6xF7
Not much to discuss really, I just monkey patched a different metric function, then results for our use case became substantially better after training a model from scratch on the same data compared to the previous euclidean model trained from scratch.
I'm currently working on massive multi agent orchestration so don't have my head in that side of things currently.
Can you share what kinds of problems were conducive to hyperbolic embeddings in your experience. Also, separately, are you saying companies are using these in practice but don’t talk about them because of the advantage they give? Or am I reading too much into your last sentence.
They are better at separating clusters and keep the fact that distances under the correct metric also provide semantic information. The issue is that training is longer and you need at least 32, and ideally 64 bit floats during training and inference.
And possibly.
The company I did the work for kept it very quiet. Bert like models are small enough that you can train them a a work station today so there is a lot less prestige in them than 5 years ago, which is why for profit companies don't write papers on them any more.
I don't think there's much use currently. But I kinda like the direction of the paper anyway. Most mathematical objects in ML have geometric or topological structure, implicitly defined. By making that structure explicit, we at worst have a fresh new perspective on some ML thing. Like how viewing the complex numbers on a 2d cartesian plane often clicks more for students compared to the dry algebraic perspective. So even in the worst case I think there's some pedagogical clarity here.
some people on this thread are asking about jobs. The bigger picture here is that previously intractable problems are going to be solved with a new combination of math, data and compute.. there are lots of commercial cases that will change dramatically. How can individual people or small groups benefit from serious problem solving, economically?
Why are we so sure that a lot of "previously intractable problems" are/will be solved with this family of methods? (and I mean real-life/real-world problems, not toy problems constructed specifically to show the proposed methods in the best light possible in the research papers)
Of course others above have pointed out drug or protein design as a potential area, but there still seems uncertainty as to the practical impact on the real world. Other than that, I don't see areas of impact for these approaches so far.
One of the authors here. Thanks for your comment! While a lot of this research is theoretical and does not have immediate use cases, we have tried to summarize some of them in the last paragraph of the paper (VII. Applications of Non-Euclidean Geometry). See page 26. We present some in Chemistry and Drug Development, Structural Biology and Protein Engineering, Computer Vision, Biomedical Imaging, Recommender Systems and Social Networks and Physics.
Note that the GPU hardware is setup for Euclidean matrix operations. Even if you had a deep structure learner, it won't necessarily help you if you have to go back to emulating it on Euclidean hardware.
- Geometric Deep Learning Grids, Groups, Graphs, Geodesics, and Gauges: https://geometricdeeplearning.com/