
Symbolic Machine Learning - tephra
http://languagengine.co/blog/symbolic-machine-learning/
======
romaniv
_> Historically, symbolic techniques have been very desirable for representing
richly structured domains of knowledge, but have had little impact on machine
learning because of a lack of generally applicable techniques._

Is this generally true? I mean, "impact" can be measured in different ways,
but this paragraph gives impression that symbolic logic was always orthogonal
to ML. However, there was clearly much research in that area.

Here is just one example:

[http://www.doc.ic.ac.uk/~shm/Papers/lbml.pdf](http://www.doc.ic.ac.uk/~shm/Papers/lbml.pdf)

Frankly, I don't understand why the field of symbolic AI was so thoroughly
abandoned. Contrary to popular belief, it did deliver results - a lot of them.
It had a good theoretical foundation and years of practice. It could (with
ease) do a lot of neat tricks, like systems explaining why it made certain
decisions. Not just that, you could implement those tricks _after_ you
implemented core functionality. And most importantly - it was scalable
downwards. You could take some ideas from a complex system, put them into a
much simpler system on vanilla hardware (i.e. normal application) and still
get very interesting and useful results.

~~~
psygnisfive
I think to some extent it disappeared, to some extent it rebranded. But mostly
it disappeared, because commercial motivations pushed a lot of money towards
machine learning research. But that's just a hunch.

------
xjia
This reminds me of a book titled Grammatical Inference [1]. If you are
interested in the blog post, you might also be interested in the book as well.
It's a different kind of "learning" though.

[1] [http://pagesperso.lina.univ-
nantes.fr/~cdlh/book_webpage.htm...](http://pagesperso.lina.univ-
nantes.fr/~cdlh/book_webpage.html)

~~~
psygnisfive
Grammar inference is a very interesting (tho quite different) topic. We have
absolutely no idea how humans do it, and we know that its really really hard
(nearly impossible!) to do given just raw text. Super interesting topic!

------
Seanny123
I'm working in the lab that built Spaun [1] that might be of interest to you.
We use the Vector Symbolic Architectures you talk about in combination with
Deep Learning and other stuff to build brain models.

As far I know, my lab is one of the few places mixing symbolic learning with
Deep Learning. Do you or anyone else know other modern attempts (Reinforcement
Learning?) with this approach?

[1]
[http://science.sciencemag.org/content/338/6111/1202](http://science.sciencemag.org/content/338/6111/1202)

~~~
Eridrus
It took a while to figure out which group you're talking about, but here is
some recent work from another group:
[https://arxiv.org/abs/1609.05518](https://arxiv.org/abs/1609.05518)

~~~
psygnisfive
Oh! Was Murray Shanahan's lab involved with this? Cool. I've chatted with
Murray a very tiny bit on twitter about this all. That paper is part of what
inspired me to write this blog post, because I was dissatisfied with the
approach they propose.

It felt to me like they had found a way to restrict "symbolic" data to a very
narrow domain where issues like hierarchical structure was totally absent, and
vectorial representation was therefore quite straight forward, and I wanted to
figure out an alternative. Instead of restricting the kinds of symbolic data
involved, do the full spectrum. I had the symbolic vector idea in my head for
about two years as an answer to the representation problem, but I hadn't seen
what the right answer was to the other aspects, at least not clearly. But that
paper motivated me to start thinking about it again, and this time I saw the
answer. :)

------
eschaton
The blog post leads with:

> This post aims to provide a unifying approach to symbolic and non-symbolic
> techniques of artificial intelligence.

This is the sort of thing people build academic careers on.

I'm extremely skeptical that a blog post will live up to this claim; I'd
expect anything that even approaches to unify the techniques to be on the
order of a Ph.D thesis or an extensive postgraduate research report.

~~~
psygnisfive
It's a very small unification, and a big, eye-catching claim.

Besides, I said "aims", not "succeeds"! :)

I certainly don't think this is The Big Solution, tho. It's far more mundane.
Unifying the techniques is nice, but all that means is that we don't have to
think about GOFAI and ML as fundamentally distinct. It doesn't mean that any
truly _deep_ problems have been solved, just that there's now a reasonably
reliable way to use both that isn't just "simultaneous" but rather "fused".

------
tree_of_item
Are there any examples of reading a graph _back_ from this representation? It
doesn't seem straightforward, unless I'm missing something. How do you handle
the nondeterminism caused by the lossy transform in to a vector?

~~~
psygnisfive
Reading a graph back can't be done with any certainty, nor is it intended to
be done. It's a representation that's intended to be used to build inputs to
ML systems, not outputs. Building symbolic outputs should be done via the
other techniques mentioned in the blog post (eg. structural editing), that way
you can take advantage of notions of correctness inherent in the symbolic
representation.

Regarding handling lossiness, it's not clear to what extent it _needs_ to be
handled, but the `k` mentioned in the definition of symbolic vectors is a
parameter, and the larger `k` is, the less lossy the representation.

------
upvotingfriend
It looks like [http://languagengine.co/blog/](http://languagengine.co/blog/)
is missing this entry.

This is a great project, I can't wait until it's released for dev preview.

~~~
psygnisfive
Oh! Thanks for pointing this out. The whole site is hand maintained, and I
routinely forget to put stuff up there. :)

------
sjg007
Thank you, very interesting to read and think about. This is why I read hacker
news. This line of research/reasoning sounds very promising!!

------
mooneater
This representation assumes a fixed set of nodes. It would be much more
interesting to allow for varying numbers of nodes.

~~~
psygnisfive
Heyo. Author here.

It depends on what you mean by that. It assumes a fixed set of node _labels_ ,
but not node identities, since we're counting subgraphs. So for instance,
multiple distinct nodes labeled `X` or edges labeled `Y` will contribute to
the same count, despite having distinct nodes involved.

------
xixixao
How do you deal with cycles in the graphs? (Apologies if I missed this in the
blog post)

~~~
yorwba
I don't think you need to do anything special for them. Your algorithm for
counting subgraphs shouldn't go in an infinite loop when fed a cycle, but
assuming you have such an algorithm already, getting the vector representation
should be straightforward.

~~~
psygnisfive
That's correct, yes. Cycles don't affect subgraph counts. There's still only
finitely many.

