Invent is probably the wrong word since it implies agency, sure. Maybe "discover" or "luck upon", since whatever it's doing was formed by updating a pile of floating point weights with gradient descent?
I think it certainly makes sense to ask what the higher level "algorithm" at work here is, though. Electrons flow through wires and transistors in (say) an adder [1]; looking at the wires and transistors you won't see an algorithm for addition, but there is certainly one present, codified in the arrangement of those wires and transistors. But maybe we can reverse engineer whatever the LM is doing by a combination of probing it with experiments like these and (maybe) inspecting the learned weights. The Curve Circuits paper did this for reverse engineering a curve detector learned by a convolutional neural network: https://distill.pub/2020/circuits/curve-circuits/
I also don't mean to imply that it's a good algorithm, or one that generalizes to arbitrary numbers, etc. Maybe it's just (effectively) a lookup table and some special cases!
[1] Please don't yell at me for this metaphor, I bailed out of physics after scraping out a B- in E&M ;)
I'm fine with "invent" actually, despite the implication of agency (I'm used to the terminology "predicate invention" [1]; although maybe I should actually re-examine the motivation behind it).
I'm more interested in the representation issue. I had a look at the quoted article on CNNs earlier. I think there is a very fine line between claiming that a CNN's weights represent an algorithm and that its weights can be _interpreted_ as an algorithm. I feel that the article leans too heavily on the interpretation side and doesn't make enough of an effort to show that the CNNs weight really represent an algorithm, rather than having activations in subsequent layers and therefore with a natural ordering.
In any case, I would like to understand how a language model can represent an algorithm.
> I think there is a very fine line between claiming that a CNN's weights represent an algorithm and that its weights can be _interpreted_ as an algorithm.
Yeah, I agree this is an issue. It feels a bit reminiscent of Searle's Waterfall argument, and so I'm inclined to turn to Scott Aaronson's response here [1; Section 6] – basically, how much work is the interpretation itself doing? If you actually tried to use the "algorithm" to do what your interpretation says it should do, how much work would you have to put into that mapping? If the work required amounts to just implementing the algorithm and effectively ignoring the CNN (or waterfall), then the interpretation is what was doing all the work.
IMO the Curve Circuits case passes this test, since they show that you can mechanically take out the learned weights, drop in the weights derived algorithm they reverse engineered, and the model still works about as well.
> In any case, I would like to understand how a language model can represent an algorithm.
Thanks for the link to Scott Aaronson's paper, that I hadn't read. I think your
comment helped clarify something that bothered me with the CNN paper: I would
feel more convinced if the claimed algorithm had been implemented _without_ a
neural net. What the authors did was manually set the weights of another neural
net. If an algorithm is discerned in the neural net's weights, then why can't it
be programmed in the usual manner, in some computer language? If it can't, then
that's very difficult for me to accept it as an algorithm, because I can't see
what it is, exactly. The authors claim that the algorithm can be described in a
few English sentences, but in truth these few English sentences are a high-level
description of the behaviour of the CNN rather than a sequence of instructions
(which is what I have in mind, perhaps erroneously, as an "algorithm").
I'm not necessarily asking for simplicity. I'm used to algorithms being
relatively simple things that can be implemented in a few lines of code and I'd
think of something more extensive as a "program" rather than an algorithm, but I
appreciate that an algorithm encoded in the weights of a deep neural net could
be something really big. I just want to see this algorithm written down in
pseudocode at least, in a form that can be executed by a device other than a
neural network (like me, following the pseudocode). I think that is the opposite
of Aaronson's point actually.
I mean it's just a generic blob of machine learning. It learned to do math because we write about math. It can't do arbitrarily long computations since it's only so recursive, but getting the basics down isn't too hard. It probably only struggles because it can't see digits, and can't do digit-wise operations (same with rhyming actually!).
I think it certainly makes sense to ask what the higher level "algorithm" at work here is, though. Electrons flow through wires and transistors in (say) an adder [1]; looking at the wires and transistors you won't see an algorithm for addition, but there is certainly one present, codified in the arrangement of those wires and transistors. But maybe we can reverse engineer whatever the LM is doing by a combination of probing it with experiments like these and (maybe) inspecting the learned weights. The Curve Circuits paper did this for reverse engineering a curve detector learned by a convolutional neural network: https://distill.pub/2020/circuits/curve-circuits/
I also don't mean to imply that it's a good algorithm, or one that generalizes to arbitrary numbers, etc. Maybe it's just (effectively) a lookup table and some special cases!
[1] Please don't yell at me for this metaphor, I bailed out of physics after scraping out a B- in E&M ;)