a) a linear SSM (a form of RNN?) is equivalent to Attention without the scaling and softmax; and
b) Attention is "all you need" and the thing that made Transformers radically outperform all the previous architectures like LSTMs that used to dominate NLP;
does that imply c) the scaling and softmax parts of the attention equation, in particular, is the magic touch that makes Transformers work so well?
The major difference is that transformer state grows as the sequence gets longer, while recurrent models use a fixed size state. So presumably at sequence length (T) > size of state space (N), the transformer will be better on some very specific tasks. Not all, especially those that require the model to select information from the beginning of the sequence conditional on something at the end of the sequence. Transformers can refocus any time, while SSNs need to guess right from the start what to keep and what to drop. SSNs could use the old trick of repeating the input twice to allow the end to condition on the beginning as well.
An important role is held by the softmax function which normalizes the attention scores, allowing the model to weigh different parts of the input sequence dynamically. This means that, unlike RNNs which sequentially process inputs and update states, Transformers can directly access and prioritize information from any part of the sequence, and they are not slower for T < N.
Doesn’t this effect happen with EVs generally, not just Teslas? (I don’t particularly mean to defend Tesla here, but I wonder if this might be misleading)
When I see an embedded DSL passed around as strings like this I can't help but think "this could be its own programming language"
Then it could have syntax highlighting, auto complete, and so on. The type system for such a language could possibly include verifying shapes at compile time.
What would a language comprised of .ein source files for manipulating tensors, which compiles down to the same low level ops, look like?
No need for .ein source files. We just need a programming language that allows the definition of embedded DSLs without shoving them into one-line strings. A language like Common Lisp.
This is also how it works in Julia, where macros digest notation for einsum-like operations before compile-time. In fact the linked file's explanatory comment:
(einsum (A i j) (B i k) (C k j))
results in the the updates
A[i,j] = \\sum_k B[i,k]C[k,j],
which is equivalent to matrix multiplication.
very nearly contains the syntax used by all the Julia packages (where @ marks a macro), which is
I wrote a library in C++ (I know, probably a non-starter for most reading this) that I think does most of what you want, as well as some other requests in this thread (generalized to more than just multiply-add): https://github.com/dsharlet/array?tab=readme-ov-file#einstei....
A matrix multiply written with this looks like this:
enum { i = 2, j = 0, k = 1 };
auto C = make_ein_sum<float, i, j>(ein<i, k>(A) * ein<k, j>(B));
Where A and B are 2D arrays. This is strongly typed all the way through, so you get a lot of feedback at compile time, and C is 2D array object at compile time. It is possible to make C++ template errors reasonable with enable_if and the like, this works well-ish on clang, but not so well in GCC (YMMV).
This library is a lot less automated than most other einsum implementations. You have to explicitly control the loop ordering (in the example above, the `j` loop is innermost because it is loop 0). If you build a good loop order for your shapes, the compiler will probably autovectorize your inner loop, and you'll get pretty good performance. Control over the loop ordering is in general a useful tool, but it's probably a lot lower level than most users want.
Einsums are the regexes of tensor programming. Should be avoided at all costs IMO. Ideally we should be able to write native loops that get auto-vectorized into einsums for which there is a CUDA/PTX emitting factory. But for some reason neither PyTorch nor JAX/TF took this route and now we are here.
Some of the einsum expressions I have seen for grouped multi headed/query attention is mind-boggling and they get shipped to prod.
JAX kind of did take this route, no? The main issue is that it’s going to be hard/impossible to compile Python loops to GPU kernels. It’s also maybe not the most ergonomic solution, which is why there is shorthand like einsum. Einsum can be much more clear than a loop because what it can do is so much more limited.
JAX tries to be a functional language that has a Python front end. The problem is if you are outside Google and don't really understand the XLA compiler, then you are screwed.
They're commoditizing their complement [0][1], inasmuch as LLMs are a complement of social media and advertising (which I think they are).
They've made it harder for competitors like Google or TikTok to compete with Meta on the basis of "we have a super secret proprietary AI that no one else has that's leagues better than anything else". If everyone has access to a high quality AI (perhaps not the world's best, but competitive), then no one -- including their competitors -- has a competitive advantage from having exclusive access to high quality AI.
And the content industry will grow ever more addictive and profitable, with content curated and customized specifically for your psyche. The very industry Meta happens to be the one to benefit from its growth most among all tech giants.
> will make most humans' (economic) value evaporate for the same reason
With one hand it takes, with the other it gives - AI will be in everyone's pocket, and super-human level capable of serving our needs; the thing is, you can't copy a billion dollars, but you can copy a LLaMA.
> OpenAI’s mission is to ensure that artificial general intelligence (AGI)—by which we mean highly autonomous systems that outperform humans at most economically valuable work—benefits all of humanity. We will attempt to directly build safe and beneficial AGI, but will also consider our mission fulfilled if our work aids others to achieve this outcome.
No current LLM is that, and Transformers may always be too sample-expensive for that.
But if anyone does make such a thing, OpenAI won't mind… so long as the AI is "safe" (whatever that means).
OpenAI has been totally consistent with saying that safety includes assuming weights are harmful until proven safe because you cannot un-release a harmful model; Other researchers say the opposite, on the grounds that white-box research is safety research is easier and more consistent.
I lean towards the former, not because I fear LLMs specifically, but because the irreversibly and the fact we don't know how close or far we are means it's a habit we should turn into a norm before it's urgent.
In general I think this approach of "super easy capture into an append-only log" is great, especially if it can be paired with features to enable editing/re-discovery/search/synthesizing old ideas together, which exist in a separate view/mode from the "just get something down as fast as possible" mode. Working on something like this, but just in nights/weekends free time with other obligations, so it's been slow going.
Yeah, it's a balance. I love being able to help, and I am generally in favor of asking questions early, but not ones of the form "hey so I ran this code and it errored. Help?"
"... did you read the stack trace? Did you look at the code referenced by the stack trace?"
This is where I've learned responding with "Sure! What have you tried so far?" is relevant.
This is an interesting point, because they are kinda infinite spaces but they also impose a structure on the space, which I would assert is a part of why they are so successful, in contrast to the "infinite structureless blank paper" that the OP is talking about.
The trick is getting the amount of structure just right, not too much to be too restrictive and not so little that users are lost in the way the person you're replying to describes.
Infinite canvases require panning and zooming with the mouse and tracking motion visually. A lot of users probably don't like the cognitive load of that compared to other types of apps.
Excel handles this better because navigating around a large sheet feels more like "snapping" as there's no UI motion as your view zooms in and out. Plus with shortcuts like CTRL + rArrow, which immediately snaps your selected cell to the rightmost end of the current range of cells, the infinite canvas feels downright zippy. Excel sheets also have tabs that signal to the users about how they can split up their data instead of filling up Sheet1 with every scenario. Infinite canvases make you create a new file.
This is fair -- the newest token can attend perfectly to the oldest token, within the context window.
but also, on a broader scale, if a transformer model is presented with a long input that does not fit in its context (e.g.: you are building a chatbot, and you have a very long chat history), it must "compress" or "forget" some of that information (e.g.: repeatedly summarizing historical messages, dropping them and appending the summary at the beginning of the input).
Mamba/RWKV/other "recurrent" architectures, can theoretically operate on unbounded input lengths; they "forget" information from earlier tokens over time, but is that not comparable to what a transformer must do with input lengths greater than their context window?
I love this! Simple and solid execution. I've been wanting to build something similar for some time now, might fork and play around with it. Thank you for open sourcing it!
I've started using Obsidian with a new note for each day and separating "blocks" with a Markdown horizontal rule (`---`) to achieve something similar, but this is much cleaner.
The strength of such an approach is making capture extremely easy -- new block, start writing, no thinking about where this goes and how to fit it into pre-existing structure. I find that if I'm trying to do that, then by the time I find where my idea goes, I've lost the idea.
The downside, of course, is finding things again. The ability to tag or title a block and search by tag or title would be great. More ambitiously, it would be cool to experiment with incorporating LLMs and embeddings to automatically tag, summarize, categorize, cluster etc. your blocks.
There's a lot of different directions one could take this, but I'll echo the sentiment of others to refrain from adding too many features and losing the original appeal of simplicity. :)
Also: How do you handle performance when the buffer gets very large?
If you put any restrictions on usage or what can be done with it (like selling), then it's absolutely not open source.
open source doesn't mean source code is there. open source has a specific definition. There is a list of acceptable open source licenses, as defined by OSI. similarly there is a list of acceptable free software licenses, as defined by FSF. Broadly, the two lists are the same. Commons Clause is definitely not open source.
There exists a niche of commercial software developers who are actively attempting to water down the commonly accepted meaning of "open source" for their own gain, and I suspect they are voting you down. :(
I didn't put too much consideration into picking a license. If someone has compelling arguments for why I should license it differently, I would absolutely consider it.
Performance is mostly handled by CodeMirror (https://codemirror.net/), the underlying editor that Heynote is built upon. It seems to handle quite large buffers well. Where I have seen some minor performance issues is when working with very large blocks in certain language modes.
I use Obsidian for my programming notes, troubleshooting logs, thinking on “paper”, writing and checking assumptions. It’s very powerful and quite performant. AMA.
a) a linear SSM (a form of RNN?) is equivalent to Attention without the scaling and softmax; and
b) Attention is "all you need" and the thing that made Transformers radically outperform all the previous architectures like LSTMs that used to dominate NLP;
does that imply c) the scaling and softmax parts of the attention equation, in particular, is the magic touch that makes Transformers work so well?
reply