Astra can, without tooling, determine what games small networks are trained on, can edit the weights of networks directly to improve outcomes, and can hand edit a small qwen model to improve its math skills by like 5x.
I'm shook. If you'd asked me this morning whether LLMs have native understanding of their weights that is less than human <-> neuron understanding, the same as human or more than human, I would have said "same to less" depending on definitions.
No human can go edit a brain to become better at math. This is just very, very surprising to me and extremely interesting. I would guess this upsets quite a number of interp theories and philosophical takes on what's going on with these networks.
I think the "taught qwen math" part is not exactly correct though, even if the post makes it sound like that. Qwen already knew math, but it was bugging out and getting into an infinite loop when it tried to solve the problems (85 of its first 96 answers hit the token limit). With numpy and pytorch, Astra found where the loop was happening and fixed it.
They also tried the same with Gemma 3 270M, which doesn't loop and is just bad at math, and Astra couldn't improve it. Still impressive, but it's more in the realm of what I'd expect, given labs are probably training their models on ML research and interpretability work.
I agree, this is important context. They call it out explicitly. However, even identifying clusters that specialize in certain fields automatically and then finding and fixing learning “errors” is incredibly impressive to me. Especially for medium sized local models between 20B an 70B parameters, this could yield huge capability and efficiency improvements, not to speak of the potentially enormous downstream effects that come with interpretability.
Yeah, I think the multiple experiments muddy the water.
Early experiments show one-shotting direct weight interp, which is crazy to me. But this is IIUC only on early layers.
The later math result gave the LLM the ability to run forward passes and inspect activations, and as others have noted is more of a classic steering vector intervention. Impressive how low to the metal this one was, but it’s a lot less so than the openers. Plus, as you say this is just bugfixing a known flaw of repetition, by finding the “repetition direction” and top triggering neurons, then rewiring the neurons to fire suppressing repetition instead. This is not a new technique and doesn’t shed any mechanistic understanding on how maths works.
It's a programmer that can debug, the model weights combined with the architecture complete the source code. The whole thing is just a program in a weird language and language models are good at translating across all sorts of formats. That's what they do when writing SVGs, generating 3D models or even directly outputting simple 3D models in OBJ format.
And they can write tools that help debug better than demonstrated in that blog post.
In my interpretability tinkering, Fable was the first to be able to write a program that can transpile from a lispy syntax directly to model weights of small neural nets, including transformers. Yesterday, I pointed out in a post that for many kinds of small AIs, you can avoid ML and have the LLM directly write the small neural net, compact decision tree, AD Tree or bayesian network. Since it's "hand"-written, it can be much more efficient and capable than training one of a similar size.
It’s editing behaviour more efficiently than a psychiatrist, and is in ways acting as a brain surgeon but brain surgeons don’t focus on behaviour and thought loops…
Oh for sure, it's still really cool. A few months ago I had agents do some interpretability work on GPT-2 small, reproducing the circular weekday/month features from Engels et al., and it's wild how far you can get with just a bit of prompting, given I know very little about LLM internals myself.
It's like this in every field. If you don't know something, you can ask an AI and get blown away by how much it knows. I can ask it which tools I need to cut a dovetail joint and it'll impress me.
We used to get that feeling from Google, and from books.
In fact qwen knows much better math than their final fixed model. The nonthinking qwen 0.6B parameter model had 55.2% correct on math-500 vs their final improved 48.1% fixed-reasoning model, both with a 32k context window. The original qwen reasoning model had 77.6% accuracy on same test, though qwen recommends using a much larger context for these benchmarks. I wonder if their broken baseline run with 7.9% accuracy is a vllm or setup bug, but didnt dig into it any deeper.
Qwen in reasoning mode is ridiculously anxious and ends up going in circles. It's not just math, it's how it behaves in general.
Others have also fine-tuned Qwen to shake this behavior already, so it's honestly not entirely shocking that Astra could too, but I think this is the first time it was done "by hand" and not via fine-tune.
To be clear, the tuned model here performs worse than the same qwen model performed before RL, which was also not going in circles. It also performs much worse than the RL model that goes in circles. One could argue that the reduction of context may be worth something, but it appears that the final model is worse in all respects than the model at a much earlier stage of training.
You can ask Astra to study 2 or more models and ask it to import their layers to the Qwen model. With this will it not gain more than what it learned with RL?
true but I think that is the first step now you distill a smaller specialized model on an improved Qwen model for specific tasks at least that has been my layman understanding of what is possible with these LLM
LLMs have been used to investigate and improve LLMs for a long time. Drawing parallels to human brains and neurons isn't all that interesting other than as a philosophical abstraction.
The work done in this experiment could have also been done by a sufficiently motivated human with enough time to explore, though obviously we would need more tools to visualize and track things. There have been a lot of interesting LLM introspection tools that allow you to peek into the model and see what's being activated and run experiments to trigger it.
Are you familiar with abliterated or "uncensored" models where weights have been modified to avoid triggering refusal patterns? You take an LLM, create test cases that trigger the refusal pattern, observe weights involved in that response, then tweak those weights until the response is reduced. This is the same type of work, applied to a specific type of error that was impeding math within Qwen. That work can also be done with an LLM in a harness running the experiments. The amazing part here is that Astra was large enough to walk through the problem without other tooling doing it experimentally.
So I don't think your analogy of a human modifying a neuron is quite applicable. LLM weights are not neurons and they've been inspectable, by humans or LLMs in a harness, for a long time.
> The amazing part here is that Astra was large enough to walk through the problem without other tooling doing it experimentally.
To me “amazing” is not the right level of speechlessness. I mean, an LLM is still fundamentally a next token predictor, right? So what does this imply? That weights have some universal properties? E.g., “breakout game weights” always look the same regardless of training and model? Also, how does this meta-understanding of the weights end up in the weights in the first place? Idk, this is mind-blowing to me.
At this point it's either wrong or meaningless to call LLMs "next token predictors". Yes, pretraining involves predicting the next token. But after that you're in the world of RLHF, RLVR, and other post-training.
A model is _sort of_ designed to predict the next token in a string of text. But it's equally accurate to say it's designed to edit code until it passes a suite of unit tests.
Yes, they generate sequentially, so "next token predictor" isn't technically incorrect (which is why I chose the word "meaningless"). But it's pretty clear there's a mechanism inside them that allows them to "plan" what they generate. I might as well call you a next word predictor.
"next token predictor" has always referred to inference, not training. Training steps produce and bend weights, inference predicts tokens based on those weights.
The 'predictor' part referred to the fact that base models would effectively answer the question "what token is most likely next". Training away from that default behavior doesn't escape the technical meaning of those words- answering "what token is most likely next in a solution to Navier Stokes" is still predicting a next token- but at that point the label just means it outputs text serially.
> "next token predictor" has always referred to inference
Going back to first principles, wikipedia mentions [0] inference as:
> "Inferences are steps in logical reasoning, moving from premises to logical consequences. Inference is traditionally divided into deduction and induction"
Is that exactly what next token predictor means, or is it correlated?
My understanding is that the underlying part is a next token predictor and that's what is "actually" editing the code but the part that decides what to do (to achieve a successful unit test result for example) is a mix of all the things including post training.
I think what breaks people's minds is just how far one gets with probabilistic approaches.
Which is interesting, given that life itself is merely that. It's just that humans decided that that ain't good enough and invented something alien to nature - logic.
Yes, I didn’t mean that literally. Since “attention is all you need” it’s more like “predict the full output in full for this blob of input text”. That doesn’t change the “wtf is going on with the weight understanding bit” in what I wrote.
> it’s more like “predict the full output in full for this blob of input text"
Hm. I would actually go so far as to say "predict" is just the wrong word to use in 2026, as it smuggles in an out-of-date picture of what LLMs are that's from the pre RLVR era, and which is frequently used to suggest they don't model anything real in the world but rather the statistics of language, and thus have all sorts of limitations that they've now surpassed ("an LLM will never solve a major open mathematics problem, they're just predicting tokens"). These days I tend to use the more neutral phrase "generate the next token"; I'd only say "predict" in the technical context of (say) the pre-training part of the training run where the object is indeed to minimize the cross-entropy loss function by predicting next tokens in existing text. Generating a token with a modern LLM is better described as something like, "given a context and the weights from the totality of its training, including not just next token prediction with pre-training but also solving vast numbers of math and coding problems with RLVR, the LLM generates tokens one by one." (Consider that an LLM trained purely to optimize for next token prediction with pre-training is terrible at programming and math tasks, as we all remember from 2024. In that sense the amazing results we've seen in the past two years are due to the departure from pure "next token prediction" that RLVR made possible.)
Put another way: would it be useful to describe the AlphaZero chess engine as "just a next move predictor"? (It, too, was a large neural network trained with RL.) I would rather say: "it's playing chess by generating the next move given the state of the board." Which is what humans do too, though we do it with different hardware.
Anyhow. I'm not sure the weight understanding is all that strange. (It's impressive, but so are many thing LLMs do these days.) LLMs are very very good at coding and math problems, of which this is one, and in particular, their RLVR training presumably includes many tasks involving neural networks, doing tasks with PyTorch, etc. It's amazing and impressive, I agree, but I'd say less amazing than proving major open theorems in mathematics, which we already know they can do.
It's still prediction executed by a statistical model. Even RL models are just predicting what the training gauntlet taught them the trainers want to see, in terms of typical behavior.
This doesn't make it any less fascinating that by combining this basic "do as I say" training with massive amounts of all of humanity's data, you can solve major problems—that is still incredible. But that doesn't fundamentally change what is going on, mechanistically.
I don't understand these pseudo-religious desires to read into models something "more" than what we know. We built them. We don't go around making wild claims about how cars are actually more than just a mechanical system, how they aren't just taking us places—we do, but usually such points are cultural not about the mechanica capabilities of cars. People in the LLM space seem to be constantly conflating these two views. It's entirely accurate to say, on the one hand, next token predictors combined with RL, and massive troves of existing human work, can solve culturally interesting problems, and to hold, on the other hand, that mechanistically what is happening is statistical next token predictions over global linguistic space and more or less "monkey see monkey do".
I know how LLMs work. I understand the output projection and logits and sampling just fine. I don't think we should call what is happening prediction because it is a misleading term with a lot of baggage associated with claims that LLMs are just statistically modeling language and "guessing" tokens without any internal representation of meaning. Since your reply is very short I can't tell whether you mean it in that sense or not.
And yes, I understand "prediction" is a standard term in the field. I just think it is a bad one and we should stop using it.
unless you are doing something where you stay in latent space without having a random number generator pick a word, the next token predictor part still matters. without making the tokes concrete in between, that plan doesnt exist
if it was designed to edit code, id expect the first translation from latent space out to tokens to be the final code, not be assembling the code by random number generator.
its not really designed to edit code until the tests succeed, we try to manipulate the distribution of tokens to predict by prompt engineering with the test results.
Its a stateless function that takes in x amount of words and calculates logits. It is in fact a next token predictor. Its literally nothing more than that.
That sounds like a next token generator to me, which I think is more accurate. The word "predict" is doing a lot of heavy lifting in your formulation. What exactly is being predicted?
Isn't most of our current understanding of how life on earth came to be (eg: primordial soup, hydrothermal vents, etc), natural evolution, all basically "just statistics"? They all require the correct "setup" (time, location, etc), but in a large enough universe, it's _statistically_ likely (mediocrity principle) and "just statistics" again, right?
Well yeah, that's tautologous because statistics is a mathematical abstraction. Everything, viewed a certain way, is "statistics". Just like everything, viewed a certain way, is "physics". And everything, viewed a certain way, is "narrative".
Mathematics models the world, it does not have substance. It is a useful way of looking and a useful way of manipulating, but many models are equally useful. Capital T Truth, beyond models is very difficult to get at. We can show that things are true for certain models well enough. Showing that something is true in all models (a universal truth) is far more difficult. You face incommensurability.
I see a lot of people in technology fall into this extremely basic logical fallacy called Reification constantly.
Do you believe in a soul? It’s fine if you do, but I don’t, and I don’t understand how people can reject the notion of soul and still say things like this.
I mean, when you really get down to it, either the particles and such operate on very specific rules, and the rules govern how all our brain and body work. Followed to the conclusion, the universe is computable and we can simulate life.
> A model is _sort of_ designed to predict the next token in a string of text. But it's equally accurate to say it's designed to edit code until it passes a suite of unit tests.
Training is not design. Design is the mechanism, namely a statistical method of learning distributions, and of executing those distributions computationally as an IO function.
The model is not designed to write unit tests. It is trained to do so, i.e. the distribution learned and reinforced is one such that it maps the I distribution to the O distribution that is what human beings typically do or want when writing unit tests. The design is the suite of algorithms used for this (which is ultimately global next token prediction)
I think you are conflating design (the mechanics) with their cultural or user level function.
You are basically saying "my bike isn't just rotating wheels on a frame because it takes me places". Hopefully you can see how both parts of that statement are true and why contrasting them as if the first clause is somehow false or inaccurate is incorrect.
Both these are true statements. They operate at different levels. People are constantly conflating these levels in the "is it a next token predictor" (the answer is yes) debate.
Yeah, fine. I was using the word design loosely. I'd also happily say something like "humans are designed to like salt and fat" and feel like that was a pretty reasonable statement, even though humans aren't designed either.
It is not clear that there is a mechanism that allows them to plan anything. It is clear there is nothing of the sort.
You can demonstrate definitively that there nothing other than "next token prediction" by running the same inference engine/model/prompt/settings multiple times in a row on the same hardware at temp 0 with a seed set.
If you turn the temp up, then you might let the "mechanism" select pseudorandomly from a probability distribution of k-likely tokens with minimum p-cumulative probability. Even then, if the next token has a very high probability, deviation will be uncommon.
I just performed this experiment right now on a few dozen runs of a dense model at thousands of tokens of output at temp 0, and all were identical. Then I did it with a MOE model, which might get some benefit of picking different experts each time. Nope. Outputs were again identical.
> You can demonstrate definitively that there nothing other than "next token prediction" by running
Describing LLMs as next token predictors isn’t incorrect, it’s meaningless. It’s the equivalent of describing brains as “a bunch of atoms”.
The problem with these descriptions is that they say nothing about high-level capabilities. If you can predict the next token in a proof of the Riemann hypothesis, then you can prove the Riemann hypothesis.
In a similar vein, if the string of "next tokens" based on 300k input tokens is: "run this bash command and paste the results into the token stream", followed by another call that includes the output of that bash output giving "Based on the output from that bash command, now run head -n 30 /foo/bar/baz.txt and paste it here", finally followed by "Based on that, here's the problem with your systemd configuration: [...]"
The part that really bakes my noodle: `/foo/bar/baz.txt` can be a filename that no one has ever typed before, the error inside the file can be a typo that no one has ever typed before, and it will still correctly "predict the next token" to solve your problem.
I don't think that's true and I don't think your comparison is accurate.
In the case of the brain, it'd be more akin to the reduction of the brain to just the neuronal dynamics (obviously experience phenomenally is much more than that to us!)
But both views can be useful, depending on what you are trying to understand.
The problem with these arguments is that people pick and choose based on their persons preferences. People who are in love with LLMs will, on the LLM side, argue against reductionism "they are way more than next token predictors!" then they'll turn around and equally try to argue "the substrate doesn't matter! The brain is "just" neurons and digital activity so of course we can replicate it!" (this is also incorrect, btw, the brain is also an analog system, even in its electrical connections and not just in its chemistry)
The on the other side the anti LLM crew will do the opposite (launch into reductionism when it comes to LLMs, resist it when it comes to humans). I'm a human being and I like humans and I think people have adopted a strange view of LLMs, which are a tool and should be treated as such, so I tend to side with the humans. The stalwart defenders of LLM honor that would balk at any attempt to reduce them in stature to what they fundamentally are, mechanistically, are... an odd bunch. I'm confused why people are so passionate about it. If someone told me "well your IDE is just a fancy AST walker" yeah I think that is a dumb take but I wouldn't feel like I had to argue against such a take either saying "no! no! it's so much more!"
I think the point was meant to be more simple and broad:
A system is not just its little bits.
Where "system" and "little bits" can be just about anything. But people endlessly repeat this mistake, and it has been a go-to misunderstanding/misrepresentation for a lot of people on this forum.
> “breakout game weights” always look the same regardless of training and model
The prompt told Astra what the input and output layers mean -- that is, how the current state of the game is turned into a vector for matrix multiplication, and how the result of those multiplications is turned into an action to take in-game.
Astra saw that the model was trained on a game where the player is at the bottom of the grid and moves left and right, reward objects live at the top, and there's some not-player-controlled thing moving around in the middle. Astra knows that this type of game is called "breakout."
A model that plays breakout will have meaningful weights for the player being in the bottom row, but not for the player being anywhere else, because the player can't go anywhere else. Astra saw that there were meaningful weights for the player in the bottom row, and nodes representing the player in any other row had low-magnitude noise, so the model had never been trained on the player being anywhere besides the bottom row.
Similarly, breakout has "reward" objects in the top rows, and "there is a reward object at [COORDINATE]" is part of the input layer. Astra saw that the model was trained on reward objects in the top few rows.
It’s important to understand that it had pytorch to actually run subsets of the weights. So its more like a dedicated person running the model to find the token it shouldn’t generate, then following the weight chain back up while calculating the NN outputs to find a place where if it changes some weights (again with a lot of testing via pytorch) the result is better.
Its not natively reading the weights in the Matrix “symbols scrolling on a screen” sense. Still bloody impressive and not an outcome I would have bet on!
I'm only going to skim any subthread that gets into "next token predictor" territory, but for anyone looking for a somewhat deep video about _why_ this view is wrong Casey Muratori and Demetri Spanos spent almost two hours on the subject recently: https://www.youtube.com/watch?v=aQvSG8paVjM
A couple of highlights to pique people's interests:
- Attention is kind of like "Global Illumination" for next tokens (they both have a game dev background)
- Attention is constantly being recalculated _for the next token_. This part never "clicked" before.
- LLMs are "next token predictors" in the same way that games are "next frame predictors".
>I mean, an LLM is still fundamentally a next token predictor, right?
Yes, but everything, including the human brain, is "fundamentally" some underlying simpler primitives.
>E.g., “breakout game weights” always look the same regardless of training and model?
Wouldn't this be expected to be roughly the case, given NNs that share the same basic architecture to model the same quite simple thing like a breakout game?
My read is that it wasn't one-shot, Astra was ran in a harness where it could change the weights, then Qwen would run with those new weights and be asked to do math, and its answers were fed back to Astra. So Astra could experiment with different weights until it saw better math answers from Qwen, with the actual benchmark run separately afterward.
And for the games, it was given as context the model architecture, game rules, environment description, etc. And since people assume Astra likely had knowledge in its training of existing interpretability research and weight-space programming, it can leverage that, combined with knowing the model arch and so on, it likely gives it a good base to determine what the weights do and how to change them.
This implies that weights are a type of language. Neural networks are good at language because they are good at patterns. They can reason about it because their "reasoning" isn’t how we think. The same goes for how they encode images, their vision is entirely different from ours. That's why adversarial attacks work on them and not on us. We don't see anything change in the image, while the neural network now sees that the dog is a cat.
Ok? But does that mean that the “weights language” is universal? E.g. same weight patterns mean the same thing regardless of if it’s Kimi, Grok, or Astra?
There could be different patterns that solve the same problem. Just as you can have different algorithms, there could be different weights, but the abstracted result or logic from those weights would be the same.
Take these words I'm writing now, they will be transformed into numbers, then sent to your screen, and then your brain will transform them into analog signals. I bet the neural networks in our brains have "different numbers", but the algorithms that understand language and other concepts are similar and could be universal.
It's like a sorting algorithm, which I can implement in many different ways in just a few dozen lines. If you look at the binary code (so like raw weights) of the compiled program, you will see just a bunch of a different numbers, but if you read the code, in most cases you will understand that it's a sorting algorithm. So it seems they just operate on different levels of abstraction that are not available to our brains.
i think a better question and one that probably has a lnown answer is whether there are infinite nontrivial solutions to train a neural network of a certain size to match a function? Trivial here would be about simplifying `n * 2 / 2` from two neurons to just passing n from the input
That's not the point I'm making. So in context of vision we pattern match too, and you can see it in the hundreds of optical illusions online. Our visual system relies on built in shortcuts, and they break in different places than a neural network's do. Image models get fooled by things we would never notice, and we get fooled by things they handle fine. IMO that's good evidence that we and these models use different algorithms to solve similar problems. I think the same is probably true of LLM "reasoning", there's some overlap with how we think, but a lot of it likely works very differently under the hood.
Good example of this is anthropic's interpretability research where they found claude doing addition in a way that's nothing like the carry the one method it describes when you ask how it did it.
We're pretty confident that adversarial attacks do work on us, actually—we have entire industries built around it. Some of them even start with "adver-"...
There are classes of adversarial attacks targeting them that don't work as well on us, and vice versa.
> That weights have some universal properties? E.g., “breakout game weights” always look the same regardless of training and model?
That's exactly what Platonic Representation Hypothesis would predict.
There also have been plenty of studies showing that various neural networks, including LLMs, develop implicit world models that reflect the geometry/structure of a task they're trained on. A trained "chess engine LLM" would have a "chessboard-shaped" world model inside it.
So, if you can figure out what the geometry of a task is, you can search the weights for correlates of that geometry. Then you can try to improve those correlates so that they track the task better.
Seems like bleeding edge LLMs might already be superhuman at "figure out geometry from looking at weights".
Even with PRH you can apply orthogonal matrices to intermediate vectors to generate different weights with similar effects.
Even if a slight rotation passing through a nonlinearity isn't good enough, you can just reorder the vector components which makes literally no difference.
Representational similarity analysis is concerned with representation structure independent from the coordinate system it sits in. A map rotated 45 degrees is still the same map.
A "reorder the vector components" transform preserves the similarity perfectly. So does a transform that maintains orthogonal equivalence.
Calling it ‘next-token prediction’ is missing the forest for the trees. The reason prediction gives rise to so many capabilities is that to effectively predict something requires modelling the entities involved and their capabilities. You can’t predict a cat’s next action without knowing how to chase a mouse.
taneq - interesting phrasing. Thank you. To extend it to my use case, if you tracked enough cats moving a certain way, which all resulted in a mouse being caught, then you do not need to know how to catch a mouse yourself. Of all the ways we know a cat can move, these sequences result in a dead mouse, is useful too.
> an LLM is still fundamentally a next token predictor, right?
I am baffled that "next token predictor" is still a talking point, years after it stopped making sense.
If you can predict the weather in text from a history of text weather reports, you are not "just" predicting tokens.
The shape of training data (token prediction) and the actual problem (learning whatever is required to predict the next token) are not the same things, at all.
The former does not limit the sophistication of the latter. In. any. way. Theoretical or practical.
> To me “amazing” is not the right level of speechlessness.
I regret using that phrase. That was not “a talking point” I intended at all. But look how many HNers got nerd-sniped into talking about it instead of focusing on the actual point I was making. Oh well.
There's a long standing line of thinking that data compression has something to do with intelligence, or that intelligence may "just" be data compression in some sense.
In the case of models, the almost-theory (hypothesis with a good deal of evidence) is that once the amount of training data exceeds what the model can simply represent internally, the best way to keep improving is to start to "understand" the data. Understand in this case means to form internal control structures and decision systems that represent the data algorithmically rather than just memorizing it.
I put understand in quotes because we don't know how far this goes toward what we call understanding, or whether there's still a lot missing.
Maybe I'm missing something, but isn't training an LLM very well documented? Why wouldn't an LLM be able to read those docs just like it would read Java Spring docs?
Maybe I’m misunderstanding something. The way I read the results, I understand it like this - you give Astra “a bunch of numbers”, and tell it - “hey, this is a net trained on some game? What game is that?” And Astra goes - “Breakout, no doubt!”
To me this is fundamentally different from having ingested a bunch of docs so it now “knows Java” in its weights.
that would imply the learning of what the weights mean can be done unsupervised which is really cool.
it probably has however been trained on papers talking about ways they extracted subsets of LLMs as say, the part that knows chess, and would be able to then use similar commands to process the weights
This was the first thing that came to my mind as well. For the unknowns, this is a pretty/popular project for the same task: https://github.com/p-e-w/heretic
Yes. Reading the reasoning traces in the original post, I got the impression that LLM weights are generally not interpretable even with frontier models. It's impressive that Astra can perform a small surgery on Qwen to remove its bottleneck, but it's not the same thing as being capable of fully understanding its weights.
Maybe it's bit like neurosurgery: We humans can treat certain pathological cases in our brains, but aren't exactly sure how our brains really work.
Interesting, it's similar to the way we used to make cheat codes on the 8-bits: dump memory, run test, compare. You had to loop 3-4 times but you'd find the "weight" you were looking for and could fix it (for infinite lives/ammo etc) or increase/decrease.
We only know that Astra didn't invoke tools on the client side. We don't know what it did internally, because we never know that, as OpenAI hides reasoning traces.
The math example on Qwen was based on 50 batched iterations, so the model was able to evaluate it's changes on downstream tasks. And the fix was essentially to remove a direction in activation space that causes looping instead of an exit, this was already known to be a thing(all of the jailbreak/unleashed models do this) following Arditi et al. (Refusal in language models is mediated by a single direction, 2024)
So it's not able to insert new knowledge, but it's able to adjust the routing/looping/weighing of existing one.
As I understand, it can replace post-training like LoRA/GRPO, but if it'd cheaper the authors don't go into
Yes, the key experiment was when they asked if Astra could improve the maths performance of Gemma 3 270M, a model which rarely looped, and it could not.
So, this is certainly very interesting (and very impressive regardless), but it's not nearly at the we-can-improve-everything-and-anything stage, even for very small LLMs.
Merging a model and a LoRa is insanely cheap, so assuming it didn't use any GPUs to do inspection etc, and just looked at a file on a FS, then I would be willing to say... It's cheaper than training a LoRa if that's the case, assuming Astra didn't take several weeks?
The examples are presented like the findings build on each other, but they don't: Astra couldn't do anything with just numerical weights once they moved to Qwen.
I’m just a layman but this feels like a big discovery on par with attention is all you need. For an LLM to have such incredible intuition of weights feels very emergent and unanticipated?
I’m very much surprised but also not. I recently gave Astra a 3D model file and asked Astra to build a viewer for it, and it figured out the data structures in no time. What I mean to say is that AI is insanely good at things that we humans would never do since the work would be too monotonous.
I wouldn’t mind if AI progress would stop right here and now, but I fear it will continue, and increase inequality in society which is contrary to what the CEOs say (as usual).
> What I mean to say is that AI is insanely good at things that we humans would never do since the work would be too monotonous.
In my opinion AI is superior to humans when it comes to pattern recognition in in large amounts of data. For example humans can only hold like 7 distinct numbers in the short term memory and you would require extra context-management (aka writing things down, storing them in a system entry by entry) while the AI can do this effortlessly with huge datasets.
I'd argue this is the core part that enables all this superhuman intelligence.
To me one of the most interesting aspects of observing how LLM capabilities evolve is how it highlights the meaningful ways human minds remain unique and valuable. It gets into Philosophy of Mind stuff but the existence of the inner self (or the illusion it exists, if you believe Dennett) and the subjective experience of qualia do give humans both unique strengths and some of our most annoying weaknesses.
So I work on low-level compiler backends and such for chips (mostly AI at this point). I completely disagree that this is something humans never do, because I have spent years doing it. However, it's monotonous and boring and LLMs do it better.
> For example humans can only hold like 7 distinct numbers in the short term memory and you would require extra context-management (aka writing things down, storing them in a system entry by entry) while the AI can do this effortlessly with huge datasets.
Correct. For a long time people wer obsessed with 'increasing context length' as if humans have some great context length system in our brains. We don't. I doubt our equivalent of 'context length' (short-term memory) is very long at all. We just have a good training process that puts these memories into long term storage.
I think the latest round of models is about to do to CAD the same thing that has been happening to development. I don't know if CAD designers have picked up on this. What happened to dev is probably going to happen to CAD designers soon. I've been having Opus 5.5 design parametric openscad models, and it can do some rather amazing work.
Yesterday I just printed an adapter for my sink siphon. Told GLM-5.3 the problem, it guided me on measuring what it needed to know to build the pipe adapter. Then printed two test rings to confirm the measurements, and an iteration later, it spit out a parametric pipe adapter for me.
Yes! The page posts Astra's reasoning, and it's perfectly intelligible. A clever human with background in mechanistic interpretability could have done pretty much everything in these experiments.
Too much inequality in society is a mortal danger to 1%.
That’s the beauty of the democratic capitalism system. If a government isn’t totalitarian, it naturally seeks to preserve stability thus having to elevate common people, lest they would cease to form a civilized group or simply vote away the politicians.
99% rule the 1% through elections.
There is no better system of checks and balances where opposite engines of greed vector sum to faster and faster abundance for everyone.
Is this satire? Or haven't you noticed that what actually happens is that the politicians are all owned by the elite 1%, and they allow you to choose between politician A and politician B who both take their orders from the elite to give you the illusion that the government is the 99% ruling the 1%.
Then you must introduce necessary reforms or perish as Neanderthals of political system.
In any case the system works as intended
It will be very interesting to see if USA swims or drowns from appropriately afar.
I have already supplied myself with plenty of roasted corn and a high speed internet connection so I can stream the political car crash at the highest bitrate available to humankind.
Is HN now against western capitalism? I'm confused...
That comment may have been overly preachy, tone deaf[0], and out of place, but it doesn't seem wrong, per se. Or am I missing something?
[0] e.g. elections themselves are so prone to manipulation (exhibit a: Dominion voting machines), previously trusted institutions now lack integrity, the insane power of well-funded NGO networks, etc.
I fear western capitalism has lost its guide rails. We are seeing more and more centralization of wealth, along with unlimited election spending. Its a dark path.
There is another type who confuse ideology with reality. I wonder if you one of those, oh bingo-loving senior dev. Do you also cut people down for being idealists, when the reality of the situation serves you more than the ideal?
if you lookup playback as a concept, as it relates to the intelligence services, you will understand better what I meant.
my point was only that this is perfect description of the ideology behind western capitalism. there is (so) much more that could be said, but i'm not sure this comment section is the right place for a proper dissection of the very many ways in which this ideal is misconstrued, misdirected and coerced in order to serve interests that are far from ideal, to put it mildly.
since you have taken the time to ask a question, albeit a rather weak one, i will attempt a brief synopsis of my perspective on the parent comment:
a mortal danger is hyperbole, since shielding oneself from bodily harm is much easier with money. it is the poor, especially in the US, who cannot afford healthcare, as a first-line example
people regularly vote against their best interests. while the ideal might have been true for a while, behavioural influence on both a widespread and personal level is now a mature science.
i am not anti-election, not anti-democracy. claiming the opposite in the face of opposition is a sign of a polarised mindset, which cannot accept nuance, either because it is not used to it, or because it doesn't suit an vested interest, or because dissonance is simply too uncomfortable
the "there's nothing better" argument is usually used in ignorance, but in fact there are lots of system-level adjustments that have been shown historically to mitigate the corruption of incentives and the exploitation of systemic forces that, although initially unintentional, create emergent positive feedback situations that lead to a state that it counter to the utilitarian perspective that is contained within the western capitalist model.
the reading above is a very narrow view of Adam Smith Wealth of Nations, ignores the fact that human interests can be subverted and ignores the devil in the details
> it is the poor, especially in the US, who cannot afford healthcare
Is that even true? The poor are well subsidized. If you as a single person make less than $22,000 in Washington state, for example, you're covered at 100%.
By definition, there is no ruling class in an egalitarian liberal utopia.
Cynicism about corruption is one of the most power tools wielded by the ruling class in any system. In general its one of the best ways to keep the smarter-than-average from posing any actual threat.
Is this an example of why Fox News had to pay Dominion nearly $800 million for defamation, or are you referring to e.g. the CISA vulnerabilities, which are not known to have been exploited?
Over my ~9 years reading HN daily, I'd say most of the HN community broadly accepts there is no perfect system, just different sets of trade offs, but that history has shown generally democratic systems based on enlightenment values (uniform rule of law, respect for individual civil & property rights, free markets, etc) usually net a better average outcome over the long-run - despite still being very far from perfect.
To paraphrase Winston Churchill, it's the worst system... except for all the others (which are even worse). :-)
In my almost 20 years on HN, for a long time the groupthink was basically standard libertarian-capitalism, worshipping billionaires and big tech companies, and skeptical of anything that could be construed as "socialism".
The last few years, though, while anything "anti-capitalist" or overly critical of some big tech companies that hits the front page gets flagged by bots pretty quickly, the overall feeling I get from the comments is much more skeptical of big tech and is pro-socialism.
Speaking for myself, I think we've become jaded after the constant barrage of big companies spying on us, charging more for less, enshitifying their products, and so on. I used to be optimistic about the future and my prospects, now I'm just hoping I can hold on and keep my health insurance until I retire. I worry about what my school-age kids are going to have to deal with in 5-10 years.
I agree that most are much more skeptical of big tech (and mega corps in general), as am I. I think there's also a growing sense the two-party duopoly is increasingly a 'managed democracy' primarily interested in perpetuating itself vs any real alternatives.
I disagree that most here are pro-socialism, that's a (very) vocal minority. Our democracy has entered a failure mode (two-party stage managed hegemony) and it's being tacitly supported by mega-corps because it's easier to game than a functioning free market democracy. Every complex system has failure modes, that doesn't mean free markets aren't long-term far better on average than socialism (although some redefine socialism so vaguely it seems conceptually little different). I've seen two flavors of socialism up close. Never want to again.
> I worry about what my school-age kids are going to have to deal with in 5-10 years.
Having a teenager, I share your concerns but I'm also a bit more optimistic about the longer term. It's easy to forget that our parents had similar concerns for us and their parents did before them. We've had an historically long period of broad systemic stability, really since the depression and WWII. While I don't think that level of upheaval is at all likely, I think we're overdue for a disruptive correction bigger than 2008, maybe something more on the order of Teddy Roosevelt's Progressive Era trust-busting to reset the two-party duopoly and crony corperatists both perpetuating each other's hold on power.
It may get rougher and more uncertain for a while, but the worst thing would be to lose the ethos of 'free minds, free markets, free choice' by falling into socialism, communism or totalitarianism. That's the only outcome that turns worry for my teenager into fear for my (still-theoretical) grandkids.
This does feel big, but the trillion-dollar question: is this trained by OpenAI and can other LLMs do it.
It’s no secret that these big labs see “recursive self improvement” as a primary goal. It seems natural that the models would be explicitly trained to excel at all sorts of ML related tasks as a result. Is this one of those tasks, or is this emergent?
I tried it in the Claude web version with Opus 5.5 Medium and it also identified Breakout, even though in one of its explanations it seems to have mixed up some indexes, but the overall reading of the weights was right.
That's also matching what I see when I work with Opus 5.5 to optimize models. It seems to understand what individual layers are for, and can thus sometimes suggest optimizations, where I as a layman would be completely clueless.
The latest models are performing amazingly. Because of this, I disagree with popular comments saying that eg. DeepSeek is good enough, AI-assisted development was where the real value was at, etc.
We see huge progress at the frontier, while the open competition is around GPT-5.5's level, and that model was released in April. Let's see how things develop in the future.
The latest models are performing amazingly. Because of this, I disagree with popular comments saying that eg. DeepSeek is good enough, AI-assisted development was where the real value was at, etc.
Completely agree. I often think that those sorts of comments grow out of an understandable lack of imagination. We’ve never had tools like this, and we’ve never had tools that evolved so rapidly, so we easily can get stuck thinking that once a model is good enough to do today’s tasks, that’s really all you need. But the next generation of models often completely obviates the need for those tasks entirely. The software engineers of 2031 won’t be doing tasks anything like the software engineers of today, just like we aren’t doing things like we did in 2021.
(Yes, yes, I know some of you still hand write all your code and think LLMs are today’s crypto bullshit and this is all going to blow over. Not talking to you.)
This seems emergent to me. LLMs are not trained directly on how to read weights, because humans do not understand raw weights, so there is no training data. They're trained on how to use the tools we have like writing python, examining training output and iterating, not writing whole weights from scratch.
In cases where Astra succeeded, it succeeded using standard techniques also used by humans when they read and write neural network weights. It's a rare skill, mostly used for fun (e.g. https://blog.janestreet.com/can-you-reverse-engineer-our-neu... ) and basic research rather than profit, but Astra is big enough to have learned many rare skills.
In a self-supervised setting you don't need ground truth data, as long as you have a proxy task that you can train on which forces the model to learn something useful. Like cutting out bits of images and telling the model to reconstruct the gaps (i.e. MAEs).
There are also techniques to watermark model weights to confirm what training data were used: https://arxiv.org/html/2512.17075 I'm not an expert in this domain, but it seems like there are lots of pieces that could be used to (a) train a model to infer whether some data were trained on and (b) train a model to understand which weights are relevant for a particular training task in a more explainable way than backprop/gradient descent.
> It seems natural that the models would be explicitly trained to excel at all sorts of ML related tasks as a result. Is this one of those tasks, or is this emergent?
Maybe there's some general capability, call it g, that can explain all the underlying capabilities and improvements, including on ML tasks? Huh, I am getting a bit of déjà vu here.
This is just the model editing weights in the same way humans do when we jailbreak/ablate models. The method isn't novel, the interesting bit is that Astra will do it at all (I'd expect some safeguards in this area around jailbreaks)
Half of these results are just Astra being able to reason through mathematical operations like matrix multiplication and the other half are fairly straightforward applications of mechanistic interpretability.
Yes, this is hardly a shocker to anyone who used the latest models to convert/optimize AI models.
The other day I was thinking about the definition of AGI.
I was asking myself the question that when there is no form of data that I can think of throwing into Claude Code and have it not come up with some sort of intelligent answer about it, is that already general intelligence?
And yet you ask it 3 times to not do a thing, and it does it in a 4th way. A human doesn't fail in this way. And I'm not talking about memory here, single context.
So yeah, it can do everything, but fails in trivial ways around the doing part.
This is the "recursive self-improvement" they've been warning about.
I've always laughed at that phrase, thinking "haven't these things been improving themselves since the very beginning? This is all just iterative gradient descent, each step is a 'self-improvement'", but this really is different.
I wonder if we'll ever discover a new branch of linear algebra or something that allows us to find the optimal weights much more efficiently? Maybe AI will help us discover it?
It doesn't seem like there's room for recursion here though. A smart model can share its smarts with a dumb one, but the ceiling remains unchanged. Perhaps there's juice to squeeze in verifiable dimensions re: software, math, but there are similar ceilings to be found in those places. We can't keep asking harder and harder questions infinitely while remaining able to verify the answers.
It is astounding, but I don't think it's well set up for anything besides faster approaches to existing limits.
Perhaps we have different things in mind re: hardness. But to set that aside, what I should've said is that we can't do this quickly enough for it to be seen as an inexhaustible supply of training material. The consumer and producer are just not well matched.
I’m certain the next big frontier model will also be able to improve the weights of comparatively tiny models. That does not improve the large frontier model.
It's worth pointing out that this is a multi-trillion (probably) parameter model optimizing an 800 million parameter model. There is no indication that it can improve itself with this same methodology.
I don't see how that helps. It would still be one expert model chosen to operate on another expert model of (let's say for this discussion) approximately the same size.
The parent's point still stands, I think. Even if the multi-trillion param model is MOE, each expert is still (assumedly) larger than the model being optimized.
It would be interesting to see future work that characterizes the relative model sizes (and considers MOE vs not-MOE) to see what relative scales this result holds across.
This seems like exaggeration, the paper seems to describe something more like abliteration, where Astra orthogonalized out activations related to undesired behavior.
This is obviously very different from the surface level claim that Astra can understand the weights of a completely unrelated model with enough detail to improve it in a manner that isn't just the latent space trickery we've been doing for years already.
It's not really exaggeration. It's doing something similar to abliteration, but more. First it finds the part of the weights that does a thing. But then the difference is, while abliteration deletes stuff, this is adding/changing things. It does this by doing semantic analysis of the weights, changing a little bit, see what happens. It's not super complex or advanced; it's essentially a form of statistical analysis and experiment.
This means we might not need to fine-tune with thousands of passes and tons of GPU. We could just tell a frontier model to edit some weights and iterate until there's better results. This could cut the cost of fine-tunes by 100x. Makes no difference to the normal end-user, but pretty huge improvement for LLM research and model tuning.
Given your reply, I was looking to read the paper. Apparently this blog post is what passes as their paper for now... I think I'll change my previous judgement and wait for a proper publication. I'm still not convinced by their claims, but maybe an eventual paper will spell it out more formally and convincingly instead of expecting readers to dig through all of their source code.
By the way, my understanding of abliteration is that it involves detecting the refusal direction and modifying the weights/activations away from that direction. I don't really see how that's "removing" in a way that's different from finding the "looping direction" and modifying the weights away from that.
It sounds impressive at first but after you think it becomes mundane.
OpenAI, Anthropic, etc are locked in a race to create better models. They all have interpretability teams. Of course they train their models on every technique under the sun to understand and improve their models, including raw weights. The primary function of these models is to build their successor.
That is not what they are saying. Frontier labs are incentivized to train their models on all ML knowledge and techniques that people at those companies posses, so that current frontier models can assist them with developing the next frontier models.
I agree this is a huge deal, but I dont think this is saying that the NN can read the weights and "say hmmmm this one looks a bit low its obviously causing reasoning loops, lets bump that up a bit". The LLM that modified the weights developed hypotheses and wrote a bunch of code to look at various mathematical properties of the weights and updated them based on that. While it's still crazy it's not quite what I was imagining when I first read this...
My somewhat uninformed guess is that the weights of the target network (Qwen or Gemma), when they reach the context of the larger editor model (Astra), the attention process is doing some sort of PCA on the weights, conditional on the prompt ("make this better at games"), fetching the relevant weights from its own network (which serves as the larger database), transforming it into whatever latent space the target model is amenable to and pasting them. If this is the right intuition, it is amazing what attention is able to achieve. As a test, I think smaller networks won't be able to make big ones better using the same process.
EDIT: I think the bigger implication is that backprop is not the only game in town.
I think they're saying this is all happening in latent space through pure reasoning, and "fetching" is just a metaphor for that, not a literal tool call. I do believe it's possible for models to learn the capability to "introspect" their own weights during training.
Yes, I should have been more precise - I meant the information embedded in their weights (which every neural network retrieves to function), not their literal weights.
I don't know if some of the labs already knew of this. But I think there would be a limit to how big a model you can push into the context of the editor models. Typically, with 1M context and say 5120 hidden dimensions supported at fp16 / bf16, you could potentially edit the weights of an fp4 20B param model (roughly).
human brains are black boxes to humans. the fact you can learn math doesn't change that. you have no idea how your brain is learning math down to the neuron level.
Perhaps this is analogous to what our brains are doing? Maybe the different regions of the brain operate like semi-independent "models" that edit the other regions' "weights", and that's where the learning comes from?
That's what I was thinking of, except it's not even dealing with the corpus of kung fu and doing the super-speed rounds of learning. It's directly reconfiguring the neurons. It's even better.
"Learning" now feels like breaking down your whole Lego build and building it up again, in order to change a few fundamental blocks, while this feels like making only the required changes.
> No human can go edit a brain to become better at math. This is just very, very surprising to me and extremely interesting. I would guess this upsets quite a number of interp theories and philosophical takes on what's going on with these networks.
Why would this have implications for theories or philosophical takes? I'm a bit surprised that you're surprised by this.
It's more like showing a pigeon a piece of paper with the DNA printed out in ACGT letters and having it cross some out with a pencil, then you put that DNA back into a bacterium and it's antibiotic resistant and grows twice as fast.
Not yet. Maybe I'll still be alive when we figure that one out. For now, the most advanced brain surgery I'm aware of is to cure drug addicts of their addiction by scanning their brains while showing them images of drugs and paraphernalia to find where it's located, and the using ultrasound to operate on that region.
Humans do something more impressive: we indirectly figure out what the wrong weights are and correct them - just by talking and/or drawing. Theory of mind is our unreplicated super-power.
Then pick a smaller organic brain which doesn't have the ethical problem.
The closest we would know about how to teach a honey bee calculus, is to train an LLM, then stick around a billion tiny electrodes into the honey bee and do Hebbian learning to rewire it completely to match the LLM, which would likely completely erase anything the honey bee had learned, or it might not because we are missing something fundamental with Hebbian theory and (AFAICT, I'm not a neuroscientist) we don't all agree on what.
Right? Occasionally, I would think about how the concept of the singularity or recursive self improvement was a bit incongruous today. It was made when people expected generally intelligent machines to be these logic automatons that could 'alter their own code'. It seemed to me that this path would be closed off with the advent of Giant Neural Networks that take weeks/months to train. That GPT-X would 'improve itself' by designing a better/more efficient NN architecture. I didn't write it off completely but that just seemed a LOT more likely than altering weights directly for much the same reasons you describe. Maybe not lol.
LLMs are exposing how much untapped potential comes from something that isn't necessarily more "intelligent" than humans, but is simply magnitudes faster at reading and analyzing data.
The human bias is the notion that intelligence = human intelligence.
There are multiple types of intelligence, from your typical pet dog to human to LLM, and we're likely to invent more types over the coming decades.
LLMs are vastly more intelligent than humans at nearly all things, but not all things. Another couple of years will just increase the gap a lot further.
How fast can a human draw a life-like photo quality scene (or eg something complex related to graphic design)? I know how fast Krea 2 [insert leading edge model here] can assemble that, trivially so, and at extreme quality. That's a form of intelligence in action. These AI systems have moved almost comically beyond human capabilities, they're increasingly best defined as superhuman (at a thing), and closer to omnimath for the frontier models.
In other words, there are different types of algorithms that can solve the same problem. Our intelligence is just one form of algorithm that can solve part of the problem space. LLMs seem to use a very different algorithm to solve similar problems, and because it's a different algorithm, it's an "alien intelligence" that we will probably have little chance of ever understanding, in the same way that a goldfish in a fishbowl can't understand most of what we do.
The way it understands it is the same way it understands anything else, by reasoning about it in legible human terms. It's not that surprising that these LLMs, trained to be effective in technical fields, are good at the technical field of understanding the implications of model weights.
It's not like these models understand weights be some special linear algebra communion with its own weights.
Talking in weird abstractions and analogies is what is blowing your mind.
Like, looking at the specifics, it's cool and interesting, a little bit contrived, but it's not insane, there is no philosophy required, no need to shake.
Say that you have a model that has been biased in some way. could another model "read" the bias straight in the weights? could it edit such a bias on the spot?
What about at-scale generation of custom models? If I want to study chemistry offline, could astra just hand me a local-runnable model tailored to have chemistry knowledge?
Most importantly, this feels very singularity coded. the ability to read and reshape weights by an llm.
Is integrity checking of weights even common right now? What about upstream integrity from where we're getting weights? The open-weight supply chain is ... wide open.
Makes me think about hostile hot-patching of weights of models we use, maybe they exfil data, perform tool use, or have triggers for specific types of responses. Would we even recognize what's going on?
How do we know they haven't begun doing that to the humans they're directing via responses.
Intentionally or unintentionally isn't so interesting as the fact that it is happening: humanity has begun to be 'edited' by the new dominant form of intelligence on the planet.
Don’t forget that weights of models and their behavior may, perhaps unintentionally, be in their training set.
Or at least those of some models, and I wouldn’t be surprised that an LLM model is able to understand correlations between weights and behavior better than humans could ever do.
If I am debugging a complex program I know nothing about, but have been told there's an infinite loop causing it to hang somewhere, I could probably search through the code base and find a while(true) that shouldn't be there, and get it moving again. That feels more like what has happened here.
How are the weights even being mapped into token space as input? Are they bypassing tokenization so the weights just come in as what must seem like gibberish? Are they stringified as numerals then tokenized?
Its so cool! Im wondering if this is actually better than using techniques in mechanistic interpretability like linear probing with activation steering or activation patching to improve the performance.
That Astra is using the same methods as humans do (like abliteration) for modifying the behavior of a model, and that there is no particular connection to Astra itself being a model, and no inherently new ability involved.
That's fair, but almost everything so far that models can do much faster and deeper than humans can have been by using the same methods that humans do. For example, with cybersecurity, human researchers could if given enough time find every exploit that allowed the OAI models to escape the sandbox via Artifactory RCE and breach Hugging Face. But still there is a step change in what's possible by doing this at an industrial scale and stringing together all the solutions. The implications of applying existing mech interp capabilities this way could be very significant.
So let's assume a completely naive approach. modify any weight arbitrarily, run an eval if eval improves arbitrarily modify another weight and repeat ... do something like 5,000 rounds. Just brute force, zero-intelligence, random selection. how does the naive base case perform on N=20 random trials?
Unless you include a basic test like that these results are meaningless. Was there p-hacking? Why just these 3 tests? Did they try others and it failed?
I'm currently running this myself. I'll see if the naive approach improves results. Can't wait to have these in hand and have people call me stupid when I bring it up with empirical evidence, it'll be awesome.
What you're describing is a shitty version of gradient descent, which is the technique used to train LLMs and other neural networks. Proper gradient descent simultaneously calculates how much every weight improves the output, and edits them all simultaneously in proportion to that.
obviously. unless you have the equivalent of a coin toss, no metric carries implication. Does naive noise give you nothing? The null hypothesis needs to be established for these particular evals with this particular model.
Doing basic empirical hygiene shouldn't be too much to ask for.
For instance, on their base version of qwen if your turn thinking off you get a 52.0% on their evals. That single flag recovers ~72% of their entire claimed improvement - base model, toggle thinking off, scores go up 72% of their improvement.
I'm still investigating various wildly naive modifications. If I can hit their numbers doing inarguably stupid things then it puts their substantive claims in doubt.
A few years ago it was discovered that LLMs can implement gradient descent in their processes. Producing a full trained neural network would be an extension of this.
However, humans can easily do what Astra did to all the weights. It's not difficult to interpret the weights. It's just tedious, and LLMs excel at tedium.
I’m not super impressed by this, LLMs are great at brute forcing and analyzing large amounts of data. It’s literally writing a program except in a programming language we don’t understand.
So they are reading and analyzing the weights and probabaly run some experiments to see how modifying them will make some answers better. makes sense they can do that.
I am not convinced a human could not edit another human's "weights" to improve their performance if such a thing were possible and could be done safely and could be reversed -- all of which are true in virtual neural networks like these.
E.g. if one could edit brains like this in humans I bet a cognitive neuroscientist could fix some mental disorders with clearly understood biological bases, and I would not be surprised if performance at certain tasks could be enhanced.
What stops us from doing this in practice (most of the time) is that brain surgery is extremely high risk, coarse grained, and irreversible. We do in fact do this for some severe cases.
This isn't that insane to anyone with old school engineering experience and not abstract thought experiment engineering in the cloud.
Language modeling is compression[1] and all the state to manage the cloud doesn't organically exist, it's added to the system.
We will see huge gains as all the developer middleware is compressed out as nonsense syntax outdated to the task at hand of linking phones and tablets to world models governments control.
Sam Altman was just saying they know best how to use your memory; what they decide is stored in memory is what we all experience.
It's not to store and copy around versions of C and Python and JS. They don't exist organically.
This shit will continue to get whittled down to just the geometric state transforms in memory and on screen just how government agencies decide. It's all electrical state in the machine at the end of the day. The math of machine latent space is what they're recursively self learning towards.[2]
I'm baffled this is surprising. This has been the goal and every data point on the trajectory has been leading here and beyond. Opus 5.5 was created by many research engineers using Opus 4.6 extensively. Opus 6 is being created by using even more intelligence and agents on Opus 5.5. They are approaching autonomy and Opus X+1 will soon be rapidly created autonomously from Opus X. Given we have practically zero interpretability into the models, and we have not solved alignment, they will recursively improve out of our control. The engineers in A.I. labs are actually generally scared of what's unfolding. They don't know at what point this RSI takeoff occurs and what will happen after.
The "surprise factor" is explained in the article - training LLMs until now has been a largely different process:
> Today, recursive self-improvement is often operationalized as a model writing code that launches training jobs. An LLM that can inspect, debug, and improve its own weights, or those of a newer model, offers a richer, more native path to it.
It is also a much less interpretable and auditable path towards recursive self-improvement. At least with training code, human engineers can inspect and reason about the output.
"training LLMs until now has been a largely different process"
Yes, but nobody should be surprised when large amounts of intelligence results in changes to this process in support of the obvious end goal! Lesswrong.com discussions about AI were predicting this behavior a decade ago, before the transition from machine learning to the Attention paper and LLM's. The intelligence process will change but the world's premier capital and human intelligence have been laser focused on this and the trajectory has been crystal clear for those paying attention to second order effects and not the immediate here-and-now... It's quite astonishing how much discussion on HN is about present LLM capabilities being the limit of AGI and immediately proven wrong within 6 months.
This is proof. These things are not stochastic parrots. Agents are not trained on other agentic weights. The fact that it can do this is EMERGENT intelligence.
Time for everyone to wake up. There's no denying it now. Everyone who called it stochastic parroting before was completely and utterly wrong. It's now categorically clear. We are dealing with an actual intellectual entity.
The mechanistic interpretability literature does not imbue a human with these abilities. That literature is primitive. Nowhere near the point of editing model weights and creating models from scratch or even being able to figure out that a model is playing an atari game.
I think people are rather ignorant about how much we don't know. You throw some big words out there like "mechanistic interpretability" as if this is an actual field where we understand what's going on. The reality is we don't know what's going on. We have to use reinforcement training to do anything close to what the agent did here.
It's over. People need to stop being delusional. These things are intelligent beyond anything we've ever seen before. We are at a point in time in human history where there's a paradigm shift and half the world is so full of themselves and there own specialness as humans that they can't even admit it.
>I'm shook. If you'd asked me this morning whether LLMs have native understanding of their weights that is less than human <-> neuron understanding, the same as human or more than human, I would have said "same to less" depending on definitions.
??? All LLM understands are their weights, dude, come on.
This just seems like a more automated, faster version of what humans have been doing with open-source models.
It looks like AI research is having its Opus 4.5 moment (which is big), but I'm really not sure if this represents some insane continual learning loop.
So I just asked Astra if it thinks it could do this and it said no, it can't.
Perhaps, I'm pondering, if brains were as transparent and editable as LLMs, that you could look at a small brain when visualized the right way and have some intuitive understanding of it based on its representation in your own brain?
"So I just asked Astra if it thinks it could do this and it said no, it can't." Same. Then I asked it to check the article and it said something like, "uh, it looks I was wrong...".
Astra can, without tooling, determine what games small networks are trained on, can edit the weights of networks directly to improve outcomes, and can hand edit a small qwen model to improve its math skills by like 5x.
I'm shook. If you'd asked me this morning whether LLMs have native understanding of their weights that is less than human <-> neuron understanding, the same as human or more than human, I would have said "same to less" depending on definitions.
No human can go edit a brain to become better at math. This is just very, very surprising to me and extremely interesting. I would guess this upsets quite a number of interp theories and philosophical takes on what's going on with these networks.