This seems like exaggeration, the paper seems to describe something more like abliteration, where Astra orthogonalized out activations related to undesired behavior.
This is obviously very different from the surface level claim that Astra can understand the weights of a completely unrelated model with enough detail to improve it in a manner that isn't just the latent space trickery we've been doing for years already.
It's not really exaggeration. It's doing something similar to abliteration, but more. First it finds the part of the weights that does a thing. But then the difference is, while abliteration deletes stuff, this is adding/changing things. It does this by doing semantic analysis of the weights, changing a little bit, see what happens. It's not super complex or advanced; it's essentially a form of statistical analysis and experiment.
This means we might not need to fine-tune with thousands of passes and tons of GPU. We could just tell a frontier model to edit some weights and iterate until there's better results. This could cut the cost of fine-tunes by 100x. Makes no difference to the normal end-user, but pretty huge improvement for LLM research and model tuning.
Given your reply, I was looking to read the paper. Apparently this blog post is what passes as their paper for now... I think I'll change my previous judgement and wait for a proper publication. I'm still not convinced by their claims, but maybe an eventual paper will spell it out more formally and convincingly instead of expecting readers to dig through all of their source code.
By the way, my understanding of abliteration is that it involves detecting the refusal direction and modifying the weights/activations away from that direction. I don't really see how that's "removing" in a way that's different from finding the "looping direction" and modifying the weights away from that.
This is obviously very different from the surface level claim that Astra can understand the weights of a completely unrelated model with enough detail to improve it in a manner that isn't just the latent space trickery we've been doing for years already.