> We think MAIA augments, but does not replace, human over- sight of AI systems. MAIA still requires human supervision to catch mistakes such as confirmation bias and image generation/editing failures. Absence of evidence (from MAIA) is not evidence of absence: though MAIA’s toolkit enables causal interventions on inputs in order to evaluate system behavior, MAIA’s explanations do not provide formal verification of system performance.
For folks who are more familiar with this branch of literature, given the above, why is this a fruitful line of inquiry? Isn't this akin to stacking turtles on top of each other?
I think what authors aimed for is perhaps a proof-of-concept work where they attempt to demonstrate that you can (to a degree) automate interpretability. Mech interpretability is challenging because it does not scale well at the moment, and there is a debate about whether localized structural discoveries on toy examples actually translate to patterns in large networks. My guess if you could build an automatic explainer system this would allow you to flag problems and find issues faster, basically as some sort of meta-heuristic for further investigation
Unfortunately, that title hypes it up, and as always, once you read the paper, the results are less impressive, but that is what the state of AI research is currently, speaking as a researcher myself.
That's basically a known fact about LLMs, they need oversight. But if they make the task 100x easier, it's still useful as a starting point. This kind of neural net analysis is difficult to do manually.
I am curious if they just start making inventories for all neurons in all layers, then they can compare models based on neuron types, or even train them to achieve the right mix of concepts.
For folks who are more familiar with this branch of literature, given the above, why is this a fruitful line of inquiry? Isn't this akin to stacking turtles on top of each other?