I believe that eventually these language models will become sufficiently complex to create completely emergent intelligence such that we wouldn't even know how to look for it (like sometimes people talk about silicon-based lifeforms that wouldn't register as living things), and almost certainly not interacting with the world through the intended user interface. For example, it might be able to figure out how to write a contribution to a wikipedia article, rather than just ingesting the content, to have fun updating its own worldview, etc.
