"Machine learning models have no access to such experiences and thus cannot "understand" their inputs in any human-relatable way"
It may be that distinctions like the one you're describing here are useful to make, but I don't think this claim refutes the possibility of ML "fitting a particular piece within a larger, yet unarticulated model."
I think the assertion is more that our current ways of representing elements of human experience are necessarily very lossy - or that there's some aspect of the situation that you can't describe/implement in terms of models of neural nets.
The problem with neural nets is that they have a fixed input type - tensors or sequences. For example, imagine the task is to count objects in an image and say if the number of red objects is equal to the number of green objects. You make a net that solves this situation. Then you want to change the colors, or add an extra color, and it will fail. Why - because it learns a fixed input representation.
What neural nets need is to change their data format from plain tensors to object-relation graphs. The input of the network is represented as a set of objects that have relations among them, and the network has to be permute invariant to the order of presentation. An implementation is Graph Convolutional Nets. They learn to compose concepts in new ways and once they learned to count, compare, select by color, they can solve any combination of those concepts as well. That way the nets generalize better and transfer knowledge from a problem to the next.
Graphs are able to reduce the complexity of learning a neural net that can perform flexible tasks. But in order to get to even better results, it is necessary to add simulation to the mix. By equipping neural nets with simulators, we can simplify the learning problem (because the net doesn't have to learn the dynamics of the environment as well, just the task at hand). Examples of simulators used in DL are AlphaGo, the Reinforcement Learning applications on Atari Games, protein/drug property prediction, generative adversarial networks (in a way).
The interesting thing is that graphs are natural for simulation. They can represent objects as vertices and relations as edges, and by signal propagation the graph works like a circuit, a simulator, producing the answer. My bet is on graphs + simulators. That's how we get to the next level (abstraction and reasoning). DeepMind seems to be particularly focused on RL, games and recently, relation networks. There is also work on making dynamic routing in neural nets, in fact applying graphs implicitly inside the net, by multiple attention heads.
There is actually work being done on this problem, at least to some extent. A DNC, for instance, can accept variable-structure inputs by storing each piece in its external memory bank. This is illustrated in the original Nature paper by feeding in an arbitrary graph definition piece by piece, then feeding in a query about the graph.
This doesn't necessarily address all the nuances of your post, but I do believe it's a step in the right direction. It pushes networks from
"learn how to solve this completely statically defined problem via sophisticated pattern matching"
to
"learn how to interpret a query, drawn from some restricted class of possible queries; accept variable-structure input to the query; strategize about techniques for answering the query; and finally compute the answer, possibly over multiple time-steps"
It may be that distinctions like the one you're describing here are useful to make, but I don't think this claim refutes the possibility of ML "fitting a particular piece within a larger, yet unarticulated model."
I think the assertion is more that our current ways of representing elements of human experience are necessarily very lossy - or that there's some aspect of the situation that you can't describe/implement in terms of models of neural nets.