If the data set is large enough then there is no way to represent it as a functional relationship between finite dimensional vector spaces. In fact, this problem already is visible in existing large neural networks because they can only work with data that conforms to the dimensional constraints of the input space. It's why image transformers trained on NxM images don't work on any other grid size.
ah. I mean, the dataset can be infinitely large and still be
covered perfectly by a function, if it was generated from a function. That's why absurd functions can be found that overfit for some subset of basic real world results. (Ask anyone who built a funky physics system for a videogame in the 90s). I think what's more interesting is the question of what that essential function is for a stop sign or a pedestrian, as opposed to the function for finding it in a 512 grid or something.
I suspect the idea is that there can be many functions that fit a particular vector, because there are more functions than vectors, but I nay be wrong.
