There's a nice talk by Yann LeCun where he goes on to explain really well why deep learning has such fast progress. [0]
He goes on to explain how theory always comes later.
I thought that information theory came before practice but turns out it also came after. (there was a bunch of heuristics for sending messages with teletypes)
A nice example is Roman technological advances in architecture and materials that predate the use of geometry or mathematics completely. All advancements a result of tinkering and heuristics.
It's not so simple as "what comes first?" (theory or experiment). What generally happens is that we stumble upon something, build a rudimentary but useful or interesting piece, then try to understand it, generalize it, and expand it (and usually succeed, with science). Both happen iteratively.
Information theory came after the telegraph and early communication systems. However, we could not have built modern communication devices without Information theory (the insights and design principles). We build, then we theorize, then we build better, etc. it's not a simple procedure. Computers were developed similarly: there were all sorts of ad-hoc logical apparatus, we built boolean theory to explain it, and then we did all sorts of experiments trying to build computers. Their architects were largely mathematicians with very good ideas of mathematical design principles (and creative new mathematical ideas), not a group stringing together electrical elements and seeing what happens. The same goes for the development of ML/Deep learning, and many other technological marvels.
Although "accidental" discoveries do happen, they happen from a methodical set, with knowledge, intuition, and good priors.
A similar historical pattern is the invention of the steam engine, and then the theoretical framework of engine & thermodynamics formulated by Carnot. (Carnot engine)
The theoretical framework came almost 100 years after the steam engines original invention
So in that sense forming a theory about something is integrating all the practical stuff you've seen before. To me that make sense, IMO that "practical stuff" are what experiments are, aren't they? Probably the experiments are far from perfect, if remotely good even, but that's how empiricism should work? I think? (hmm... not entirely sure)
It can go both ways. E.g. often, experimental physicists will try to prove/disprove a priori predictions from theoretical physicists.
Theory is just an explanation of hw the world works: you can come up with that explanation as a reason for observations, or as a logical consequence of other theories (which is then verified by observation).
He goes on to explain how theory always comes later.
I thought that information theory came before practice but turns out it also came after. (there was a bunch of heuristics for sending messages with teletypes)
A nice example is Roman technological advances in architecture and materials that predate the use of geometry or mathematics completely. All advancements a result of tinkering and heuristics.
[0]: https://www.youtube.com/watch?v=gG5NCkMerHU