Is there a meaningful / well defined difference? One could say that lossy compression is extraction of useful information. You need to identify unnecessary information to know what to discard safely.
One is a subset of the other. Compression being the larger category.
A mean is a compression of a dataset and useful information.
My issue in my comment is that 'compression' corresponds to a massive class of technquies
and there isnt a lot of content in the observation that useful information is compressive.
However there is some hypey people out there who think this observation has legs. Precisely the people who think intelligence is a mathematical problem, and not an engineering one -- which is my view.
ie., that a body isnt incidental to intelligence, but the heart of it.
I tend to think about it the same way as you. Having an algorithm for multiplying two numbers is qualitatively different than having a lossy compression of a huge dataset of multiplication tables. The latter is what GPT3 has and it just doesn't scale.
Consider a generative adversarial network for faces. Photos have let's say some unique scars
A successful model could create faces with scars in them. But not the exact scar and face and background that they trained on without additional information. What you are looking for is mutual information between the images not compression
Yes you could use a very well trained GAN for face compression. But a GAN model itself would not be able to reconstruct its training input without being shown the images again
