In my own impression, math is best done with total disregard for applicability, which is to remain entirely accidental.
Concerning "The problem is epistemic: why is mathematics, which is developed primarily with aesthetic considerations in mind, so crucial in both the discovery and the statement of our best physical theories?"
Abandoning the primary goal of "aesthetic considerations" would undoubtedly lead to such math becoming unusable, or even degenerating into non-math.
Therefore, I believe that in everybody's best interest math should stick to its own goal of exploring mere aesthetics.
You've missed a step here. No one thought to do math until people started to seriously worry about proportions, distance, quantity, time, etc of the natural world. Once you've set up those rules (and necessarily rigged them in favor of explaining the natural world) you can stay in the platonic realm, change the rules so they apparently stop reflecting reality, and rely on your sense of aesthetic, but then I think the onus is back on you to explain where this sense comes from.
I consider applicability one of the primary aesthetic qualities in mathematics, but most mathematical applications are internal to mathematics or science, not about solving some highly specific practical problem like improving web search performance. Applications of complex analysis to proving the prime number theorem or to designing airfoils are in the same category for me aesthetically. They both enrich my understanding and appreciation for complex analysis. The applications might be more or less shallow. The applications of complex analysis to number theory go very deep. But potential theory is also a beautiful and theoretically rich subject that extends into higher dimensions, and the airfoil design example is striking because it relies on the non-rigidity of conformal mappings in 2D, which is the main feature of potential theory that fails to extend to higher dimensions. If you study any application, it will invariably improve your understanding of the original ideas.
Concerning "The problem is epistemic: why is mathematics, which is developed primarily with aesthetic considerations in mind, so crucial in both the discovery and the statement of our best physical theories?"
Abandoning the primary goal of "aesthetic considerations" would undoubtedly lead to such math becoming unusable, or even degenerating into non-math.
Therefore, I believe that in everybody's best interest math should stick to its own goal of exploring mere aesthetics.