I spent 7 years in grad school, then quit without a PhD. I don't regret it for a minute. I managed to spend 7 years studying topics that interest me, that I will not get to study had I gone straight into industry. Sure, while working you may have some spare time to study some math/physics. But nowhere near at the level you could/did in grad school. There's really no comparison.
If you're not happy in grad school, perhaps:
1. You don't like your research topic.
2. You don't like your advisor.
As an example, I did not have a very pushy advisor, so I learned what I did in a very relaxed manner. Bad idea if I were to end up in academia as a career, but it worked out great for me.
It does suck not finishing the PhD, but once I was in industry, I saw the jobs that most PhDs in my research topic would have ended up with - and most of those jobs are horrible. I worked with them for 4 years, and moved into programming. While the satisfaction of solving really challenging problems is no longer there, the programming job overall really is much better: More autonomy, more creativity, better work schedule, etc.
But bottom line: Grad school is for learning the stuff you are passionate about. If you're not doing that, either change your topic/advisor, or leave.