Take a minute to skim through the history of a bunch of "successful" companies on that rocket trajectory. Does it sound like it made the founders happier? Or do they seem like they were actually doing better at some sweet spot earlier on in the story?
That's what it always seems like to me. So when I built my business, I set out with the goal of finding that sweet spot of good profits, minimum workload, no staffing or investor headaches, and most importantly, maximum Jason freedom.
Reading your comment, it sounds like you did the same. Don't worry for a second about what might have happened. What did happen sounds pretty cool.
Founders can look back and say "I could have had more" or "I could have been like X multi-million dollar company", but then the company is no longer a side project. Then the company is your life, which seems contrary to the initial motivation if the point of a side project is to not work 24/7 in a CEO lifestyle filled with meetings and managing.