Yes, fast-growing companies can grow out of my preference zone and as other commenters said, jumping ship when that happens may be the correct way to go.
There are other issues with slow-growing or not-growing companies. When the company is not growing, people are incentivized to take a zero-sum approach to their work relationships. If the pie is not growing, you need to guard your own slice and take from others. This creates a toxic environment. If the company is growing, then collaborating on growing the pie can become the shared attitude.
Slow growth and no growth companies tend to be under a lot of pressure for cost-optimization (which makes sense in a lot of ways, but is grueling to live through…)