I share the sentiment. It was my wife's (then girlfriend's) explicit wish that I don't chase my passions in such an environment, and I'm really grateful for that after the years.
It's probably also the case that (not so well-paying) passion jobs become less interesting when they're your full-time, perhaps long hours, job and you don't really control what you work on. I was pretty passionate about photography once upon a time but I realized (correctly) that I'd probably end up doing photography for some small-time newspaper (when those still existed), some organization's public relations, or commercial photography--and not Life Magazine.