As the ad in the in-flight magazine says (I think I remember right), you don't get what you deserve, you get what you negotiate.
The impact on your visibility, personal brand, and ability to find future roles should be a factor when you sign up for a job. Just as you'd consider your job title, for example.
If you're signing up for something you can't talk about or show off -- worse, if you're signing a contract that says you can't do anything that you _can_ talk about on the side -- then you'd better be sure you're getting paid extra, or in some other way getting compensated.
From what I've seen, working on super-secret trading software for Wall Street _does_ pay a lot more than working on an open source project, on average.
If the price is that you have to use references and other means to show future employers what you can do, then that's the tradeoff.
If you're not getting paid much, have no spare time, aren't allowed to code in your spare time, etc. then those are some items for the "cons" column that might nudge you to look for something new, all else equal...