There are many niche skill sets out there where the demand far exceeds the currently supply, and as such hourly rates from $100/hour (lower level) to $200+/hour (architect level) are quite common and you really can pick your job/location/work from home, etc... The trick is finding one, and learning it quickly/well.
I've often thought about grabbing a bunch of folks with a similar relevant skill set, investing 6-12 months in training them, and then running a hugely profitable consulting company for a while. The problem being it's hard to prevent people from hanging out for your 12 months of training, and then jumping ship to do their own thing afterward...
The kind of expertise that consistently commands the high rates requires a level of experience that cannot be developed through training. It requires in-field domain expertise developed over a few years. What you are talking about is what many consulting companies actually do, and it is one of the reasons that consulting companies have such a poor reputation for execution. Customers are paying for the expertise you will not pick up in training or by reading a book.
I ran a successful consulting business many years ago (and hated it). One of the things you learn is that you live and die by the quality of the people you can deploy because that goes straight to your reputation. In practice, consulting businesses are very difficult to scale. While you may start out high-end with a great reputation because you really know what you are doing, it is very difficult to bring in additional people with your level of skill because there is almost no overhead to them working independently. As a result, "scale up" often involves diluting the quality of your overall team to the point where the rates you can charge decline as well. Consequently, you either have to run a very small boutique consulting shop that can maintain its standards and billing rate or run a large generic shop that makes up for the lower billing rate in volume. For people that care about the quality of their work, the former is more rewarding but the economics work out such that you will make about as much money by going direct as a highly-paid contractor to another company.
In this particular case I'm not talking about grabbing random people, I'm talking about people who are very skilled in a similar tech stack (which has a much higher supply/demand ratio, and getting them up to speed with the very similar but niche stack. Basically you're just learning some new APIs and some new service providers (i.e. proprietary ORM instead of Hibernate).
But I agree with your assessment of consulting firms and the scaling/quality issues therein. I also feel that consulting shops expenses tend to scale a lot closer to profits than some other businesses, so I stay out of it:)
How does one go about finding developer resources for closed applications like Oracle ATG (or really any of the above list)?
I'm employed by an eCommerce vendor myself, so I'm sure I'd have an intuitive understanding of general architecture, but actually understanding the internals of a system as an outsider seems like a difficult proposition.
Until Oracle bought ATG it was basically impossible. But Oracle has made ATG available for download, including documentation, to anyone, without needed a million dollar support agreement. You can't run a production site for free obviously, but at least now you can read the docs, and play with the software locally.
The only real hope of learning it though is to have a company that you work for use ATG, send you to training, and get you coding on it every day.
This is why there is such a huge gap between the demand and supply of this skill set currently.
They're cutting some of the fat off, but they just dropped 1 billion dollars on the ATG product and are pushing it pretty hard and Oracle doesn't have anything that's a real competitor in the space. It'll get renamed as Siebel Commerce or something, but it's not going anywhere for a long time...
Note: what I'm about to say has zero technical base since uh, you know, I don't care about technicality or purity, we're talking about pure business direction here.
I'm unsure regarding Scala since a few articles have risen regarding getting burned by Scala (Yammer) or plainly suggested that Scala may not be it (David Pollack, one of the Scala celebrities).
Having said that, knowing Java developers, most (not all) of them would definitely flock to either Scala or Groovy instead of LISP. Have you heard anything about Groovy lately? Yep, nada, cricket. The cool kids (Yammer, Twitter) are using Scala (to some extend) so there you go.
Clojure on the other hand has that "LISP" tales behind it (LISP programmers are like gods or something like that) and it runs on Java so that gives people some kind of hope and smile on their face or something.
I don't know if either would be in high-demand but you may want to look around and do a bare minimum, out of the thin-air, lots of grain of salt type of assumption:
If someone is making a consulting agency that gets clients, deals with most of the risk, etc., then that is worth something to me and I'd be willing to take a less then I could get if I were doing the same work freelance. I'd give up, say 10-20% of the value of the contract to pay for such services. If the owner wants to take much more than that then he/she will have to expect people to jump ship after training. The salary difference is worth taking on these other responsibilities yourself or finding an agency that will take less.
As far as your second example of people, I don't think enough people are like that to warrant consideration. People are more likely to stay with a company beyond the point of rationality.
The problem is that if you spend 6-12 months training people you've got WAY more costs to recoup than a normal contracting firm so you'd have to take a higher percentage off the top. The rates would be higher for the niche so the take home would still be good, but a smart person would see you taking 30+% off the top, and know they can only give up 10-20% at another place or none by going freelance once they have some network in place. The sunk costs of the training and bench time around that are very hard to make back.