The recruiters receive details of a job-opening from a company (hey, we're hiring), calculate how much the fee will be to them (OMG $20k!!) and then proceeds to cast a wide net by spamming twitter, email, cold-calling, linkedin etc to get as many CV's as possible then cut the list down and try to get them into interviews. Its driven by the payoff because if a recruiter does their job amazingly well, then they place you in a good job and you never need them again... companies on the other hand are always reruiting so better to help the company.
...the problem is all those people who sent in their CV and didn't make the first cut are naively under the assumption that you are proactively searching for an opportunity that can help nurture their dreams/desires and provide room to grow in a new role etc etc. Rubbish.
I don't know the answer, but thats my take on the situation - based on experience of going through the system a few times with different recruiters.
Some recruiters do actually have your best interests at heart and work hard to find you the right role. Why else would you go back to them in the future or refer your smartest friends?
It's a good sign if a recruiter refers you to a competitor or is 100% honest that they don't think they'll have a position for you in the near future. Any recruiter who operates with a shred of integrity will be glad to do either.