And although 'I'm not looking for a new challenge' is fine if it's honest, it's a lot less engaging than 'I'd be very excited to join your team for such and such reasons' i.e. a specific, legitimate active interest in something the team is doing is a bit more ideal than the neutral/passive 'not really looking'.
Counterpoint to that is when a recruiter initiates contact they are often very coy until they have you on the phone, frequently not even sending a JD until after that first conversation (where this line of questioning usually arises). I get wanting to provide an engaging response but hard to do so until the recruiter has provided sufficient information.
And although 'I'm not looking for a new challenge' is fine if it's honest, it's a lot less engaging than 'I'd be very excited to join your team for such and such reasons' i.e. a specific, legitimate active interest in something the team is doing is a bit more ideal than the neutral/passive 'not really looking'.