Also bear in mind what future positions you might have available.
I remember about fifteen years back I was interviewing for somebody who could grow into a technical lead (they needed to come in as a developer and help build a team around them).
One of the people we talked to was completely unsuitable for that role - but was amazingly impressive in other ways ("I do Linux. Here's a phone number where you can dial up into my Linux box and play with the web server." In the mid-90's this was... unusual... for the average candidate).
Six months later we had a technical-support-growing-into-ops role. Guess who was first in line for that job.
As far as I am concerned saying 'thanks but no thanks' politely and quickly is all win. It makes the candidates keep some respect for you, helps the company reputation, overall saves time (IMHO) due to the people who repeatedly nag, ensures that you have good concrete well defined hiring criteria, etc.