"do not guess that it will be your weakest opponent nor guess what they will try, prepare for your strongest adversary and all that they can do" -- mutatis mutandis, Sun Tzu, Art of War
Not to disagree with infallible thousands of years old advice, but surely resources are in far shorter supply than potential attacks, so we have to prioritize and filter.
Realistically though, they'll probably timeout by themselves automatically if they haven't seen a password prompt after N seconds. TCP connections can hang overall, so having that would be basics anyway.