Elegant though it is, I don't think that idea will work; people who I don't "follow" can "message" me in Twitter by including my name in a posting. That doesn't work if I explicitly have to be polling their server to see anything they say.
No, people who you don't follow can ping you saying they've sent you a message. Your software can simply drop them on the floor if you don't care to receive messages from people who you aren't following (AKA people who aren't on your whitelist.)