German maintains the difference (and a lot of other stuff), but yeah, as a substitute for the direct object it's pretty much dead. (Nobody says "Whom did that")
"To whom it may concern" is formulaic though, just like "Dear Mr. President."
Those are words used as they are, verbatem, in a specific context. They are thus ossified and preserved in a much more conservative state than the rest of the language.
It is not unreasonable to suppose that even as our language changes, that phrase will continue to remain intact in root and morphology even if the morphology disappears elsewhere.
Thank you! Your Sapir reference led me to Googling "sapir whom", which led me to http://www.bartleby.com/186/7.html. Paragraph 10 is where his "whom" discussion starts. Good reading.