Agreed. Consider that we don't even know if one-way functions exist, and all public-key cryptography depends on the assumption that they do. Maybe some classical attack will break all the commonly used systems before a big enough quantum computer is built.
