If people moved to your scheme MITM would become common and we'd be writing about FirePiggyInTheMiddle or similar. They can be automated. You need some trust infrastructure for encryption to work.
In order to do FirePiggyInTheMiddle, you need to control the router. You can't just sit in the subnet and start sending commands to random clients to hijack their connection, especially when it is symmetrically encrypted.
Ok you guys can stop with the "but what about this" now.
We all realize there are 1000 different attacks with various levels of difficulty and all of which have appropriate countermeasures and are nowhere near the success rate nor ubiquity of simply receiving packets.
