Yes, patents infringe on people's rights, to foster innovation. Getting to patent something isn't a right, it's a privilege. As you've illustrated, all patents are inherently undesirable--at least for certain parties. And I agree. They should be abolished.
That is precisely what they don't do. All they do is stop someone else from claiming it as an innovation with the associated IP rights. There is nothing to stop someone innovating like a fiend. But surely it is blindingly obvious that copying something someone has already patented is not innovating. Everything else however is fair game. Knock yourself out and innovate.