1. Before the near-universal requirement for confirmed opt in, it was a reasonably common attack/prank to sign up someone else for tons of lists, causing them to receive unmanageable amounts of email per day. Alternatively, sign them up for lists they'd consider offensive, or that'd get them in trouble at work. Confirmed opt-in stopped this attack.
2. We've (computer geeks, security experts, etc.) spent years trying to teach users that no, they really shouldn't click on that link in an unexpected email, even if it promises cute kittens. That continues to be a problem. I don't know how an expert, much less random Joe, could tell a real "click here to unsubscribe" link from a fake one, if you have no idea who the sender is. Researching if its a legitimate unsubscribe link—and not a link that'll confirm to a spammer that you're reading his email, or pull up a page full of the latest browser or Flash exploits, makes it far more expensive for the random person to handle an opt-out than it is for the person who wants the subscription to handle a confirm.
Confirm emails are easy for a random person to handle: just ignore them. You won't receive any more emails if you take no action.
3. Not all email addresses are nearly-free to send to. Well, they're all nearly free for the sender, but not the recipient. Some email addresses cost 20¢ per message (e.g., SMS gateway). Some email addresses page people, waking them at night. These addresses aren't always very different from other people's addresses (I've personally had misdirected mail to both types). Someone can also do this as an attack. Sending a single message is bad enough, but is the minimal harm you can do; but imagine if anyone could subscribe your text message address or pager number to linux-kernel.
4. Realistically, while some mailing lists are very easy to get off of (e.g., development list for a random open source project), many commercial lists aren't. I've had people give my email address (assumably by mistake) to the College Board on their PSAT (hundreds of emails per month, many from different senders, no way at all to stop them); sign me up for an Xbox account (took me at least half an hour to figure out how to stop that—-apparently you're supposed to phone Microsoft; thankfully I finally found another way); sign me up for various things at EA (nothing on their web site, contacting support was absolutely useless); etc.
5. Your ISP probably has policies against sending unsolicited bulk and/or commercial email. If you aren't confirming your subscriptions, you can expect your ISP to receive complaints, and you are probably in violation of their TOS.
Not sure how much this still happens, with spam being so much cheaper to send with botnets.