Rather, I've always imagined that it's not that it'd be closer to the perks like "unlimited vacation time": you can't just decide "well I'm taking the next 5 years off", because that'd be a dereliction of duty. In other words, if you have the premier release of your main project coming up in 2 weeks, it may be in your best interest to not take that 20% time for the time-being, but instead use that time to fully ensure that your project has a successful launch. But, if you don't have any big deadlines coming up soon, and you're reasonably on schedule, then by all means take the 20% time.
The 20% time is a perk, and just like many other perks, your use of the perk cannot preclude any tacit duties that you may have, which include delivering on your assigned projects.
But, this is just an outsider's view, I could be completely wrong.