There's some incentive not to do that, however, because in many cases it will inconvenience you just as much as everyone else. But if you could just deselect all the floors between the current one and your destination, there's a pretty strong incentive there.
Until someone punches you out, of course, because it'd be hard to hide and you're in a confined space with them...
Elevators sometimes have an anti-mischief feature where they will deselect every floor if all the floors are selected in a brief period of time. The cab will slow to a stop if no new buttons are pressed. It’s kind of eerie, actually.
The elevators at my office were upgraded a few years ago and one of the elevator techs happened to mention it to me. I tried it out while alone, for obvious reasons.