With time zones, you have to know which time zone you're in and set your watch accordingly. With no time zones, you have to know what time local noon is and interpret your watch accordingly. The amount of information required, and the difficulty of using it, appears to be the same to me.
For the common case of infrequent changes in time zones (relative to the number of times one does a time lookup or talks about time), I think what we use now is less difficult to use. You only have to make the adjustment once, instead of every time you talk about times.
It seems to me like the opposite is the case. I have to coordinate times across time zones way more often than I actually travel (for online meetings and such). If there was a single time zone, I'd never have to make adjustments for those, and only adjust things when traveling.
The problem there is that in a lot of cases you pretty much use timezones implicitly anyway. If you want to call someone in China from North America, for instance, if it's 3pm where you are, you might know that it's 3pm in China (in this hypothetical no-timezone world), but that still doesn't tell you if it's a reasonable time to call. You have to think, "Well, people in China usually work from time X to time Y, so their time X is my 9am, which means the offset is Z, which means 3pm in China is equivalent to my 3pm + Z, which means it is/isn't a good time to call."
You basically have to reinvent timezones every time you make any kind of calculation involving people in another timezone.
OK. So you are traveling and your plane goes at 15:00 (with this hypothetical clock, I would drop the PM stuff). Do you have time for lunch beforehand? Should you try to?
Similarly, you are on holiday and someone tells you a museum is open till 04:30. Can you go to dinner first?
I guess that a main question is what kind of query is more frequent: "What time is it now in X", or "At what time is X where I am now?". I thought the former was way more frequent, but now, I do not know anymore.
Maybe the best thing to do is to drop absolute references altogether. IMO, "Shall we meet in 3 hours" is easier for handling across timezone discussions, and will work equally well for "in a different timezone than the one I am used to".
Of course, your SMS/ping/twitter/email client would have to automatically count down such timers for you.
Your travel examples are indeed more difficult, but I travel much less than I coordinate times across time zones, and I suspect most other people do as well, even if it's something as simple as calling family in another state.
For someone who travels a lot, a single time zone may well be more inconvenient, hard to say.
Times relative to now would work well but could be unwieldy. How do you do it if you're e-mailing or texting? If you want to meet next Monday afternoon, do you say "in 2 days and 5 hours"?