In my experience the Webmaster Tools performance indicator is terribly inaccurate for small sample sizes (relatively unpopular sites).
For one site I have intimate details of, Googlebot visits roughly one page every three minutes. I have every indication (knowledge of what the site is doing, local and distributed load testing, analysis of server logs, etc.) that the site performs consistently and well, but the Site Performance chart on Google's Webmaster Tools is all over the place: anywhere from 0.4 seconds to load (99th percentile) (which is where it happens to be now and is probably slightly overestimating the site performance) to more than 9 seconds to load, oscillating pretty wildly. I'm fairly confident this isn't accurate (unless GA, the only external component on the site, is itself to blame).
On other, more heavily trafficked sites the Web Performance chart seems much closer to reality--in line with our load test results and anecdotal experience, so whatever the issue is it seems to pan out when they take a large enough sample size.
Edit: I see in other comments here that Google is using end-user data from the Toolbar to track this. That's helpful information and may explain the variability I and apparently others have seen. Again, this is a good reason to hope they'll only take this into account for large sample sizes. Otherwise, hope that your customers aren't using dialup connections.
That will include a whole lot of empty caches. But it will be representative as well then.
Is anyone else having similar weirdness with their Site Performance graphs?
The worst offenders to the speed of my site is that I have both google adsense and google analytics on it. I find it ironic.
From what I understand the time is sampled from visitors, and includes the time from the page request until the page is fully rendered.
I'd love to know what people do that get their times well below sub-second.
They don't say though that this is the same data being used in their ranking factor.
It's pretty clear they're just using this to weed out broken sites that take forever to load. If your visitors aren't already bouncing because of long load times, you probably have nothing to worry about.