Just adding my personal experience: my sites went from a 30ms response time (measured by Google's webmaster tools, as Googlebot sees it) to over 250ms on CloudFlare, for static (cacheable) content.
Turning it back off gave a very clear improvement both in my own experience and according to Google's chart.
Turning it back off gave a very clear improvement both in my own experience and according to Google's chart.