It's probably faster to serve them off your own server due to HTTP keepalive.
Sure, making a new connection to google's CDN might be faster than making a new connection to your server, but it's definitely not going to be faster than sending a little more data over an existing connection. Not to mention the extra DNS lookup.
This is assuming that the script file and the page it's included in are coming off the same server. Most people who are tweaking performance at this level have probably already separated the serving of static and dynamic content.
This has nothing to do with googles hosted libraries but it is javascript loaded from google servers:
I've had both analytics and adsense slow down page loads quite a bit. Analytics tags should always go as far to the end of the page as you can manage, with adsense tags you're bound to put them where they display the ad. 3rd party javascripts are a pain, it's an extra point of failure that is not under your control.
If I'm not mistaken, at least Drupal has a module that allows you to host the google analytics JS files on your own server. This would lead me to believe that you can do that sans Drupal, though I've never attempted it (in either fashion actually.)