Its pretty obvious - you are giving third party direct access to your visitors, a spying beacon. Same case as FB Like button and other social media garbage https://easylist.to/easylist/fanboy-social.txt
Of course there is Referer header send with every request, thats the whole point of Google providing this "for free out of the goodness of our heart". Same reason they offer "free" DNS servers and why Chrome will call their DNS server even if you explicitly did NOT want them used.
Its all used for tracking.