If the Govt/NSA wanted access to certain metadata and a company refused (like some claim Twitter did), what's to stop them from going to Amazon or Rackspace and throwing their weight around to get access that way? Or, if that didn't work, they could just keep going up the OSI layers (or tier 1 providers) until they get the access they want OR can force it be threatening to disrupt service.
My point is, even awesome companies like Rackspace are dependent on less-than-awesome companies for some types of infrastructure.
Any company that doesn't comply and hand the data by API will have it's links scraped. Sure it's more costly. That's why they go through the arm twisting.