It astounds me that people use Rails and end up doing all the craziness with making custom APIs when Rails already does it the right way out of the box. That they then to go on and pat themselves on the back and call what they've done "RESTful" when they've made an API that's less RESTful than what comes out of the box with Rails is bizarre.
In Java-land, Apache Jersey is also doing it the right way (content negotiation) and is trivially easy to setup suffix fallback.
My favorite data is raw file on FTP servers, i.e. bulk data. For me, it is the easiest to work with. I can translate it to XML or JSON if I want to, but many times I do not even need to go through that intermediary step to get what I want. If others wanted the data in, say, JSON, I'd be happy to share my parsers with them. I'd bet others would too.
I think Tom's point was "Cut the BS, and just give us the damn data, already." (Correct me if I'm wrong, Tom.) And I think he is spot on. This API nonsense has gotten out of hand.
Might as well go back to serving XML and letting the browser transform it (http://www.w3schools.com/xsl/xsl_client.asp), that would be more elegant for such a task (even though it never caught on due to limited browser support).
The content negotiation method would seem best except that most content-driven web pages will contain much content that isn't interesting to the scraper on every page, so a well-designed API with very specific queries will be faster and more useful. Whatever content is on each page is also a moving target for the scraper, even if she gets it as JSON, not so with a straight API.
You'd have to do hacky browser xslt detection as detailed here: http://www.informit.com/articles/printerfriendly.aspx?p=6779...
XML in this context apparently pretty much died because of too many badly written parsers, libraries and browsers - and to some extent probably also because dominant search engines would not index XML files properly (see http://news.oreilly.com/2008/06/why-xslt-support-in-the-brow...).
But nowdays at least there are some reasonably portable solutions like http://archive.plugins.jquery.com/project/Transform (but that's unnecessarily slow).
On the server side, it should be fairly simple, unless you hit bugs/deficiencies (like wasting memory / leaks) in libraries ...