Ok, here is a question. I went to the site and tried the document conversion service. I gave it three very clean PDF files containing CS articles. In all cases, what I got in response was "The input document failed to be converted because An invalid XML character (Unicode: 0x0) was found in the element content of the document."
Tika is able to parse them without a glitch, and has been doing so for a few years at least.
So I tried with some sample HTML taken from the main italian news sites. In this cases, it parsed it, but returned some garbage HTML tags and no content. BoilerPipe or similar libraries are able to extract a clean body of the news in all cases.
Thanks for giving it a try. I agree that particular test does not sound like state-of-the-art. But don't judge the whole package just based on one test of one API. It'd be great to have access to these files (you can email them to me at my YC ID @gmail.com). Best would even be to post your experience on our forum: https://developer.ibm.com/answers/smartspace/watson/. We are usually pretty responsive.
Hey pathsjs, sorry for the bad experience... Nevertheless, thanks for the feedback. TLDR: I pushed a fix for the bad character issue, and those PDFs should convert now.
The long version: It has to do with the underlying structure of the PDF; some of the characters in the above PDF have glyphs for display but don't actually map the characters to code points. So, when we pull out the text, they come through as invalid characters, which we should have filtered out. This is an issue we've seen with (all?) PDF viewers; the text you copy from a sentence isn't always what you expect... But, we're aware of that shortcoming and are looking at some ways to improve the quality.
In regard to the extra content in the news articles, we're not currently trying to do what BoilerPlate does. If you want to include or exclude specific content from a page, we have config options to do that via XPaths. Though, we're always open to ways of improving our services, and incorporating something like that would probably be useful.
Tika is able to parse them without a glitch, and has been doing so for a few years at least.
So I tried with some sample HTML taken from the main italian news sites. In this cases, it parsed it, but returned some garbage HTML tags and no content. BoilerPipe or similar libraries are able to extract a clean body of the news in all cases.
How is that state of the art technology?