Evaluating Natural Language Understanding Services [pdf]

[jjwiseman]

https://wwwmatthes.in.tum.de/file/6djp1mgfl78x/Sebis-Public-Website/-/Evaluating-Natural-Language-Understanding-Services-for-Conversational-Question-Answering-Systems/SIGDIAL22.pdf

[jjwiseman]

"From a high-level perspective, LUIS performed best with an F-score of 0.916, followed by RASA (0.821), Watson Conversation (0.752), and API.ai (0.687). LUIS also performed best on each individual dataset: chatbot, web apps, and ask ubuntu. Similarly, API.ai performed worst on every dataset, while the second place changes between RASA and Watson Conversation."

[valentinvieriu]

Just in time. I was just researching this today. Quick question ( maybe not related). Is there a good open source solution for Entity Extraction, and more exactly mapping it to Wikipedia, ( Like GOOGLE NATURAL LANGUAGE API ) ?

[jjwiseman → valentinvieriu]

Rasa, evaluated in the paper, can do entity extraction.

Other open source libraries that can do entity extraction: OpenNLP (https://opennlp.apache.org/docs/1.5.3/manual/opennlp.html#to...), Stanford NLP (https://nlp.stanford.edu/software/CRF-NER.shtml), MALLET (http://mallet.cs.umass.edu/),

[valentinvieriu → jjwiseman]

Maybe my question was misleading. I'm more interested, for my current project on mapping those entities with Wikipedia articles ( see https://goo.gl/mNdDRD ) or try the https://cloud.google.com/natural-language/ example