
Ask HN: Experience from implementing and using Ipsoft Amelia chatbot? - throwaway-aadeq
I was recently asked by a senior executive to look into the business case and feasibility of the ongoing implementation of a chatbot. The company are in the early phases of implementing IPsofts Amelia chatbot for customer interactions (activation, support, and up-sell) in an online retail bank setting.<p>The project immediately triggered a number of red flags from my side:
- Significant manual work (&gt;1 man year) to the document the full possible conversation tree in a type of flow chart as well as full time employee only writing every possible question that can be asked in as many variants as possible, e.g. &quot;How do I withdraw deposits?&quot; formulated in +100 ways
- No quantified goals or examples from other customers on the timeline for the bot to reach the target accuracy of handling +90% of incoming questions independently
- Senior executives not understanding the technology in-depth but talking about the software as a real person that will in time &quot;handle all customer interactions&quot;, while product owner jumping ship as quickly as possible when they understand the maturity of the technology
- Bank in the same market struggling for multiple years to implement the product without succeeding due to poor accuracy and very limited domain in where the bot can work independently<p>I would love to be proven wrong, it would be great if the technology is mature enough to handle all these fairly complex customer interactions independently. Using throwaway account not risk disclosing any client information.<p>Anyone here has experience from working with IPsofts Amelia product (or something similar) and have any insights to share on how the project went (resources needed, results, etc)?
======
throwaway149
I have worked with Amelia. Few quick comments:

\- understanding intent is tricky in general (not just Amelia). Typical
approach is to ask for annotated data from chat logs. Most companies don’t
have this, so in our case we had a bunch of people (~10) enter a form as if
they were asking the question, and used that as a baseline to train the intent
recognition classifier.

\- distinguished 3 possibilities in chat outcomes:

    
    
      1. Intent understood and handled
      2. Intent understood and redirected to the correct agent
      3. Intent not understood
    

In our case we had +90% for the first 2 combined after about 1-2 months, with
handled rate (1) increasing as processes were ported. (Figures will depend on
how varied your processes are)

Time involvement varies. In hindsight it’s probably best to sit someone
currently handling those chats down with the team and work rapidly through the
implementation. Depending on your back end systems (and processes/bureaucracy)
the integration with the back end can take up more time than expected

------
olavgg
Creating training data for a chatbot that has thousands of intents in a single
domain like banking, takes MANY months even if you have a team creating the
training data.

We have worked around this with creating pre-trained intent modules with
complete training data for all the most common inquires. I would be surprised
if IPSoft did not sell pre-trained data that you can use immediately.

