In this article, we’ll be spelling out the key criteria to take into consideration to choose the right Conversational AI technology partner to fit your unique needs. The judging criteria are divided into 3 main categories:

• Product-related criteria
• Process criteria
• Business considerations

By the end of this, you will hopefully have a better understanding of how to choose the right conversational AI platform to achieve your business objectives.

Conversational AI Platform Evaluation: Where to Start

Before delving into the three categories of criteria to be examined in this evaluation process, it is crucial to understand that the choice of a conversational AI (CAI) platform depends largely on the specific goals and objectives of a business.

No technology vendor is the perfect universal solution for all businesses. If that would be the case, everyone would rely on that one and we wouldn’t need to write a whole article about the criteria. We would just tell you who’s the absolute best, and that would be it. The reality is that different vendors provide different advantages, catering to different organizational needs.

That’s why the first step in CAI platform evaluation does not involve technology vendors and solutions at all. 

Before evaluating different technology vendors, it's important to reflect introspectively on your organization. Focus on spelling out your goals and why you think that utilizing CAI technology might support you in reaching them.

Try to be specific about your objectives and clearly define:

1. What metrics you want to achieve (e.g., 30% containment rate)
2. What are the business results you’re expecting (e.g., 1000 more leads per month)
3. Which channels you want to automate first (e.g., website chat, phone contacts)

Once your goals for automation are clear to you, the next step is to figure out what are the characteristics and capabilities a platform needs to have to support you in reaching these results.

That’s where the criteria we present below come into play. 

Depending on these goals, as well as your team, technological stack, and other factors, each of the aspects might be more or less relevant to consider. The best way to go about the evaluation process is to define the most important ones upfront and let those guide your evaluation of your potential technology partners. 

How to Choose the Conversational AI Platform That’s Right for You

As anticipated, we have defined a list of criteria to take into account when evaluating and deciding between different conversational AI solution providers. This list has been compiled combining our own experience in guiding organizations in this process with Gartner’s, Rinf.tech’s, and Upsilon’s.

To make the list more accessible, we divided the criteria into 3 broader categories:

• Product-related factors reflect on what the platform itself offers and enables you to do
• Process criteria reflect on the onboarding and implementation processes
• Business considerations are related to legal, administrative, and economical aspects that fall outside the realm of Conversational AI, but that, as a business, organizations need to account for.

Let’s start from Product Criteria.

Product Criteria for Conversational AI Platform Evaluation

In this context, we think of the product as the platform you’ll be working on. By reflecting on the criteria listed in this section, you should be able to figure out:

What does the platform offer, and enable you to do?

These are the criteria to examine to comprehensively answer this question:

• SPEED & ROBUSTNESS. There are two perspectives to consider that regard the speed and robustness of a conversational AI platform. One is to ensure the platform is able to promptly deliver answers to users, verifying that the responses are delivered within a time frame that doesn't feel slow to them. It's important to note that faster isn't always better, but it's worth considering that we - humans - can react to conversations in as little as 200 milliseconds. On the other hand, designers, developers, and anyone within your organization can trust the platform to support their operations, i.e. that it doesn’t lag, saves changes automatically, doesn’t require refreshing too often, and so on.

• CHANNEL AVAILABILITY. Does it support the deployments of conversational assistants for phone, browsers, email, and different messaging apps (e.g., WhatsApp, Facebook Messenger)? When it comes to specific messaging apps, does it allow you to take advantage of all the capabilities of that channel (e.g., lists on Whatsapp, carousels on Facebook Messenger)? 

• NLP MODEL & LLM. Does the platform offer a first-party NLP model that you can rely on to quickly get a proof of concept up and running? How many languages does it support and how easy is it to activate/deactivate multi-lingual support? How many voice options does it offer out of the box? Will you be able to easily integrate a third-party NLP solution? How does the platform use LLMs (e.g., intent & entity recognition, response generation)?

• DESIGNING & TESTING. For evaluating designing and testing capabilities, you’ll want to investigate different aspects:

1. Low-code/no-code accessibility
2. Intent & entity definition
3. Version management
4. Test tools
5. Testing metrics & debugging
6. Personalization level, in terms of being able to use your own content, text formatting, graphic design tools, API management

• ANALYTICS & REPORTING. Does it offer clear dashboards that can contribute to bringing you relevant insights? What metrics does it measure out of the box? How easy is it to customize dashboards and visualize your custom metrics?

• PREBUILT COMPONENTS LIBRARY. Does the platform allow you to save components of your designs to reuse them across your projects? Does it offer common and/or domain-specific templates that you can use as a starting point to quickly spin off prototypes of upcoming features?

• SCALABILITY. How well does the platform cope with high volumes of users? Are you able to deliver the same quality of service at peak times?

• OPEN SOURCE/CLOSED SOURCE. Does the vendor provide open access to the source code of their software, or is it controlled by them? How does this impact the platform’s flexibility and customization options, as well as ease of use and technical complexity?  

Process Criteria for Conversational AI Platform Evaluation

The second set of criteria to evaluate conversational AI platforms examines the way the solution and its providers are able to support and ease design and implementation. The goal is to help you answer the question of:

How much time and effort are required to get started? And how much for upkeep?

Let’s look at the different aspects to consider:

• USER-FRIENDLINESS (or Designer-friendliness). As Robb Wilson argues in Age of Invisible Machines, “If you can’t get your team using a solution, then it’s not a solution”. So, it’s crucial to find out how user-friendly, or in this case conversation designer-friendly, the platform is. Aspects that can improve the designer experience (or DesignerX) of a CAI platform include:

1. Intuitiveness of the interface and commands
2. Onboarding aids and educational content
3. Clearly written and well-structured documentation
4. Tools and capabilities that improve the communication with developers, PMs, clients, and other stakeholders

• IMPLEMENTATION & MAINTENANCE EFFORT. How do your team’s current competencies and way of working fit with the technology? If needed, does the vendor offer staff training? What about consulting and development services? Once the initial setup is completed, will your internal team be able to manage the platform autonomously? Or will you always be partially dependent on the vendor’s services?

• OUT-OF-THE-BOX INTEGRATIONS. How easy is it to integrate your current CRM system, knowledge sources, and other business apps with the platform?

• CUSTOMER SUPPORT. As a user of the platform, you want to make sure that you’ll be able to be assisted when issues arise. Verify the opening hours and languages supported by the vendor’s customer care and, if you can, try putting them to a test and see how quickly and efficiently they respond to an inquiry.

Business Criteria for Conversational AI Platform Evaluation

Finally, there are also some legal, administrative, and general business aspects that fall outside the realm of Conversational AI, but that you need to account for in the choice of a technology partner. This well help you finalize:

How well are your general business requirements met by the vendor?

For this, you’ll reflect on:

• MANAGEMENT & ADMINISTRATION. Can you easily set up and change platform accesses, according to a role-based management system? Will you be able to oversee user accounts from an admin profile?

• PRIVACY. How does the platform store and deal with PII content? Has GDPR compliance been accounted for? Does it ensure end-to-end encryption?

• COMMERCIAL TERMS. When agreeing on a partnership with the vendor, especially if your organization will also rely on their consultancy and implementation services, it’s necessary to negotiate contract terms with particular attention to NDAs, IP rights, and NEA clauses. 

• PRICE. Some of the most common pricing models offered by conversational AI platforms include subscription models, pay per usage, and pay per performance.

• ON PREMISE/CLOUD INSTALLATION. Is the provider able to meet your needs in terms of on premise/cloud installation? How easy would it be to migrate, if necessary?

• VENDOR STABILITY and REPUTATION. Last but not least, it can definitely be reassuring to know that the technology is backed by an established and profitable company. Take a look and ask for references, recommendations, feedback, testimonials. Try to figure out what the vendor’s specific business verticals are, but also how aligned you are in terms of company culture and values.

Bonus Track: Questions to Vet your Technology Partner

The criteria we’ve discussed will allow you to perform a very well-rounded evaluation of conversational AI vendors you’re considering to partner with. To make sure you’ve got the full picture on the candidates, there are a few extra vetting questions you can ask.

Comparing so many solutions can surely create a lot of confusion. One way to bring clarity into the picture is to see how the same proof of concept performs on different platforms. As Moveworks recommends, ask vendors to get interactive demos using your data and see for yourself how well your vision is allowed to come to life on different workspaces.

On the other hand, as Robb Wilson points out in his book Age of Invisible Machines, you also want to make sure that “the cobbler’s children have new shoes”. 

Meaning: you want to choose a conversational AI vendor that believes in the technology they’re selling and that also internally makes use of it with positive outcomes. When the cobbler, i.e. the vendor in this metaphor, is using old shoes that belong to someone else, be skeptical.

The questions that Wilson recommends to ask to investigate this subject are:

• Have you successfully automated internal processes?
• Are your internal machines integral to your own operations?
• Are employees within your organization clamoring for more machines because they see automation working?
• What skills and use cases have you tackled? Can you show us?

In sum

As we saw from the beginning, there is no "one-size-fits-all" solution when it comes to choosing the right conversational AI platform for your business. The best technology varies based on the specific needs and characteristics of each individual company and, most importantly, on the goals you’re hoping to achieve.

In this article, we’ve shared the criteria that we trust to get the complete picture on what are the pros and cons of every option on the market. Along with these, we’ve also added some bonus questions that we recommend for vetting your final candidates and making a fully informed decision.

We understand that the process of evaluating different technology vendors can be overwhelming and complex. That's why we also offer strategic guidance and support to help businesses choose the best technology for their unique requirements.

‍Looking to figure out how conversational AI can help your business achieve your desired outcomes and drive growth? Feel free to reach out via voice message or good old email.

‍

And if you’re still hungry for knowledge, follow us on LinkedIn for weekly updates on the world of conversational AI, or check out our article on Customer Support automation, where we delve into different automation opportunities and discuss the cases of Decathlon, domestic violence lines, the Big Table Group, and Vodafone.