Ongoing Answer Threads Management

Monitor and maintain Answer Threads performance #

It’s helpful to think of training your generative AI assistant as you would train an Agent. Similar to ramping a new Agent, we suggest reviewing Answer Threads daily for the first 1-2 weeks after implementation. Like an Agent, once Answer Threads is fully launched and situated within your setup, continue to monitor the AI assistant on a weekly basis in order to identify areas for improvement.

Metrics #

You’ll want to make sure you are monitoring your Answer Threads metrics regularly by using the Sidekick Dashboard.

1. Open the Sidekick Dashboard.
2. Take a look at resolutions and engagement by Thread, in particular, “Chat – Answer Thread” and “SMS – Answer Thread.”
   • Do you want Answer Threads to engage in more? If so, think about:
     • What other parts of your Chat experience could benefit from Answer Threads?
     • Are you transferring too many interactions?
     • Are there top topics (see the Topics dashboard) for which you still need to write Public Answers?

Maintenance #

The metrics above provide you with a helpful gut check as to how Answer Threads is performing, while evaluating individual Conversations within the Conversations page offers a more in-depth look at specific Answer Threads engagements. For routine maintenance, we suggest analyzing the Conversations page.

1. Log into Gladly and then click on Sidekick Beta.
2. Go to Conversations.
3. Here, you will see a list of Conversations that Answer Threads have had with your Customers.
4. Review a handful of Conversations where Answer Threads has engaged. Use Manage Answer Threads Contacts Page to evaluate responses.

Daily Answer Threads Management #

Review key metrics and Answer Threads engagements on a daily basis following initial implementation. These steps help ensure your assistant is delivering expected results and aligning with your company’s standards in the first 1-2 weeks post-launch:

1. Review the Sidekick Dashboard by going to Reports > Sidekick Dashboard.
2. In addition to a general review of the dashboard, pay attention to the following two metrics:
   • Number of Assists in comparison to SMS and/or Chat volume
   • Number of Resolutions

Once you have a sense of how Answer Threads is performing, use the Conversations page for a deeper dive into individual engagements.

1. Identify a handful of Conversations with a Sidekick resolved or a Sidekick handed off status.
2. Consider how the Quality Check has been evaluating generated replies.
   • Has the Knowledge Check been failing?
     • Create or update Public Answers to make response more accurate and/or available.
     • Consider adding Draft Answers.
   • Has the Hallucination Check been failing?
     • Review tips for avoiding Hallucinations.

Manage Answer Threads Conversations Page #

Before you start #

• Get to know Answer Threads Concepts
• If you’re brand new to Answer Threads, review New to Answer Threads? Start Here.

Search Answer Threads interactions #

As a Gladly Team Manager or Administrator, you can review every customer interaction Answer Threads engaged in through the Conversations page.

This is a great place to understand how Answer Threads interacts with Customers, how it responds, and why it hands off certain interactions to agents. This is information you need to understand how Answer Threads is performing and where you could make improvements.

On this page, you can search for interactions by entering a customer’s phone number, email address, or name. Icons indicate if Answer Threads interacted with the customer through SMS or Chat.

Quality Check review #

If any of the quality checks — Knowledge, Hallucination, or Unsupported Actions — fail, Answer Threads will automatically hand off the Customer to an Agent.

Avoid Hallucinations #

Hallucinations occur when generative AI attempts to generate a response but Answer Threads believes that the response is not accurate because it’s unable to find a suitable Public Answer.

Hallucinations can be avoided by:

• Ensuring you have Public Answers for as many answers to questions that Customers may ask.
• Advice is fine-tuned to align with expected brand tone and overall behaviors you’d like the generative AI assistant to adopt. Review tips for managing Advice to ensure it’s configured for success.
• The Transfer Reasons list includes as many scenarios where you want Answer Threads to handoff to an Agent instead of attempting to generate a possible inaccurate reply.