Root Cause Analysis
What's RCA
Root Cause Analysis (RCA) is a technique to identify the reason behind LLM solution response aka why the chatbot generates certain outcomes and how it relates to user feedback (both explicit and implicit). Feedback Intelligence provides a Deep Insights Layer that automatically troubleshoots every response of LLM and gives comprehensive overview on what went well, what didn't, what's the root cause, and most importantly how it can be resolved.
Signals
When analyzing user experience through both implicit and explicit feedback, as well as user queries, Feedback Intelligence identifies key signals, which can be either positive or negative. Negative signals indicate dissatisfaction—either through explicit complaints where users directly state their needs aren't being met, or through insights derived from analyzing user queries. The same approach is applied to detect positive signals, identifying instances where user intentions are fulfilled, and satisfaction is high. This dual signal system allows for a deeper understanding of user sentiment and behavior.
Issues
Negative signals indicate that users are not getting their the needed outcomes, highlighting underlying problems referred to as "issues," which are then classified for further analysis. For example, if a financial analyst queries Nvidia’s Q3 2023 revenue for the period ending in October and receives an inaccurate or insufficient response, this generates a negative signal classified under the issue of "inaccurate response." Each group of issues is then assessed for its impact, with scores calculated to prioritize resolution efforts (as explained in the next chapter). The root causes of issues are deeply analyzed as well. For instance, a user may receive an inaccurate response due to a knowledge hole in the context, or maybe the chunk size is incorrect (more details to follow).
Topics
Topics reveal the categories and subjects users are querying when interacting with the LLM solution. This insight helps AI teams pinpoint users' key areas of interest, enabling them to refine their RAG strategies. By adding more relevant test cases and enhancing context based on actual usage patterns, teams can ensure the system aligns more closely with user needs.