Chatbot Response Evaluation
Tensor and CP decomposition applied to a university chatbot's responses — turning “is it any good?” into interpretable factors over inquiry categories.
Question
A university chatbot answers thousands of heterogeneous student questions — financial aid, registration, housing, advising. Aggregate satisfaction tells you almost nothing about it. The question this work asks is structural: which inquiry categories degrade, along which quality dimensions, and does that structure hold over time or drift?
Why decomposition
Interaction logs are naturally multi-way rather than tabular: an interaction has an inquiry category, a set of response-quality dimensions, and a position in time. Flattening that into a matrix throws away exactly the interactions you want to see. Low-rank decomposition of the tensor gives back a small number of factors, each of which is readable as a pattern across all three modes at once — a category profile, a quality profile, and a temporal profile that belong together.
Method
CP decomposition factors the interaction tensor into R rank-one components. Unlike a black-box score, each component can be read back out and argued with — which is the point when the audience is the people who run the chatbot.
Pipeline
- Interaction corpus structured into a tensor over inquiry categories, response-quality dimensions, and time
- Classification of AI-generated responses so quality dimensions are assigned consistently before decomposition
- CP decomposition to extract latent factors behind response quality
- Category-level evaluation of contextual accuracy in retrieval and question answering
- Trend analysis over the temporal mode to locate where information-retrieval quality degrades
Companion work
A parallel line at the William and Barbara Leonard Transportation Center built reward-modelling pipelines and evaluation frameworks measuring confidence calibration and faithfulness of LLM outputs, with automated experiments that varied prompts and conversational context (October 2025 – August 2026). The two share a premise: the evaluation apparatus deserves as much modelling care as the model being evaluated.
Status
The research period has ended. There are no published numbers to report, and this page will not invent any.
- Complete
- Qualitative
- None yet
Limitations
- Category labels are themselves a modelling choice; a different taxonomy would produce different factors, and the sensitivity of results to that choice is not yet characterised
- CP rank selection is not identified by the data alone — interpretability and fit pull in opposite directions
- Interaction volume is uneven across categories, so temporal trends in thin categories are weaker evidence than they look
- Contextual accuracy is graded against a working definition, not an external gold standard
Lesson
Evaluation is a modelling problem in its own right. A decomposition that returns interpretable factors is more useful to the people running the system than a single score that is more accurate and cannot be argued with.