What I learned building a retail performance chatbot (beyond the demo)

When I began building this retail-performance chatbot, the appeal was obvious, let users ask natural-language questions about sales, margin, and store performance without involving an analyst, building a report, or writing any SQL. It felt like a straightforward and exciting use case for Snowflake Intelligence, Cortex Analyst, Cortex Agent and semantic views. And to be fair, the demo did exactly what it needed to do, it showed what was possible. However, once I moved beyond the demo and into the reality of how people would actually use it, I realised the hard part was not the chat interface at all. It was everything underneath it.

It quickly became clear to me that the quality of the experience was far more dependent on the quality of the foundations than on the sophistication of the chatbot itself. The system could only be as useful as the metrics and definitions behind it. If sales, margin, or store performance were not clearly defined and consistently governed, the chatbot had nothing reliable to stand on. In that sense, conversational analytics did not reduce the need for strong data modelling, it made any weakness much more visible. That was one of the most important shifts in how I thought about the work. The visible product was the chatbot, but the real product was trustworthy business meaning.

This led me to realise that semantic modelling mattered much more than prompting. At the beginning it was tempting to think that the main challenge would be how to phase instructions well to help shape the chatbot’s responses. However, in practice, that was only a small part of the problem. The harder task was translating the way business users spoke into governed metrics, dimensions and relationships that matched how they actually understood the business. Prompting could make an answer sound better, but semantics determined whether the answer meant the right thing in the first place. I came away from the pilot feeling that if the semantic layer is weak, no amount of clever prompting can compensate for it.

Once real users began interacting with it, I saw that people do not ask neat system-friendly questions, no matter what training they are given. In a demo, it is easy to show clean examples that work, however, in practice, users asked things such as “Show me the top underperforming stores”, which sounds simple until you stop and ask what “underperforming” actually means. Is it based on sales, margin, growth, forecast variance, or something else? Over what period? Compared to what baseline? I found that non-technical users often expected the system to infer much more context than it realistically could. The answers it gave were not technically wrong, but they just were not what the user was wanting to know. This was one of the clearest moments where the gap between demo success and production usefulness really showed up.

That ambiguity flowed directly into one of the biggest sources of friction: metric definitions. Some of the most difficult issues were not model failures in the technical sense, but definition failures in the business sense. Even familiar terms could carry different expectations depending on the user or the report they had in mind. I was reminded again and again that in analytics, confusion often comes less from how a number is calculated and more from what that number is understood to represent. Without shared definitions, even a correct answer can feel wrong. For me, this was a strong reminder that governed metrics are not just a technical nice-to-have; they are central to whether users trust what they are seeing.

Because of that, validation became essential. I found that trust did not come from fluent answers or a good interface. It came from whether the output matched the daily Excel report people already knew and relied on. That report quickly became the benchmark for credibility. When the chatbot produced different numbers, the issue was not just a quality problem to investigate; it was a trust problem to solve. Usually, the discrepancies pointed to something deeper in the semantic mapping, metric logic or business interpretation. This showed me that users trust a new interface when they can reconcile it with the numbers they already believe.

This also changed how I thought about the explanatory side of the chatbot. It was capable of producing narrative responses that sounded polished and confident, but I became much more aware that a plausible explanation is not the same as a grounded one. Fluent language can create a false sense of confidence if the underlying business context is thin or if the semantics are not strong enough. That was an important discipline for me during the pilot: not to confuse a well-worded answer with a trustworthy one. In an analytics setting, sounding smart is never enough. The explanation still needs to be anchored in business reality.

At the same time, when working with users it became clear, they did not really care about the architecture. They cared about whether they could get a fast, clear and useful answer. For non-technical users, the value was never in how sophisticated the AI stack was. It was in whether the interaction reduced friction in their day-to-day workflow. That was a helpful reset for me, because it kept the project grounded in user value rather than technical novelty. A solution can be elegant under the hood, but if it does not make someone’s work easier, the elegance does not matter very much.

That perspective also helped explain why adoption was slower than the demo had suggested. Even when the pilot worked well, uptake depended on much more than technical capability. People had existing habits, trusted reports and established ways of working. They also needed to learn how to ask better questions, and that kind of behaviour change does not happen overnight. Looking back, I do not see the slower adoption as a failure of the technology. If anything, it was a reminder that workflow change, trust-building and enablement are part of the product, not something that happens after the product is finished.

If I were doing this again, I would make a few changes much earlier. I would invest sooner in governed metric definitions and spend more time on semantic modelling before polishing the chat experience. I would test with real user questions earlier, rather than waiting until the solution felt more complete, and I would design more deliberately for ambiguity, follow-up clarification, drill-downs and comparisons. I would also validate against trusted reports from the beginning, rather than treating that as a later check. Most importantly, I would treat adoption and enablement as part of the build itself. That is probably the biggest lesson I took from the project: the demo proves possibility, but production exposes meaning, trust and workflow. Retail performance chatbots do not remove the need for strong data foundations. If anything, they raise the standard. The chatbot may be the visible interface, but the real product is trust in the numbers behind it.

Want conversational analytics that work in the real world? Altis can help you set up trusted data, clear definitions and strong governance. Contact us to enquire.