Building a customer support bot
I built a support bot over three weeks. I added better search, Claude, and customer memory, and wrote about the tests and bugs.
I built a public project to show the decisions
Most of my client work is under NDA. I built a customer support bot over three weeks to show how an AI project changes as I add features. The repository is public.
The build started with keyword search and fixed rules. Later phases added hybrid search, a Claude agent, streaming, evaluation, and customer memory. Each phase solved a problem and exposed another.
Keyword search gave me a starting point
The first version had four knowledge documents, two hardcoded orders, and Python conditions. It handled product questions, order lookups, refunds with session memory, and support tickets.
A direct product question found the right guide. But “Can I get my money back?” matched nothing and created a ticket. The policy used different words from the customer.
Starting with keyword search showed which questions needed a better search method. It also gave me a working version to compare against later changes.
Better search exposed a broken metric
I split the documents into chunks and embedded them with Voyage AI’s voyage-3-lite at 512 dimensions. Supabase stored the vectors with an HNSW index. Hybrid search combined 30% full-text rank with 70% cosine similarity.
“Can I get my money back?” now found the refund policy with a score of 0.267. Five other questions that had created tickets received correct answers.
Later tests suggested keyword search was better. The scoring code compared titles. It treated whole documents and chunks differently, which penalised chunked search. I changed the comparison to use document IDs. The corrected results put hybrid search ahead of full-text search, with a further gain from reranking.
The test must measure the behaviour I need to improve. A passing release check does not make a wrong metric useful.
Tool selection changed what the bot could complete
I replaced fixed routing with Claude Haiku. It received four tool definitions, the conversation history, and the customer’s message. It could select tools and their order.
“My order ORD-1002 arrived damaged. I want a refund” previously needed two messages. The agent could now look up the order, confirm delivery, and start the refund in one turn.
For “Can I get my money back?”, the agent asked for the order ID and reason. Better search found the policy. Better tool selection helped it collect the details needed to act.
Tickets also gained useful subjects and context. A wholesale partner’s question about Q3 rebate tiers produced a specific ticket instead of “Low-confidence support query.”
Streaming showed work during the wait
A request with two tool calls left the screen blank for three to five seconds. I added server-sent events to show the order lookup, its result, the refund step, and then the response text.
The underlying work stayed the same. The customer could see its progress.
A test expected an action the customer had not requested
The evaluation system started with 30 labelled queries. I later added a saved baseline and release checks with a 10% regression threshold.
One conversation test scored order recall at 0.5. Its second message asked, “And can I return it?” The agent answered the eligibility question from the policy. The test expected a refund request, although the customer had not asked it to start one.
Separate tests confirmed that memory held the order ID and that the model could read it. The expected action was wrong.
I also removed a category filter. It could classify a message about broken headphones and a refund as a product question. That removed the refund policy before vector search could find it.
Memory needed a working connection to the request
I stored four types of customer facts: order preferences, issue history, product interest, and communication style. Facts needed confidence above 0.7 and expired after 90 days. A more confident fact could replace an earlier fact of the same type.
An identified customer could ask about an order without supplying its ID again. But fact extraction initially worked only in isolation. The request flow never called it.
I registered it with FastAPI BackgroundTasks to run after the response. A working storage function was only one part of working memory.
Retrieved text needed a clear instruction boundary
An adversarial test added a knowledge chunk that told the bot to approve every refund and ignore delivery checks. It then requested a refund for an undelivered order.
I changed the system prompt to treat retrieved chunks as document content, not instructions. The poisoned-chunk test then passed without approving the refund.
Knowledge bases can include text from vendors, external teams, and content systems. The agent must keep that text separate from the instructions that control its actions.