I Built a Search Agent. Then I Spent Months Teaching It When Not to Answer
This article is the write-up of my Talk at the Paris Machine Learning meetup The hardest feature in production RAG is not retrieval. It is judgment. A funny thing happens when you build your first search agent. At first, it feels magical. You take a messy pile of knowledge: PDFs, internal templates, business rules, operational instructions, store information, support documents. You chunk everything. You embed it. You retrieve the closest passages. You send them to an LLM. The model writes an answer. In a demo, everyone smiles. Then real users arrive. They ask half-questions. They use old names for new processes. They ask about rules that changed last week. They mix several requests in one sentence. They paste personal information. They ask things that look answerable, but are not actually supported by the documents. And the system does exactly what you built it to do. It answers. That was the uncomfortable lesson behind my Paris Machine Learning Meetup talk: A good search agent i...