Adding a chatbot that knows your private data
Scope first, RAG second: notes from building one at Kyu
At Kyu we build dashboards on employment and training in France. Our clients are mostly OPCOs, the bodies that fund vocational training, and the people who open those dashboards ask the same kind of question again and again. What does this indicator mean? Why did it drop? Which filter am I looking at?
We wanted a chatbot to answer those questions. The catch is that the useful answers live in private data, so a generic chat model is not enough. This post is the path we took, in the order we took it. The order matters more than the tools.
Start with the scope, not the model
The first decision was not about models or vector databases. It was about what the chatbot is allowed to talk about.
Two rules came out of that discussion:
- It only answers from data we have explicitly allowed.
- It only answers questions related to the page the user is currently on.
The second rule is the unusual one. It makes the bot less impressive in a demo and much safer in production. A user on the “apprenticeship contracts” page gets answers about apprenticeship contracts. If they ask about something on another page, the bot says so and points them there.
user question
│
▼
┌───────────────────────────┐
│ 1. Which page is the user │
│ on? (page id + filters)│
└─────────────┬─────────────┘
▼
┌───────────────────────────┐
│ 2. Is the question about │──── no ───▶ "That is on another page,
│ this page? │ try <link>."
└─────────────┬─────────────┘
│ yes
▼
┌───────────────────────────┐
│ 3. Retrieve only what is │
│ allowed for this page │
└─────────────┬─────────────┘
▼
┌───────────────────────────┐
│ 4. Generate, with sources │
└───────────────────────────┘Writing the scope down first also gave us the test set. Every in-scope and out-of-scope question we could think of became a test case before any code existed.
Choose the sources
Private data in a company is scattered, and most of it should stay out of the bot. We picked a small set and left everything else out on purpose.
| Source | What it gives the bot | What we restrict |
|---|---|---|
| Active Directory | Who the user is and which groups they belong to | Used for permissions, never indexed as content |
| Google Cloud Storage | Methodology notes, indicator definitions, documents | Only selected buckets, not all of them |
| BigQuery | The figures behind the dashboard | Only selected tables, not the whole warehouse |
The rule we kept: a source is excluded until someone decides it is included. Adding a source later is easy. Discovering that the bot has been quoting an HR spreadsheet is not.
Active Directory plays a different role from the other two. It does not feed the bot knowledge. It decides what each person is allowed to see, which matters because an OPCO should never get answers built from another OPCO’s data.
┌──────────────────┐
│ Active Directory │ who is the user, what groups
└────────┬─────────┘
│ identity + permissions
▼
┌──────────────────────────────────────┐
│ Chatbot API │
└───────┬───────────────────────┬──────┘
│ │
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ GCS (some │ │ BigQuery (some │
│ buckets) │ │ tables) │
│ definitions, │ │ indicators and │
│ methodology │ │ aggregates │
└─────────────────┘ └─────────────────┘Prepare the RAG
RAG means retrieval-augmented generation. Instead of hoping the model knows our indicators, we look up the relevant material at question time and hand it to the model along with the question.
Preparing it is most of the work. The two sources need different treatment.
Documents from GCS are chunked, embedded and stored in an index. Each chunk carries metadata, and the most important field is the page it belongs to. That field is what makes the “only this page” rule enforceable: retrieval filters on it before any similarity search happens.
Tables from BigQuery are not embedded. Numbers are bad material for similarity search. They are queried, using the same filters the user has on screen, and the result is passed to the model as context.
Offline (when content changes) Online (per question)
────────────────────────────── ─────────────────────
GCS documents question + page id
│ │
▼ ▼
chunk + tag with page id ┌── filter index by page id ──┐
│ │ │
▼ ▼ ▼
embed ──▶ vector index ───────▶ top chunks BigQuery query
│ (screen filters)
└───────────┬─────────────┘
▼
prompt = question
+ chunks
+ figures
│
▼
model
│
▼
answer + cited sourcesA few choices that paid off:
- Metadata before embeddings. Tagging every chunk with its page and its owner took longer than the embedding step and mattered more.
- Query the numbers, do not retrieve them. The bot never invents a figure because it never has to recall one.
- Cite sources. Every answer lists the document or table it came from. Users trust it more, and we can debug it.
Keep it on the page
Scope is enforced in layers, because no single layer is reliable alone.
| Layer | What it does |
|---|---|
| Request | The front end sends the page id and active filters with every question |
| Retrieval | Only chunks tagged with that page id can be returned |
| Prompt | The model is told to answer only from the supplied context and to say so when the context is empty |
| Output | If nothing relevant was retrieved, the bot declines instead of improvising |
The prompt is the weakest of the four, which is why it is the third layer and not the first. Instructions can be argued with. A filter that returns nothing cannot.
Privacy and GDPR
Some of this data relates to people, so GDPR applies. What we did about it:
- The bot answers from aggregated indicators, not from individual records.
- Identity comes from the existing directory, so permissions have one source of truth.
- Questions and answers are logged for quality, with a retention period, and the log is treated as personal data.
What I would tell someone starting
- Write the scope as a sentence before writing any code.
- Decide which sources are in. Everything else is out.
- Put permissions in the retrieval step, not in the prompt.
- Build the out-of-scope test cases first. They fail most often.
- Cite sources from day one.