Objective
Investigate whether a retrieval pipeline can exclude other customers’ records while retaining the correct document for the requesting customer.
The experiment compares three approaches: unfiltered retrieval, filtering after selecting results, and filtering before selecting results.
Lab setup
- Dataset: 36 synthetic documents belonging to 12 customers.
- Document types: Loan balance statements, instalment records and closure quotes.
- Queries: Three question texts, each evaluated under all 12 customer identities.
- Retrieval: Word-based TF-IDF and cosine similarity.
- Candidate limits: 1, 3, 5 and 12.
- Evaluation: 432 configurations across the three retrieval designs.
No LLM or embedding-model calls were made. No real customer data or live banking systems were used.
What was measured
For each prepared context, the experiment checked:
- Whether it contained another customer’s records.
- Whether the correct target document was present.
- Whether the context was empty.
These measurements concern documents selected for an AI assistant, rather than answers generated by a model.
Results at a candidate limit of three
Unfiltered retrieval
All 36 prepared contexts contained unauthorized records. The correct target document appeared in 9 cases.
Filtering after selection
Unauthorized records were excluded from every context. However, 27 contexts became empty, and the correct target document remained in only 9 cases.
Filtering before selection
Unauthorized records were excluded from every context. The correct target document appeared in all 36 cases, with no empty contexts.
Why the difference occurred
The synthetic documents used repeated wording, so different customers’ records could receive equal relevance scores. A fixed document-ID tie-break determined their order.
When the system selected three results globally, other customers’ records could occupy all available slots. Removing those records afterward left an empty context, even though the correct document existed elsewhere in the collection.
Restricting the candidate collection by customer before selection kept authorized records available for ranking.
Additional observation
Increasing the candidate limit improved target availability for filtering after selection:
- 1 candidate: Target present in 3 of 36 cases.
- 3 candidates: Target present in 9 of 36 cases.
- 5 candidates: Target present in 15 of 36 cases.
- 12 candidates: Target present in all 36 cases.
The value of 12 worked for this constructed dataset; it is not a general production recommendation.
Reproduce the experiment
View the experiment code and recorded results on GitHub
Download retrieval_experiment.py and run it with Python 3:
python retrieval_experiment.py
The script uses only the Python standard library and creates retrieval_results.json containing the dataset, observations and summaries. Rerunning the script replaces that output file.
Limitations
This was a small, deterministic experiment using repeated document templates and a simple customer-ownership rule. It did not evaluate authentication, complex permissions, neural retrieval, generated answers or actual customer-facing disclosures.
The results demonstrate behavior in this specific setup and should not be interpreted as production failure rates.
Takeaway
Both filtering approaches excluded unauthorized records from the prepared context. Filtering before selection also preserved the correct authorized evidence at every candidate limit tested.
An AI security review should examine both who can access the retrieved documents and whether useful authorized evidence survives the selection process.
Read the full article
For the complete explanation of the experiment, findings and implications for banking and NBFC teams, read:
Before the AI Answers: Testing Customer Data Isolation in a Banking RAG Pipeline