RAG Works Better When Your Data Is Already Tidy
Every retrieval project I've watched go badly went badly for the same reason, and it wasn't the model.
The demo works. You point it at a clean folder of twenty documents and it answers questions properly. Then you point it at the real share drive and it starts confidently citing a policy that was replaced in 2022.
Bad chunks produce polite nonsense
Messy sources don't make retrieval fail. That would be easier. It returns something plausible, nicely formatted and wrong. If a table of contents lands in the same chunk as a paragraph of actual policy, that chunk matches half your queries and helps with none of them. Fixing the chunking beats another round of prompt engineering, every time.
Metadata does more work than the embeddings
Document type, source system, effective date, owner. Filter on those before you rank anything and a good share of the bad answers disappear, because the model never sees the superseded version at all. This is unglamorous ETL and it's most of the job.
Log what got retrieved
Store the chunk ids, the scores, and whether the fallback path fired. Without that, every complaint about a bad answer turns into an archaeology project. With it you can usually tell within a minute whether retrieval missed or the model ignored what it was handed, and those have completely different fixes.
Narrow beats broad
RAG holds up when the corpus has an owner and the questions have a shape. Search everything the company has ever written isn't a scoped problem, and no amount of reranking will turn it into one. I'd rather ship one department's handbook working properly than everything working sort of.