Filtered vector search returned four rows instead of ten, and nothing failed
A 10% filter cut pgvector's post-filtered answer to 3.9 of 10 rows and OpenSearch's filtered recall to 0.745 on 381,559 vectors. No error was raised.
12 min read
Topic
RAG
Retrieval over corpora that keep growing: chunking, metadata filters, index choice, and the TTL policy that stops a vector store becoming a landfill.
A retrieval index is not something you build once; it is a data pipeline with additional ways to be confidently wrong. Chunking should respect source structure, metadata filters often carry more precision than embeddings, and retention needs an owner before the index grows without bound. These notes focus on evaluation, provenance, deletion, re-indexing, and the operational half of retrieval that short demos usually skip.
3 posts
All writingAgents read your definitions. Somebody has to own them.
Agents fail on stale or contradictory knowledge before they fail on reasoning. One concept per file, an owner, a date, a Disputed section, and generate the rest.
13 min read
A RAG pipeline on a Delta lakehouse, and the bug that survived being fixed
Scrape to bronze, silver, gold on Spark 4 and Delta, embed into Chroma, and serve answers that cite their sources or decline to answer.
~1 h, most of it pulling images and the first Ivy resolution