RAG Versioning

RAG Versioning

RAG versioning records exactly which documents were embedded, with which model, at which source commit — so a retrieval can be tied back to the exact corpus state, and a stale index is caught before it answers anything.

rag For: governance & audit

When you’d reach for this

Your support-triage agent answers policy questions from a retrieval index built over a support_kb. The team revises a document and adds a new one, but the index isn’t rebuilt — so the agent keeps citing the old version and routes a ticket on outdated guidance. A versioned manifest makes the drift detectable: check_invalidation tells you precisely what changed before you trust a retrieval.

How it works

1. Build a manifest. create_embedding_batch then create_manifest fingerprint the indexed documents into an EmbeddingManifest.

2. Check invalidation. check_invalidation compares the current documents and model against the manifest to detect a stale index.

3. Rebuild. When sources changed, rebuild_index produces a fresh manifest so retrievals reflect the current corpus.

flowchart LR
    A["Documents"] --> B["create_embedding_batch()"]
    B --> C["create_manifest()"]
    C --> D["EmbeddingManifest"]
    E["Current documents + model"] --> F["check_invalidation()"]
    D --> F
    F --> G{"is_valid?"}
    G -- "no" --> H["rebuild_index()"]
    G -- "yes" --> I["Reuse index"]

Diagram description

Documents flow through create_embedding_batch() and create_manifest() to produce an EmbeddingManifest. check_invalidation() compares that manifest against the current documents and model: if it is no longer valid the index is rebuilt with rebuild_index(), otherwise the existing index is reused.

Install

pip install briefcase-ai[rag]

Note

The rag extra is pure Python and installs no third-party dependencies.

from briefcase.rag import VersionedEmbeddingPipeline, Document

Build an Index

A Document carries an id, content, and optional metadata. The pipeline’s embedding_model is any object with an embed(texts) -> list[list[float]] method; optional name and version attributes are recorded on the manifest.

from briefcase.rag import VersionedEmbeddingPipeline, Document


class HashingEmbedder:
    """Trivial deterministic embedder for the example."""

    name = "demo-embedder"
    version = "1.0"

    def embed(self, texts):
        return [[float(len(t) % 7), float(len(t) % 11)] for t in texts]


pipeline = VersionedEmbeddingPipeline(embedding_model=HashingEmbedder())

documents = [
    Document(id="kb-1", content="How to reset your password.", path="handbook/auth.md"),
    Document(id="kb-2", content="Refund policy for digital goods.", path="handbook/refunds.md"),
]

batch = pipeline.create_embedding_batch(documents)
manifest = pipeline.create_manifest("support_kb", [batch])

print(manifest.index_name)        # "support_kb"
print(manifest.document_count)    # 2
print(manifest.model)             # "demo-embedder"
print(manifest.status)            # "current"

rebuild_index() chains both steps when you just want a fresh manifest:

manifest = pipeline.rebuild_index("support_kb", documents)

Detect Staleness

check_invalidation() compares the latest manifest against the current document set and model, and returns an InvalidationReport describing what changed.

# A document changed and one was added since the manifest was built.
current = [
    Document(id="kb-1", content="How to reset your password (updated)."),
    Document(id="kb-2", content="Refund policy for digital goods."),
    Document(id="kb-3", content="Two-factor authentication setup."),
]

report = pipeline.check_invalidation("support_kb", current)

print(report.is_valid)            # False
print(report.status)              # "stale_documents"
print(report.added_documents)     # ["kb-3"]
print(report.changed_documents)   # ["kb-1"]
print(report.removed_documents)   # []
print(report.model_changed)       # False

status is one of: current, stale_documents, stale_model, stale_both, rebuilding. When the index is stale, rebuild it:

if not report.is_valid:
    manifest = pipeline.rebuild_index("support_kb", current)

EmbeddingManifest

The manifest is the versioning artifact. Persist it to compare future builds against it.

Field	Description	Why it matters
manifest_id	Unique id for this build	Names the corpus version a retrieval ran against.
index_name	Name of the index	Groups builds of the same index over time.
model / model_version	Model that produced the embeddings	A model change invalidates the index too.
source_commit	Source commit the documents came from	Ties the corpus to a versioned source.
document_count	Number of documents embedded	Quick sanity check on coverage.
document_hashes	doc_id -> content_hash at embed time	What check_invalidation compares against.
status	Current ManifestStatus value	current vs stale_*.
manifest_hash	Deterministic SHA-256 over the manifest content	Tamper-evident fingerprint of the whole build.

print(manifest.manifest_hash)     # integrity hash
serialized = manifest.to_json()

Instrument Retrieval

InstrumentedRetriever captures version provenance on each retrieved document. It is a reference implementation: the base retrieve() returns placeholder results (and emits a RuntimeWarning) so the provenance shape is clear. Override retrieve() with a real vector-store query — returning your own RetrievalResult objects — to silence the warning and use it for real.

from briefcase.rag import InstrumentedRetriever


class KbRetriever(InstrumentedRetriever):
    def retrieve(self, query, top_k=5, similarity_threshold=0.7):
        # Query your real vector store here, then wrap hits in RetrievalResult.
        return super().retrieve(query, top_k, similarity_threshold)


retriever = KbRetriever(
    vector_store=None,        # your vector store client
    lakefs_client=None,       # resolves document_version (commit SHA)
    repository="support_kb",
)

results = retriever.retrieve("how do I reset my password?", top_k=3)
for r in results:
    print(r.rank, r.document_id, r.score, r.document_version)

Each RetrievalResult carries document_id, content, score, rank, document_version (the commit SHA the document was read at), and metadata.

Where this fits

Related: External Data Tracking The same versioning idea for non-RAG upstream values.

Related: Validation Engine Validate the content a versioned retrieval returns before a decision.

Guide: Reproducible RAG Wire manifests and snapshots into a reproducible pipeline.