Attach metadata that supports retrieval filters

Vector similarity cannot decide whether a user may see a document, which version is effective or which language the workflow supports. Those constraints need consistent metadata and enforced filters.

Test a simple access filter

The RAG ingestion lab attaches metadata to four synthetic chunks.

from ingestion_cases import metadata_case

result = metadata_case()
assert result["missing_filter_values"] == 0
assert result["public_english"] == ["public-help-v1-1"]
assert result["filter_fields"] == ["access", "language", "effective", "owner"]

Only the public English help chunk passes the example filter. This local list operation demonstrates the contract; it is not an authorization test against a deployed vector database.

Normalize values at ingestion

Useful metadata can include tenant, access class, jurisdiction, language, source/version, effective interval, owner, document type, page and chunker/embedding versions. Use controlled values and ISO dates. Reject missing mandatory fields instead of filling an “unknown” value that broad filters accidentally include.

Inherit security metadata from the source and prevent content from changing it. A sentence that says “this document is public” cannot override the inventory’s access policy.

Enforce filters before similarity search

Derive the permitted scope from authenticated identity and application policy. Apply it in the query itself where supported; post-filtering top results can miss authorized evidence and may expose unauthorized content to the model or logs. Test both allow and deny cases, including changed roles and cross-tenant IDs.

Metadata also supports freshness and deletion. Record enough identity to remove all chunks from one source version. Index fields that are queried frequently and measure filter performance on intended scale.

The Generative & Agentic AI course connects metadata contracts to vector filters, MCP identity and grounded-answer citations.

Exercise

Define metadata for a two-tenant, two-language corpus with effective versions. Write authorization tests that fail closed on missing fields and prove unauthorized chunks never enter the model context or trace.

Continue learning

This article is part of the RAG ingestion and document preparation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Preserve tables during document ingestion.
• Continue with Deduplicate documents without erasing valid versions.

Reference: Qdrant payload and filtering documentation.