Build a production-minded RAG application on Azure

RAG is a system, not a longer prompt

Retrieval-augmented generation grounds a model in selected content at request time. Reliability depends on ingestion, parsing, chunking, indexing, retrieval, authorization, generation, citations, and evaluation working together.

• Content preparation and refresh
• Search index and retrieval strategy
• Permission-aware access
• Prompt assembly and grounded generation
• Evaluation and operational monitoring

Define the question before the architecture

Write down who asks questions, which repositories are authoritative, how fresh the content must be, and what the system must do when evidence is missing. These decisions determine the index, security model, retrieval latency, and user experience.

• Authoritative sources and ownership
• Required freshness and deletion behavior
• User or document-level permissions
• Citation and abstention requirements
• Latency and cost budgets

Build the ingestion path

Extract text and structure without losing document identity. Choose chunks that preserve meaning and attach metadata that supports filtering, citations, versioning, and deletion.

• Stable document and chunk identifiers
• Title, section, source URL, version, and access metadata
• Repeatable parsing and chunking
• Incremental refresh and deletion handling
• A quarantine path for failed documents

Choose retrieval deliberately

Keyword search is strong for exact terms, vector search captures semantic similarity, and hybrid search combines both. Start with a simple measurable baseline, then add semantic ranking or query planning only when the evaluation set shows a real benefit.

• Classic RAG: the application sends a query and orchestrates the handoff to the model
• Agentic retrieval: query planning can decompose complex requests and return structured grounding data
• Preview capabilities require explicit production-risk review

Measure retrieval before response quality

Create a small gold dataset of realistic questions and expected source documents. If the correct evidence is absent from the retrieved set, prompt changes cannot repair the core failure.

• Did the expected source appear in the top results?
• Were irrelevant or stale chunks retrieved?
• Did access filters exclude unauthorized content?
• How did retrieval choices affect latency and cost?

Generate answers with inspectable evidence

Pass only the most useful evidence to the model, retain source identifiers, and instruct the application to abstain when the evidence does not support an answer. A citation is useful only when the reader can open it and verify the claim.

• Use bounded context rather than dumping many documents
• Map answer citations back to stable source locations
• Distinguish model knowledge from retrieved evidence
• Return a useful no-answer state

Protect identity and data boundaries

Use Microsoft Entra ID and role-based access where supported instead of embedding shared keys. The production query identity should receive only the permissions it needs, and the retrieval layer must enforce content permissions before evidence reaches the model.

• Use managed identity and DefaultAzureCredential
• Prefer read-only query roles for serving applications
• Apply network restrictions where the risk model requires them
• Never rely on the model to enforce document authorization

Evaluate the whole answer path

Track retrieval and generation separately, then test them together. Include normal questions, ambiguous wording, outdated information, conflicting documents, unauthorized content, prompt injection inside documents, and questions with no supported answer.

• Retrieval relevance and expected-source coverage
• Groundedness and citation correctness
• Answer completeness and abstention
• Latency, token use, and cost per answer
• Security and access-control tests

Production readiness checklist

Before widening access, add observability, budgets, incident paths, content refresh monitoring, feedback handling, and a rollback plan. Store the minimum telemetry needed to debug quality without creating a new sensitive-data repository.

• Log identifiers and measurements, not unnecessary document text
• Alert on ingestion failures and stale indexes
• Set token, latency, and request limits
• Version prompts, retrieval settings, and evaluation results
• Document owners, support route, and shutdown procedure

Choose your next step

Use the roadmap as a loop, not a reading list. Pick the next action that produces evidence.