Microsoft Fabric RAG Tutorial — Build Trustworthy AI

What Is RAG and Why It Matters for Enterprise AI

Retrieval-Augmented Generation (RAG) is an AI pattern that combines a search step with a generation step. When a user asks a question, the system first retrieves the most relevant documents from a knowledge base, then passes those documents to a language model as context. The model generates its answer from that retrieved context — not from its pre-training weights alone.

For enterprise use, this distinction is what makes RAG deployable where standalone generative AI is not. A language model’s pre-training knowledge is static, anonymous, and unverifiable. A RAG system’s answers are grounded in your specific documents, traceable to specific sources, and reflect the current state of your data — which is the minimum bar for regulated industries and internal business intelligence use cases.

Microsoft Fabric RAG Architecture

A production RAG system on Microsoft Fabric has four layers: storage, indexing, retrieval, and generation. Each maps to a specific Fabric or Azure service.

The Fabric Lakehouse acts as the source of truth for raw documents — not Azure Blob Storage directly — so that the knowledge base inherits Fabric’s governance controls, workspace permissions, and sensitivity labels. Azure AI Search sits outside Fabric but is configured to private endpoints within the same Azure virtual network, keeping data within the trust boundary.

Setting Up a RAG Pipeline on Microsoft Fabric

1. Provision the Fabric workspace and LakehouseCreate a Fabric workspace on F2 or above. Add a Lakehouse — this is where raw documents land. Create a folder structure: raw/ for incoming files, processed/ for cleaned and chunked outputs.
2. Configure Azure OpenAI and AI SearchDeploy an Azure OpenAI resource in the same region as your Fabric capacity. Deploy a text-embedding model (text-embedding-3-large recommended over ada-002 for new deployments). Create an Azure AI Search resource in the same region. Configure a private endpoint if working in a regulated environment.
3. Upload and prepare documentsUpload source documents to the Lakehouse raw/ folder. Use a Fabric Notebook to extract text from PDFs and Word files using pypdf or python-docx. Write cleaned text to the processed/ folder as plain text or JSON with metadata (source filename, last modified date, section title).
4. Chunk documents and generate embeddingsSplit cleaned documents into chunks (see Section 04 for chunking strategy). Generate embeddings for each chunk using the Azure OpenAI embedding deployment. Write chunks and their embedding vectors to the Azure AI Search index.
5. Build the retrieval and generation pipelineAt query time: embed the user question, search the AI Search index for the top-K most similar chunks, assemble a prompt combining the retrieved chunks with the user question, call Azure OpenAI GPT-4o for generation, and return the answer with source citations.
6. Evaluate and iterateRun the Fabric evaluation notebook against a test question set. Check groundedness, relevance, and faithfulness scores. Adjust chunking or retrieval strategy based on where scores fall below threshold (see Section 07).

from azure.search.documents import SearchClient
from azure.search.documents.models import VectorizedQuery
import openai

def retrieve_and_generate(user_question: str, top_k: int = 5) -> dict:
    # 1. Embed the question
    embedding = openai.embeddings.create(
        model   = "text-embedding-3-large",
        input   = user_question
    ).data[0].embedding

    # 2. Retrieve relevant chunks from AI Search
    vector_query = VectorizedQuery(
        vector         = embedding,
        k_nearest_neighbors = top_k,
        fields         = "content_vector"
    )
    results = search_client.search(
        search_text    = user_question,   # hybrid search
        vector_queries = [vector_query],
        select         = ["content", "source_file", "chunk_id"],
        top            = top_k
    )
    chunks = [{"content": r["content"], "source": r["source_file"]} for r in results]

    # 3. Assemble the grounded prompt
    context = "

".join([f"[{c['source']}]
{c['content']}" for c in chunks])
    prompt  = f"""Answer the question using only the context provided below.
If the context does not contain the answer, say "I don't have enough information."
Cite the source file for each claim.

Context:
{context}

Question: {user_question}"""

    # 4. Generate the grounded answer
    response = openai.chat.completions.create(
        model    = "gpt-4o",
        messages = [{"role": "user", "content": prompt}]
    )
    return {"answer": response.choices[0].message.content, "sources": chunks}

Chunking Strategy — The Decision That Most Affects Retrieval Quality

Chunking is the process of splitting source documents into segments before embedding them into the vector index. Chunk size is the most important single configuration decision in a RAG system — it determines whether the retrieved context is too broad (losing precision) or too narrow (losing surrounding meaning).

Overlap between adjacent chunks ensures that a sentence spanning a chunk boundary is present in full in at least one chunk. Without overlap, the retriever may return a chunk that begins mid-sentence and lacks the context needed to generate a correct answer.

Embedding Models — Choosing the Right Representation

Embeddings convert text chunks into numerical vectors in a high-dimensional space where semantically similar content is positioned close together. The quality of your embedding model determines how well the retriever can find relevant chunks for a given question — including questions phrased differently from the source document wording.

RAG vs Classic Chatbot — Why It Matters for Compliance

The distinction between RAG and a standalone generative AI chatbot is not primarily about user experience — it is about evidence. A chatbot generates from pre-training; a RAG system generates from your data. That difference has direct implications for whether AI output can be trusted, cited, and audited in business decisions.

Evaluating RAG Accuracy in Microsoft Fabric

Microsoft Fabric provides evaluation notebooks that compute three core RAG metrics against a test question set. Each metric targets a different failure mode, and together they tell you exactly where the system is breaking down.

from azure.ai.evaluation import GroundednessEvaluator, RelevanceEvaluator

groundedness = GroundednessEvaluator(model_config=model_config)
relevance    = RelevanceEvaluator(model_config=model_config)

result = groundedness(
    response = answer,
    context  = "
".join([c["content"] for c in retrieved_chunks])
)
print(f"Groundedness: {result['groundedness']}")  # Score 1-5

RAG for Regulated Industries — Compliance Architecture

For financial services, healthcare, insurance, legal, and government use cases, the auditability of source citation is the compliance differentiator. RAG on Fabric is well-positioned for these environments because the full trust boundary is configurable within Azure and Fabric infrastructure the organization already controls.

Troubleshooting RAG Accuracy in Fabric

Low-Code RAG with Fabric AI Skills

For teams without Python expertise or those building an initial proof of concept, Fabric AI Skills provides a graphical interface for connecting data sources to a natural language query experience — without writing orchestration code.

1. Create a new AI Skill in the Fabric workspaceNavigate to the workspace → New item → AI Skill. This opens the AI Skill configuration interface.
2. Connect data sourcesAdd a Lakehouse, Warehouse, KQL database, or Power BI semantic model as a data source. The AI Skill uses the connected sources to answer questions using auto-generated SQL, DAX, or KQL.
3. Configure natural language instructionsProvide plain English instructions about your data — business terminology, important tables, and any rules for how the AI should interpret ambiguous questions. These instructions are injected into the prompt at query time.
4. Test and publishUse the built-in test interface to ask questions and review the generated queries and answers. Publish the AI Skill to make it available as a REST endpoint for teams to integrate into applications or Power BI reports.

FAQ — Microsoft Fabric RAG Tutorial

Official Resources — Microsoft Fabric RAG Documentation