Retrieval-Augmented Generation, commonly known as RAG, has become a practical architecture for building AI applications that need access to private, domain-specific, or frequently changing information.
In the previous article, we looked at what RAG is, why modern AI applications need it, how retrieval works, and where RAG can be useful.
This article takes the next step.
Instead of discussing RAG only as a concept, we'll build the pieces of a RAG application using C# and .NET.
The goal is not to create a complicated enterprise AI platform. We'll start with the core pipeline and then look at how the same architecture can evolve into a production application.
A simplified .NET RAG architecture looks like this:
facebook
linkedin
instagram
twitter
Documents
↓
Document Processing
↓
Text Chunks
↓
Embeddings
↓
Vector Store
↓
User Question
↓
Semantic Search
↓
Relevant Context
↓
LLM
↓
Generated Response
Modern .NET provides abstractions for AI services and vector stores, allowing applications to avoid tightly coupling their business logic to a single AI provider or vector database.
Let's build the architecture step by step.
For this example, imagine a small knowledge assistant.
The application contains documents such as:
Company Policy
Product Documentation
Return Policy
Support Guide
Technical Documentation
A user can ask:
What is the return period for damaged products?
The application will:
The important part is that the LLM does not need to contain all of this information in its training data.
The application retrieves the relevant information at runtime.
There are several ways to build RAG applications in .NET.
For a modern .NET application, a useful starting point is:
C#
ASP.NET Core
Microsoft.Extensions.AI
Microsoft.Extensions.VectorData
Embedding Model
Vector Store
LLM
Microsoft.Extensions.AI provides common .NET abstractions for AI services, while Microsoft.Extensions.VectorData provides abstractions for working with vector stores.
This separation is useful because your application code does not necessarily need to know the implementation details of every underlying provider.
For example:
Application
↓
Microsoft.Extensions.VectorData
↓
Vector Store Provider
↓
Actual Vector Database
This makes it easier to change infrastructure later.
For a simple proof of concept, you can start with a console application.
dotnet new console -n DotNetRag
cd DotNetRag
For a real application, you could instead start with an ASP.NET Core Web API:
dotnet new webapi -n DotNetRag.Api
cd DotNetRag.Api
If the RAG functionality will be part of an existing business application, there is no need to create a separate application.
You can add the RAG layer to your existing ASP.NET Core architecture.
For example:
MyApplication
│
├── Api
├── Application
├── Domain
├── Infrastructure
├── AI
└── Tests
The AI layer can contain services responsible for embeddings, retrieval, prompt construction, and AI responses.
The exact packages depend on the AI provider and vector store you choose.
For the .NET abstraction layer, the important packages are:
dotnet add package Microsoft.Extensions.AI
dotnet add package Microsoft.Extensions.VectorData.Abstractions
You then add the provider-specific packages for your embedding model, LLM, and vector store.
For example, during development you can use an in-memory vector store.
Microsoft's current .NET documentation uses CommunityToolkit.VectorData.InMemory in its minimal vector-search example. This is useful for learning and prototyping because it does not require a separate vector database.
For production, you would normally select a persistent provider.
Before creating embeddings, define what a searchable record looks like.
For example:
public class KnowledgeDocument
{
[VectorStoreKey]
public int Id { get; set; }
[VectorStoreData]
public string Title { get; set; } = string.Empty;
[VectorStoreData]
public string Content { get; set; } = string.Empty;
[VectorStoreData]
public string Category { get; set; } = string.Empty;
[VectorStoreVector(dimensions: 1536)]
public ReadOnlyMemory<float> Embedding { get; set; }
}
The exact embedding dimensions depend on the embedding model you use.
The important idea is that a record contains:
ID
Title
Content
Metadata
Embedding
The vector store uses the embedding for semantic search while the other properties provide the actual content and metadata.
Microsoft's vector store abstractions use attributes such as VectorStoreKey, VectorStoreData, and VectorStoreVector to describe how properties participate in the vector store schema.
A RAG system is only as useful as the information it can retrieve.
Suppose you have this document:
Product Return Policy
Customers can request a refund for damaged
products within 14 days of delivery.
The product must include the original invoice.
You could store the entire document as one record.
But real documents are often much larger.
A 100-page PDF should not normally be inserted into the vector store as one enormous record.
Instead, split it into smaller chunks.
For example:
Document
↓
Chunk 1
Chunk 2
Chunk 3
Chunk 4
...
Each chunk can contain metadata:
DocumentId
Title
Section
Page
Category
Content
Embedding
This makes retrieval more useful and allows the application to provide source information later.
The next step is converting each chunk into an embedding.
Conceptually:
"Customers can request a refund within 14 days."
↓
Embedding Model
↓
[0.12, -0.43, 0.81, ...]
The embedding represents the semantic characteristics of the text.
When a user asks:
How long do customers have to request a refund?
the question is also converted into an embedding.
The application can then compare the query embedding with document embeddings.
This is what allows semantic search to find related information even when the exact wording is different.
For development, an in-memory vector store can be convenient.
Conceptually:
var vectorStore = new InMemoryVectorStore(
new()
{
EmbeddingGenerator = embeddingGenerator
});
Then create a collection:
var collection =
vectorStore.GetCollection<int, KnowledgeDocument>(
"knowledge");
You can ensure the collection exists:
await collection.EnsureCollectionExistsAsync();
The current Microsoft example follows this same general pattern for creating a vector collection and searching it.
Once you have the document content and embedding generator, you can add records.
For example:
var document = new KnowledgeDocument
{
Id = 1,
Title = "Product Return Policy",
Content =
"Customers can request a refund for damaged " +
"products within 14 days of delivery.",
Category = "Returns"
};
The embedding needs to be generated for the searchable content.
Conceptually:
Document Content
↓
Embedding Generator
↓
Vector
↓
Vector Store
Then the record can be inserted:
await collection.UpsertAsync(document);
For a real ingestion pipeline, you would normally process many documents and chunks rather than manually inserting individual records.
Now comes the retrieval part.
Suppose the user asks:
How long can I return a damaged product?
The application sends the question through the embedding generator and searches the vector store.
A simplified search can look like:
await foreach (
var result in collection.SearchAsync(
"How long can I return a damaged product?",
top: 3))
{
Console.WriteLine(result.Record.Title);
Console.WriteLine(result.Score);
}
The vector store returns the most semantically similar records.
For example:
Result 1
Product Return Policy
Score: 0.91
Result 2
Shipping Policy
Score: 0.73
Result 3
Customer Support Guide
Score: 0.68
The application can then use the highest-quality results as context.
Retrieval by itself does not generate an answer.
The retrieved records need to be combined into context for the LLM.
For example:
var context = string.Join(
"\n\n",
results.Select(x => x.Record.Content));
The resulting context might be:
Customers can request a refund for damaged
products within 14 days of delivery.
The product must include the original invoice.
Now the application has the information required to answer the question.
The final prompt can be structured like this:
Answer the user's question using only the
provided context.
Context:
Customers can request a refund for damaged
products within 14 days of delivery.
The product must include the original invoice.
Question:
How long can I return a damaged product?
The model can then generate:
Customers can request a refund for a damaged
product within 14 days of delivery. The original
invoice is also required.
This is the generation part of RAG.
The complete flow is:
Question
↓
Embedding
↓
Vector Search
↓
Top Results
↓
Context
↓
LLM
↓
Answer
In a real application, you should avoid placing all of this logic inside a controller.
Instead, create a dedicated service.
For example:
public interface IRagService
{
Task<string> AskAsync(
string question,
CancellationToken cancellationToken = default);
}
Then implement it:
public class RagService : IRagService
{
private readonly IEmbeddingGenerator<string, Embedding<float>>
_embeddingGenerator;
private readonly IChatClient _chatClient;
public RagService(
IEmbeddingGenerator<string, Embedding<float>>
embeddingGenerator,
IChatClient chatClient)
{
_embeddingGenerator = embeddingGenerator;
_chatClient = chatClient;
}
public async Task<string> AskAsync(
string question,
CancellationToken cancellationToken = default)
{
// Retrieve relevant context
// Build prompt
// Ask the LLM
// Return answer
return string.Empty;
}
}
The exact registration depends on the provider you choose, but the architectural idea remains the same.
Then your API can remain simple:
[ApiController]
[Route("api/rag")]
public class RagController : ControllerBase
{
private readonly IRagService _ragService;
public RagController(IRagService ragService)
{
_ragService = ragService;
}
[HttpPost("ask")]
public async Task<IActionResult> Ask(
[FromBody] AskRequest request)
{
var answer =
await _ragService.AskAsync(request.Question);
return Ok(new
{
answer
});
}
}
This separation keeps AI-specific logic out of the HTTP layer.
For a production application, the architecture should be more structured.
ASP.NET Core API
↓
RagService
↓
┌────────────┴────────────┐
↓ ↓
Retrieval Service AI Service
↓ ↓
Vector Store LLM
↓
Knowledge Base
For ingestion:
Documents
↓
Document Processor
↓
Text Extraction
↓
Chunking
↓
Embedding Generator
↓
Vector Store
This separation gives you more control over the system.
One common mistake is generating embeddings every time a user asks a question.
That is unnecessary.
Document processing should normally happen separately.
For example:
New Document
↓
Process
↓
Create Chunks
↓
Generate Embeddings
↓
Store Vectors
Then user queries only perform retrieval:
User Question
↓
Query Embedding
↓
Vector Search
↓
LLM
This avoids repeating expensive ingestion work.
Microsoft's current .NET AI documentation also separates data ingestion from retrieval workflows and provides ingestion abstractions that can create chunks and write them to supported vector stores.
Semantic similarity alone is not always enough.
Imagine your application contains documents from multiple companies.
Company A
Company B
Company C
A search for:
What is the return policy?
could find a highly relevant document from the wrong company.
Therefore, retrieval should often combine semantic search with metadata filtering.
For example:
TenantId = 1001
Category = "Returns"
Language = "en"
Then semantic search happens only within the permitted scope.
Conceptually:
User Question
↓
Authorization
↓
Metadata Filter
↓
Semantic Search
↓
Relevant Documents
This is critical for multi-tenant applications.
Vector search is powerful, but it is not the answer to every search problem.
Some queries work better with exact text matching.
For example:
Invoice INV-10452
A semantic search engine might understand the meaning, but an exact identifier is often better handled with keyword or structured search.
This is why many modern retrieval systems support hybrid search, combining semantic/vector retrieval with traditional text search.
Microsoft's vector-data documentation includes support for vector search and filtering, while managed search services such as Azure AI Search can combine full-text and vector search.
A practical architecture can therefore be:
User Question
↓
Query Understanding
↓
┌────────┴────────┐
↓ ↓
Keyword Search Vector Search
↓ ↓
└────────┬────────┘
↓
Combined Results
↓
LLM
If your application already uses PostgreSQL, you do not necessarily need to introduce a completely separate database for every AI feature.
Depending on your PostgreSQL setup and vector extension/provider strategy, vectors can be stored alongside application data.
For example:
PostgreSQL
│
├── Products
├── Customers
├── Orders
├── Sales
└── KnowledgeChunks
The exact vector implementation depends on your infrastructure and provider.
The more important architectural decision is to keep structured business data and AI retrieval responsibilities clearly separated.
For example:
Product table
↓
Transactional business data
KnowledgeChunk
↓
Searchable AI knowledge
This makes the purpose of each data set clearer.
Let's apply this to an inventory management application.
Suppose the system contains:
Products
Sales
Purchases
Suppliers
Stock
Product Documentation
Return Policies
A user asks:
Why can't I return this product?
The application may need information from several sources.
First:
Product
↓
Product Policy
Then:
Return Policy
↓
Relevant Knowledge
The AI can combine those results and produce a response.
For example:
The product is outside the standard return
period. Your policy allows returns within
14 days of delivery for eligible products.
But imagine another question:
How many units of Product A are currently available?
This is different.
Current stock is transactional data.
You should not necessarily embed every stock quantity into a vector database and expect semantic search to answer accurately.
Instead:
User
↓
AI / Application
↓
InventoryService
↓
PostgreSQL
↓
Current Stock
Now consider:
Product A is selling quickly. Should we reorder it?
That may require both types of information:
Current Stock
+
Sales History
+
Reorder Rules
+
Product Documentation
↓
AI
This demonstrates an important point:
RAG does not have to replace your existing business services.
It can work alongside them.
If your application already uses Repository Pattern or layered architecture, don't throw it away just because you are adding AI.
A practical design can look like:
API
↓
Application Service
↓
AI/RAG Service
├── Retrieval
├── Embeddings
└── LLM
↓
Infrastructure
├── Vector Store
├── PostgreSQL
└── External AI Provider
For example:
InventoryService
↓
PostgreSQL
RagService
↓
Vector Store
↓
LLM
The AI layer can call existing application services when it needs structured business information.
This is generally better than allowing the LLM to directly query the database.
Security is especially important in RAG applications.
Imagine an internal company knowledge base containing:
HR Documents
Financial Reports
Customer Information
Internal Policies
Technical Documentation
The retrieval layer must respect user permissions.
A simple architecture might look like:
User
↓
Authentication
↓
Authorization
↓
Tenant / Role Filter
↓
Retrieval
↓
LLM
Do not assume that because a document is in the vector database, it is safe to provide it to every user.
The retrieval layer should enforce the same access rules used elsewhere in the application.
Also consider:
AI retrieval is still an application feature and should follow normal security practices.
A RAG application should not be judged only by whether the AI produces fluent answers.
You should evaluate at least two things:
Did the system find the correct information?
Did the LLM produce an accurate answer using the retrieved information?
Consider:
Question
↓
Expected Documents
↓
Retrieved Documents
↓
Generated Answer
You can build a test set such as:
Question 1 → Expected Source A
Question 2 → Expected Source C
Question 3 → Expected Source B
Then test the retrieval pipeline after changing:
This is important because a change that improves one query can sometimes make another query worse.
Large chunks can contain too much unrelated information.
Very small chunks can lose the context required to understand the information.
Metadata such as tenant, document type, language, category, and permissions can significantly improve retrieval design.
Exact IDs, numbers, dates, and structured values may require traditional database queries.
More retrieved text does not automatically mean a better answer.
The application should retrieve useful information rather than everything that looks remotely relevant.
A RAG application can look impressive during a demo and still perform poorly on real questions.
Create a representative test set before considering the system production-ready.
There is no single vector database that every .NET application should use.
Your choice depends on:
The .NET ecosystem provides vector store abstractions so applications can work with different provider implementations through a common programming model. Current Microsoft documentation lists provider integrations across technologies such as Azure AI Search, SQL Server, Cosmos DB, MongoDB, Elasticsearch, and others.
For a prototype:
In-Memory Vector Store
can be enough.
For production:
Managed Search Service
or
Persistent Vector Database
or
Database with Vector Support
may be more appropriate.
The important thing is to choose based on the application's requirements rather than selecting a vector database simply because it is popular.
If your .NET application is already running heavily on Azure, Azure AI Search can be an attractive option.
It supports traditional search capabilities as well as vector and hybrid search scenarios.
This can be useful when the application needs:
Full Text Search
+
Vector Search
+
Filtering
+
Metadata
+
Enterprise Search
Azure AI Search is also part of Microsoft's current guidance for RAG and AI search scenarios.
However, Azure AI Search is not mandatory for building RAG.
A RAG architecture can use other vector stores depending on the project's requirements.
Once the basic pipeline works, there are several ways to improve it.
Experiment with chunk size and overlap based on the document type.
Restrict retrieval to documents the user can actually access.
Combine keyword and semantic search.
Retrieve a larger candidate set and rerank the most relevant results.
Keep document metadata so the final answer can reference its sources.
Use previous conversation information carefully without allowing irrelevant history to overwhelm retrieval.
Create automated tests for retrieval and answer quality.
These improvements should be introduced based on measured problems rather than added all at once.
A more complete production architecture could look like this:
User
↓
ASP.NET Core API
↓
RagService
↓
Query Processing
↓
┌────────┴────────┐
↓ ↓
Metadata Filter Query Embedding
↓ ↓
└────────┬────────┘
↓
Vector Search
↓
Reranking
↓
Relevant Context
↓
Prompt Builder
↓
LLM
↓
Final Response
And the ingestion side:
Documents / Data
↓
Data Ingestion
↓
Text Extraction
↓
Chunking
↓
Metadata
↓
Embeddings
↓
Vector Store
Keeping ingestion and query processing separate makes the system easier to maintain and scale.
Building a RAG system in .NET does not require rebuilding your entire application around AI.
The core pipeline is relatively simple:
Data
↓
Chunks
↓
Embeddings
↓
Vector Store
↓
Semantic Search
↓
Relevant Context
↓
LLM
↓
Answer
The difficult part is usually not writing the first few lines of C#.
The real engineering work is deciding how to prepare your data, how to chunk it, how to retrieve the right information, how to enforce permissions, how to combine semantic and structured search, and how to evaluate whether the final answers are actually useful.
For .NET developers, Microsoft.Extensions.AI and Microsoft.Extensions.VectorData provide useful abstractions for building this kind of application while keeping AI and vector-store implementations less tightly coupled. Microsoft's current .NET AI ecosystem specifically positions these libraries as building blocks for applications that need AI and grounding with their own data.
A good way to start is with a small dataset and an in-memory vector store:
10–20 Documents
↓
Chunk
↓
Embedding
↓
Vector Search
↓
LLM
Once the basic retrieval quality is acceptable, you can move toward a persistent vector store, metadata filtering, hybrid search, citations, evaluation, authentication, and production monitoring.
The most important architectural lesson is that RAG should complement your existing application rather than replace it.
Your PostgreSQL database can continue handling transactions.
Your application services can continue enforcing business rules.
Your repository layer can continue managing persistence.
And the RAG layer can provide the AI with relevant knowledge when it needs it.
That makes RAG a practical addition to an existing .NET application rather than a reason to redesign the entire system.
For developers already working with C#, ASP.NET Core, PostgreSQL, SQL Server, or Azure, this is one of the most practical ways to start building AI-powered features around real application data.
Contact us today to schedule a free, 20-minute call to learn how DotNet Expert Solutions can help you revolutionize the way your company conducts business.
About us
Comments 0