RAG System Overview: Embeddings, HNSW, BM25, Hybrid Search, and Reranking

1. Core pipeline

A practical RAG system takes a user query, retrieves the most relevant information, then augments the LLM prompt with that retrieved context before generation.

The important engineering point is that retrieval is a multi-stage pipeline. Each stage narrows down the search space and improves context quality before the LLM sees anything.

User Query
  ↓
Embed Query (Transformer)
  ↓
Vector Search (HNSW)
  ↓
(Optional) BM25 Search
  ↓
Hybrid Merge
  ↓
(Optional) Filters
  ↓
(Optional) Reranking
  ↓
Top-K Results
  ↓
Prompt Augmentation
  ↓
LLM Response

2. Embeddings and cosine similarity

Embeddings convert text into vectors that capture semantic meaning. The main benefit is that similar ideas end up close together even when the wording changes.

Cosine similarity is then used to compare the query vector against document vectors. That gives a semantic closeness score rather than a direct keyword match score.

This is why semantic search can retrieve relevant documents that do not repeat the user query exactly. The tradeoff is that embeddings are weaker when the query depends on exact identifiers, names, or rare tokens.

"cheap winter travel" → [0.21, -0.88, ...]

similarity = dot(v1, v2) / (|v1| * |v2|)

• Use embeddings when meaning matters more than exact wording.
• Use cosine similarity to rank candidate chunks by semantic closeness.
• Do not rely on embeddings alone for IDs, codes, or exact phrases.

3. Why HNSW matters

If you compare a query vector to every stored vector directly, retrieval becomes too slow at scale. HNSW solves that by using an approximate nearest-neighbor graph instead of brute-force comparison.

The index organizes vectors in multiple layers. Upper layers help make a fast coarse jump toward the right region, and lower layers refine the search around nearby candidates.

This gives you a strong speed gain with a small accuracy tradeoff, which is usually the right decision for production retrieval systems.

• HNSW stands for Hierarchical Navigable Small World.
• It is approximate, not perfectly exact.
• The approximation is usually worth the latency improvement.

4. BM25 and exact keyword retrieval

BM25 is a statistical ranking method based on term frequency and inverse document frequency. It is still extremely useful in RAG because many user queries depend on exact wording.

When the user asks for a specific product name, place, acronym, or error code, BM25 can outperform semantic retrieval because it rewards direct term overlap.

The weakness is that BM25 does not understand meaning well. It knows token statistics, not semantic relationships.

• BM25 is strong for exact terms, rare words, and names.
• BM25 is weak when relevant documents use different wording than the query.

5. Hybrid search is the practical default

Most real systems should not choose only semantic retrieval or only keyword retrieval. Hybrid search combines both signals into one ranking process.

The usual pattern is to take a vector score and a BM25 score, normalize them, and merge them with a weight parameter such as alpha. That lets you tune the system based on your data and query patterns.

This is one of the biggest practical improvements you can make in a RAG stack because it protects you from the weaknesses of both methods.

score = alpha * vector_score + (1 - alpha) * bm25_score

6. Reranking and filtering improve precision

After the first retrieval pass, a reranker can score query-document pairs more precisely. Unlike embeddings, which encode query and document independently, a reranker evaluates them jointly.

That makes reranking slower but more accurate, so the usual pattern is to retrieve a wider candidate set first, then rerank a smaller top slice.

Filtering is the structured side of retrieval. If your data includes budget, date, rating, region, or document type, filters should be applied to narrow the search space before or after retrieval.

• A common pattern is top 20 candidates first, then rerank to top 5.
• Filters are equivalent to SQL-style WHERE clauses on structured fields.
• Use semantic retrieval for meaning and filters for hard constraints.

7. Vector database and collection design

A vector database such as Weaviate stores structured properties and embeddings together, then provides retrieval methods on top of both. In practice, that means one object can have semantic fields and filterable fields at the same time.

The key design rule is simple: text that carries meaning should be vectorized, and structured attributes should remain filter fields. Mixing those responsibilities usually creates weak retrieval behavior.

{
  "place": "Banff",
  "description": "...",
  "budget": "Moderate",
  "user_ratings": 4.7
}

• Collection is the rough equivalent of a database table.
• Properties are the rough equivalent of columns.
• Descriptions, attractions, and titles are good vectorized fields.
• Ratings, IDs, dates, and enums are usually better as filters.

8. Ingestion flow and query types

When new data is inserted, the semantic fields are vectorized, stored, and indexed so they can participate in retrieval. This ingestion path is effectively part of your indexing pipeline.

At query time, you usually choose between filter-only fetches, pure semantic search, BM25, hybrid search, and reranked retrieval depending on the question.

Insert Object
  ↓
Vectorization (Transformer)
  ↓
Store vector + properties
  ↓
Update HNSW index

• Fetch means structured lookup only.
• Semantic search means vector similarity lookup such as near_text.
• BM25 means exact keyword ranking.
• Hybrid search merges semantic and keyword signals.
• Reranked search adds a second-pass precision step.

9. Retrieval metrics and system tradeoffs

Precision at K tells you how clean the top results are. Recall at K tells you how much of the relevant set you managed to retrieve. You usually cannot maximize both at once.

As K increases, recall tends to improve because you include more candidates, but precision drops because lower-ranked results are weaker. This matters because the LLM context window is limited and low-quality chunks pollute prompts.

In practice, smaller high-quality top K values usually outperform large noisy context windows.

10. Practical RAG design principles

The most important design rule is that retrieval quality controls generation quality. If the context is wrong, the LLM can only produce a confident answer from the wrong material.

That is why chunking, hybrid retrieval, filters, reranking, and evaluation matter more than most prompt tweaks. Prompting matters, but it only starts after retrieval is already good enough.

• Retrieval quality matters more than prompt cleverness.
• Use small top-K windows with strong relevance.
• Default to hybrid retrieval in production systems.
• Apply structured filters whenever hard constraints exist.
• Use reranking when precision matters.
• Chunk long text carefully before embedding.

11. Backend mental model

If you already think in terms of databases and indexing, RAG becomes easier to reason about. The pieces are not mysterious. They are just retrieval infrastructure adapted for LLM consumption.

RAG is not about the LLM. It is about retrieving the right context before calling the LLM.