Choosing an Embedding Model for Domain RAG
Why the leaderboard winner may not be the right model for your data, and how to test candidates on the questions your experts actually ask
The embedding model decides what your RAG system can find. If it places the right passage far from the question, no prompt, re-ranker or larger LLM can fix the answer, because the right information never reaches the model.
HomeoAI, an AI assistant used by more than 1,400 homeopathic practitioners, learned this by trial and error. Its builder, Tejesh Vaish, described on the Codersarts Podcast how the choice played out:
Attempt | What happened |
OpenAI's small embedding model | Cheap, but doctors said it wasn't catching all the relevant medicines |
OpenAI's large embedding model | Doctors were happy with the results, but it was expensive |
Google's Gemini embedding model | Doctors were happy and the cost was acceptable, so it became the final choice |
The decision wasn't made from a benchmark. It was made by showing results to doctors. This guide explains how to make the same decision deliberately for your own domain. For the full HomeoAI architecture, read How HomeoAI Was Built.
What an embedding model does in RAG
An embedding model turns text into a list of numbers, a vector, that represents its meaning. In a RAG system it is used twice:
At indexing time: every chunk of your documents is embedded once and stored in a vector database.
At query time: each user question is embedded with the same model, and the database returns the chunks whose vectors are closest.
Two consequences follow. The question and the documents must be embedded with the same model. And changing models later means re-embedding every document.
Why domain RAG is harder
General-purpose embedding models are trained mostly on everyday web text. Professional knowledge stresses them in specific ways:
Specialist vocabulary: remedy names, drug names, legal terms and product codes may be rare in training data, so their vectors are less precise.
Near-identical entries: in homeopathy, one symptom can map to a hundred medicines. In law or tax, many provisions share most of their wording. The model must separate passages that look almost the same.
Mixed languages and scripts: many professional sources mix English with Latin, Sanskrit, Hindi or other languages.
Structured text: tables, numbered rubrics and short reference rows behave differently from paragraphs.
That's why a model that ranks highly on a public leaderboard can still miss answers in your domain.
The criteria that matter
1. Retrieval quality on your data
This is the deciding factor. Measure how often the correct passage appears in the top results for real questions from your domain. Everything else is a tie-breaker.
2. Domain and language coverage
Check whether the model handles your vocabulary and every language your documents and users use. Test with the rarest, most specific terms you have.
3. Cost
There are three costs to estimate:
Indexing: embedding your whole corpus once.
Queries: embedding every user question, which grows with usage.
Re-indexing: embedding everything again when documents change or you switch models.
For HomeoAI, the large model's quality came at a cost that the Gemini model matched for less.
4. Vector size
Larger vectors can capture more detail but need more storage, more memory and slightly slower search. Some models let you shorten vectors with only a small loss in quality. Test whether the shorter version still passes your quality bar.
5. Input length
Each model has a maximum input length. Your chunk size must fit within it, or text is cut off silently. If you change models, re-check your chunking.
6. Hosting and privacy
Hosted APIs are quick to start and need no infrastructure, but your text is sent to a third party.
Self-hosted open-source models keep data on your servers and avoid per-call fees, but you run and scale the model yourself.
For regulated data, this can decide the choice before quality is even measured.
7. Lock-in
Switching models means re-embedding everything and re-testing retrieval. Choose carefully, and design your system so switching is possible.
Your options at a glance
Option | Examples | Strengths | Trade-offs |
Hosted, cost-focused | OpenAI text-embedding-3-small | Low cost, simple to use | May miss subtle or specialist matches |
Hosted, quality-focused | OpenAI text-embedding-3-large, Google Gemini embedding models | Strong general retrieval quality | Higher cost per token, larger vectors |
Open-source, self-hosted | BGE, E5, Nomic Embed and other sentence-transformers models | Data stays on your servers, no per-call fees, can be fine-tuned | You host, scale and update the model |
Fine-tuned on your domain | An open-source model trained further on your question–passage pairs | Best fit for specialist vocabulary | Needs training data, effort and ongoing maintenance |
Prices, model versions and quality change often. Check providers' current documentation before deciding, and treat this table as a map, not a ranking.
Leaderboards are a starting point
The MTEB (Massive Text Embedding Benchmark) leaderboard compares embedding models across many tasks and languages. It's useful for building a shortlist. It can't tell you how a model performs on your documents, your vocabulary and your users' questions. Use it to pick three candidates, then test them yourself.
How to choose: a 6-step test
Shortlist three models: typically one low-cost hosted model, one high-quality hosted model and one open-source model.
Build a test set: 50 to 100 real questions, each with the passages a correct answer should use. Our guide to building a golden dataset explains how.
Embed a representative sample of your documents with each model, using the same chunking.
Measure retrieval: recall@k (how often the right passage is in the top k) and mean reciprocal rank (how high it ranks).
Show results to experts. For borderline cases, ask domain experts which results they would trust. This is what decided HomeoAI's choice.
Compare cost and pick the cheapest model that passes your quality bar, not the one with the highest score at any price.
Python sketch for comparing models
This example compares embedding models on a test set using cosine similarity. Each embed function takes a text and returns a vector from the model you're testing.
import numpy as np
def build_index(chunks, embed):
vectors = np.array([embed(c["text"]) for c in chunks], dtype=float)
return vectors / np.linalg.norm(vectors, axis=1, keepdims=True)
def search(question, embed, vectors, chunks, k=5):
q = np.array(embed(question), dtype=float)
q = q / np.linalg.norm(q)
top = np.argsort(-(vectors @ q))[:k]
return [chunks[i]["id"] for i in top]
def recall_at_k(retrieved, relevant):
return len(set(retrieved) & set(relevant)) / len(relevant) if relevant else 0.0
def evaluate_model(embed, chunks, golden, k=5):
vectors = build_index(chunks, embed)
scores = [recall_at_k(search(item["question"], embed, vectors, chunks, k), item["relevant_ids"])
for item in golden]
return sum(scores) / len(scores)
# models = {"small-model": embed_small, "large-model": embed_large, "open-source": embed_oss}
# for name, embed in models.items():
# print(name, round(evaluate_model(embed, chunks, golden), 3))
In production, embed in batches, cache results, and store vectors in a vector database rather than in memory.
Embeddings are one part of retrieval quality
Even the best embedding model has blind spots. Three other choices interact with it:
Chunking: HomeoAI found chunks that were too small missed medicines, and chunks that were too large pulled in irrelevant ones. Re-test chunk size whenever you change models.
Hybrid search: adding BM25 keyword search catches exact terms that embeddings blur. See Hybrid Search for RAG.
Re-ranking: a re-ranking step can promote the best passage even when the embedding ranks it only fifth.
If a cheaper embedding model plus hybrid search and re-ranking passes your quality bar, that combination may beat an expensive model on its own.
Switching models safely
Store the model name and version with every vector, so you always know what produced it.
Build the new index in parallel while the old one keeps serving users.
Run your test set on both and switch only if the new model is at least as good.
Budget for re-embedding the whole corpus, and for re-testing chunking.
Keep the old index briefly so you can roll back.
Embedding model checklist
Area | Question to ask | Good answer |
Quality | How was the model chosen? | Tested on real domain questions, with recall@k, MRR and expert review |
Vocabulary | Does it handle your rarest terms? | Tested with specialist names, codes and terms |
Languages | Does it cover every language in your sources and queries? | Multilingual tests included in the test set |
Cost | Do we know indexing, query and re-indexing costs? | Estimated for current data and expected growth |
Vector size | Are vectors larger than we need? | Shorter vectors tested where the model supports them |
Input length | Do chunks fit the model's limit? | Chunk size checked against the model's maximum input |
Privacy | Can this data be sent to a hosted API? | Hosted or self-hosted chosen based on data rules |
Complements | Are we relying on embeddings alone? | Hybrid search and re-ranking tested alongside |
Switching | Can we change models later? | Model version stored with vectors, parallel indexing planned |
Get your RAG retrieval right with Codersarts
Choosing an embedding model is one of the most important early decisions in a RAG system, and one of the most expensive to get wrong. Codersarts helps teams make it with evidence:
Embedding model evaluation: we test candidate models on your documents and questions, with retrieval metrics, expert review and a cost estimate.
Retrieval tuning: chunking, hybrid search and re-ranking tuned together with the model you choose.
Self-hosted and fine-tuned embeddings: for teams whose data can't leave their servers or whose vocabulary is highly specialised.
Model migrations: re-embedding and switching models without downtime or quality loss.
Talk to us: https://www.codersarts.com/contact
Explore our AI services: https://ai.codersarts.com
FAQ
What is the best embedding model for RAG?
There is no single best model. The right one is the cheapest model that retrieves the correct passages for your real questions often enough. Shortlist candidates using a leaderboard such as MTEB, then test them on your own data and questions.
How do I evaluate embedding models for my data?
Build a test set of 50 to 100 real questions with the passages a correct answer should use. Embed your documents with each candidate, measure recall@k and mean reciprocal rank, and have domain experts review borderline results.
Should I use an open-source or a hosted embedding model?
Hosted models are quicker to start and need no infrastructure. Open-source models keep data on your servers, avoid per-call fees and can be fine-tuned. If your data can't be sent to a third party, self-hosting may decide the choice.
Do I need to re-embed my documents if I change the embedding model?
Yes. Vectors from different models aren't compatible, so every document must be embedded again with the new model. Build the new index in parallel, test it, and switch only when it performs at least as well.
Does a bigger embedding model always give better RAG results?
No. Larger models are often stronger in general, but they cost more and may not be better on your domain. HomeoAI's doctors were happy with a large model, but chose a Gemini model that matched the quality at a lower cost.
Can Codersarts help choose and set up embeddings for our RAG system?
Yes. Codersarts evaluates embedding models on your data, tunes chunking, hybrid search and re-ranking, and sets up hosted, self-hosted or fine-tuned embeddings. Contact us at https://www.codersarts.com/contact.
Related reading:
Want to be a guest on the Codersarts Podcast? Email jitendra@codersarts.com with your name, role and the topic you'd like to talk about.
Comments