Embedding Model Review: Open-Source vs Hosted for RAG

The verdict first

I run a hosted embedding API in production, and I would probably do it again — but not for the reason I expected. The quality gap between the best open-weight model and the hosted APIs, on my corpus, was small enough that it barely showed up in my evaluation set. The reason I pay for a hosted API is that I do not want to own a GPU, and the operational difference turned out to be worth more than the quality difference.

The important thing to say up front: switching embedding models is a migration, not a config change. Every vector in your index was produced by one model, and vectors from two different models are not comparable even if the dimensions match. Change the model and you re-embed every chunk, re-insert everything, invalidate every cached query embedding, and re-tune the retrieval thresholds that were calibrated against the old score distribution. I did this once and it cost me a weekend and about a day of degraded retrieval while both indexes ran in parallel. That is the number I remember.

So pick carefully on the first try, and treat the choice as a multi-year commitment rather than a tunable knob.

What I tested on

• Corpus: roughly 1,200 technical articles, averaging around 1,400 tokens each, plus a smaller set of product pages and reference docs.
• Content mix: mostly English technical prose, with a meaningful minority of Chinese-language pages and some code blocks in the chunk stream.
• Chunking: 512 tokens with 64 overlap for prose, 768/128 for reference docs, following the RAG pipeline guide.
• Open models: bge-m3, the gte family, and nomic-embed, run locally through a Python inference service.
• Hosted APIs: one general-purpose hosted embedding endpoint and one vendor’s dedicated embedding API, both called through my gateway.
• Evaluation: a labelled set of roughly 80 real queries with the chunk a human would consider correct, plus the full corpus re-indexed per model.
• Cost magnitude: embedding a corpus this size is a small one-off bill per model, but the storage cost is permanent and scales with dimensions; the hosted API charges per token on every re-embed.

What it does well

• bge-m3 handles mixed-language content without a separate index. It embeds Chinese and English passages into one space and retrieves across both, which is the single most valuable property in my corpus. Maintaining one index per language would have doubled my ingestion pipeline and introduced a cross-language retrieval gap I would have had to design around.
• Long-context support is real, not marketing. Several of these models accept inputs well beyond the 512-token comfort zone of older embedding models, so a chunk size of 768 or 1024 does not get silently truncated. That matters more than it sounds: a truncated chunk loses its tail, and the tail is often where the answer is.
• Dense plus sparse output in one pass. bge-m3 produces a dense vector for semantic similarity plus a sparse vector for lexical matching. Running both in the same retrieval step gives you the recall of exact-match keyword search and the precision of semantic search, which is a genuinely better result than tuning either alone. The vector database comparison covers which stores handle hybrid search well.
• No per-token cost once running. After the GPU is paid for, embedding another document costs electricity. For a corpus that grows continuously, that changes the long-run arithmetic in a way that per-token pricing never quite matches.
• Hosted APIs remove capacity planning entirely. This is the whole reason I stayed hosted. My content volume spikes unpredictably, and a hosted endpoint absorbs a re-index spike without me provisioning anything, while keeping the model warm so there are no cold-start seconds on the first query. One provider, one key, one invoice is also a smaller operational surface than a GPU, a driver stack and a model file.

Where it fell short

• The open-weight models are large. bge-m3 in particular is not a model you run on a small instance. Serving it at a useful throughput needs real GPU memory, and if you quantise it to fit, retrieval quality drops in ways that are hard to predict and easy to misattribute to your chunking. Self-hosting “free” embeddings means buying hardware that competes with your generation workload.
• Dimension count is a permanent tax, not a one-off. A model at 1024 dimensions stores roughly twice what a 512-dimension model stores for the same corpus, and you pay that on every chunk, every backup, every re-index and every search. On a large corpus this dominates the cost difference people think about when they compare free versus paid. And the dimension is not negotiable — it is a property of the model.
• Hosted APIs make your embedding vendor a hard dependency. The moment you embed your corpus with a hosted endpoint, you cannot switch to a different provider without re-embedding everything. I have accepted that dependency, but I want to be clear that it is a decision, not a default.
• Vendor embedding models have quiet rate limits and batch constraints. The limits are not always published clearly, and the difference between a per-request limit and a token-per-minute limit changes how you batch ingestion. I found out my ingestion rate was being throttled by watching a job that should have taken an hour take most of a morning.
• Evaluating “is the new model better” is genuinely hard. Vendor leaderboards are measured on generic retrieval benchmarks with their own chunking and their own query distribution. On my corpus the gap between the top open-weight model and the hosted API was inside the noise of my evaluation set, and in a few cases the ranking flipped. The only comparison that told me anything was running my own queries against my own index.
• Nothing warns you that your index is stale. If the embedding service changes its default model, or you accidentally point a re-index job at a different endpoint, the old index and the new query embeddings disagree. Every search quietly returns worse results, and nothing errors. I now record the model identifier alongside the index and assert on it.

The setup

The local path is straightforward — a model, a tokenizer, and a normalisation step most people forget:

from sentence_transformers import SentenceTransformer
import numpy as np

# bge-m3 outputs dense (semantic) + sparse (lexical) + colbert vectors.
# For a plain vector index you take the dense output and normalise it.
model = SentenceTransformer("BAAI/bge-m3")
model.max_seq_length = 8192          # do not let it truncate at the default

def embed_passages(texts: list[str]) -> np.ndarray:
    vectors = model.encode(
        texts,
        batch_size=8,                # keep batches small; long inputs blow up GPU memory
        normalize_embeddings=True,   # cosine similarity == dot product on normalised vectors
        show_progress_bar=False,
    )
    return np.asarray(vectors, dtype="float32")

Two details do most of the work. normalize_embeddings=True means your similarity search can use inner product instead of cosine, which is faster in most vector stores. And max_seq_length must be raised explicitly if your chunks exceed the model’s default truncation point — the default is often much shorter than the model’s advertised capacity, and silent truncation is the most common cause of “why did the tail of my chunk never retrieve anything”.

The hosted path is a couple of HTTP calls, and batching is what keeps ingestion time reasonable:

import httpx

def embed_hosted(texts: list[str], model: str) -> list[list[float]]:
    out: list[list[float]] = []
    with httpx.Client(timeout=120.0) as client:
        for start in range(0, len(texts), 64):          # respect the batch limit
            batch = texts[start:start + 64]
            resp = client.post(
                "https://api.example-embeddings.com/v1/embeddings",
                headers={"Authorization": f"Bearer {EMBEDDING_API_KEY}"},
                json={"model": model, "input": batch},
            )
            resp.raise_for_status()
            data = resp.json()["data"]
            out.extend(item["embedding"] for item in sorted(data, key=lambda d: d["index"]))
    return out

Now the part that matters more than any of the above — the migration you will eventually need. The model identity has to travel with the index, and a dual-index comparison has to be explicit:

# The index is tagged with the model that produced it. Never let a query
# embedding come from a different model than the one the index was built with.
INDEX_MANIFEST = {
    "index": "articles-v3",
    "embedding_model": "bge-m3",
    "dimensions": 1024,
    "normalised": True,
    "built_at": "2026-10-01",
}

def assert_model_matches(index_manifest: dict, query_model: str) -> None:
    if index_manifest["embedding_model"] != query_model:
        raise RuntimeError(
            f"query embeddings from {query_model!r} cannot be compared with "
            f"an index built by {index_manifest['embedding_model']!r}"
        )

For a migration, build the new index under a new name, re-embed the corpus, then run both indexes against the same evaluation queries and compare recall before you cut over:

# 1. build the new index under a new name, never overwrite the live one
python scripts/build_index.py --model bge-m3 --index articles-v3 --corpus ./corpus

# 2. compare recall against the labelled query set
python scripts/eval_retrieval.py --index articles-v3 --queries ./eval/queries.jsonl \
  --out ./eval/bge-m3.json

# 3. only after the numbers agree, flip the alias
python scripts/promote_index.py --from articles-v2 --to articles-v3

The three-step order is the whole point. The alias flip is a one-line change that takes effect immediately; everything before it is reversible.

What it cost

Direct cost. The hosted API charges per token, and for a corpus this size the one-off re-embedding bill is small — check current pricing for the endpoint you use, since rates move. The self-hosted route has no per-token charge but no lower total cost either once you account for hardware: a GPU instance costs roughly what a modest hosted embedding bill costs over the same period, and it costs more if your corpus is small enough that the GPU idles most of the day.

Storage cost — the one that compounds. This is the number most comparisons omit. Storage is roughly chunks × dimensions × 4 bytes for float32, before the index overhead. Double the dimensions and you double that. For a large corpus this is the dominant embedding-related cost, and it recurs on every backup, every replica and every re-index. A 512-dimension model that is 95% as good is often the cheaper choice once storage is counted.

Inference cost. Self-hosting also means paying for the GPU during re-indexing spikes, which is exactly when you want the whole machine for other things. Hosted APIs let you burst.

The hidden cost: the migration. This is the one that actually hurt me. My last change involved re-embedding the entire corpus, running both indexes in parallel for a comparison period, watching retrieval quality for a week, and then re-tuning the similarity thresholds because the new model’s score distribution was not identical to the old one. That is a day of engineering work and a week of careful watching, and none of it appears in any pricing comparison.

What broke along the way

Retrieval quality dropped silently and I blamed the chunking. After a routine maintenance change, search results got noticeably worse and I spent half a day adjusting chunk sizes. The actual cause was that a config default had changed and my query embeddings were being produced by a different model than the index. Similarity scores between two unrelated vector spaces are near-random, and nothing errors — the system returns confident, wrong, plausible-looking results. The fix was to record the model identifier in the index manifest and assert on it at query time. That assertion has caught two mistakes since.

Long chunks were being truncated at the tail. My reference-doc chunks are 768 tokens, and for one model the default max sequence length was shorter than that, so the end of every long chunk was silently discarded before embedding. Symptom: chunks about a specific error message never retrieved, even when the error message was right there. The fix was raising max_seq_length explicitly and, more importantly, adding a check that logs when an input is truncated. I keep chunk sizes under whatever limit the chosen model imposes by measuring the corpus first rather than trusting the default.

A re-index job hit undocumented rate limits. My hosted re-embed stalled at a fraction of the expected speed. The API was accepting requests but throttling tokens per minute, and the response did not make that obvious. The fix was to batch requests, add explicit retry with backoff, and slow the job down deliberately rather than letting it hammer the endpoint. Check the published limits for your provider before you plan a bulk re-embed.

Mixing cosine and dot product. After switching stores, I configured the index to use inner product but kept the vectors un-normalised. Scores were off by a constant factor, which meant the absolute thresholds in my retrieval logic stopped working — everything above a fixed cut-off became either everything or nothing. Normalising the vectors at embedding time, as in the snippet above, makes the two equivalent and removes the class of bug entirely.

Would I use it again?

Yes, and I would change one thing: I would pick the embedding model once, deliberately, with an evaluation set already built, and I would write the index manifest and the model assertion into the ingestion code from day one. The thing that cost me was not the model choice — it was not having the infrastructure to notice when something drifted.

I would recommend a hosted embedding API to anyone who does not already run GPU inference, especially if your corpus is small enough that the quality difference is inside your evaluation noise. I would recommend an open-weight model when you have GPU capacity you are already paying for, when the content cannot leave your network, or when your corpus is large enough that per-token pricing genuinely adds up.

I would not recommend switching models for a small quality gain you cannot measure on your own queries. You will pay the migration cost for a difference you cannot demonstrate, and you will not be able to tell whether the migration helped.

Pros and cons

What it gets you

• Open-weight models: no per-token cost once the hardware is paid for, multilingual retrieval in a single index, and content that never leaves your network
• Hosted APIs: no hardware, no capacity planning, no cold starts, and burst capacity for re-index spikes
• Open-weight models: dense plus sparse output from the same pass, which is the cheapest way to get hybrid search
• Hosted APIs: newer models without an upgrade cycle, and one billing relationship instead of a GPU you maintain

What it costs

• Open-weight models: a GPU, a driver stack and a model file you now operate, plus cold-start latency and your own capacity planning
• Open-weight models: dimension counts that carry a permanent storage tax, and quantisation quality loss that is hard to attribute when it happens
• Hosted APIs: per-token cost on every re-embed forever, and a hard vendor dependency that makes switching expensive
• Hosted APIs: rate limits and batch constraints that are not always obvious, and your corpus lives off your infrastructure

Who this is for

This is for whoever owns the retrieval layer of an application with a real corpus — a few hundred documents or a few hundred thousand. If you have a real query set and can measure recall, this decision is a half-day of work. If you do not have an evaluation set yet, build that first; it is worth more than any model choice, and the evaluation guide covers how to put one together.

For a small internal tool with a few dozen documents, the model barely matters and you should not spend time on it — use whatever your stack already provides and focus on chunking. For a large corpus where retrieval quality is the product, the dimension tax and the storage line item deserve more attention than the leaderboard ranking does, and the LLM cost breakdown shows how to fold embedding cost into your per-request maths.

FAQ

Q: Are open-source embedding models good enough for production? A: Yes, for most retrieval workloads. The multilingual open-weight models are competitive on quality and cost nothing per call once they are running. The reasons to pay for a hosted API are operational — no GPU, no cold starts, no capacity planning — rather than a large quality gap on typical corpora.

Q: What happens if I change my embedding model later? A: You rebuild the entire index. Vectors from different models are not comparable, so every document has to be re-embedded and re-inserted, and every cached query embedding has to be discarded. Plan for that as a project, not a config change.

Q: Does the number of embedding dimensions matter for cost? A: Yes, directly and permanently. More dimensions means more storage per chunk and more compute per search. A model at 1024 dimensions roughly doubles what a 512-dimension model costs to store, and that cost recurs on every re-index, backup and index rebuild.

Q: Do I need a different embedding model for each language? A: Usually not, if you pick a genuinely multilingual model. A multilingual model handles mixed-language content in one index, which avoids the complexity and the cross-language retrieval gap that comes from maintaining one index per language.

Q: How do I know if a new embedding model is actually better? A: Run your own evaluation set against both indexes and compare recall and ranking on real queries. Vendor leaderboards are measured on generic benchmarks, and the gap on your corpus is usually much smaller than the leaderboard suggests — and sometimes reversed.