Nvidia's Nemotron 3 Embed Leads RTEB, but the Rank Is Not a Cost Verdict

• Nvidia's open-weight Nemotron-3-Embed-8B-BF16 is No. 1 on the live RTEB beta leaderboard, ahead of Voyage 4 Large and Octen-Embedding-8B.
• Nvidia's 78.46 score covers 16 public tasks at a 4,096-token evaluation length; the live page lists 30 tasks but shows no mean for the leading models.
• The smaller 1.14B BF16 checkpoint ranks 14th, and switching between Nvidia's 1B and 8B tiers can require separate vector indexes.
• Nvidia's throughput and agent-cost advantages remain vendor-run claims tied to selected hardware, workloads and pricing assumptions.

Nvidia has put an open-weight 8B embedding model at the top of RTEB, a retrieval benchmark intended to test which documents, code and stored memories reach an AI application. The result is a meaningful quality marker for Nemotron 3 Embed. It is not a finding that the model is the cheapest retriever to run, or even a transparent average over every task on the current leaderboard.

That distinction matters because Nvidia released three checkpoints with different operational roles. The largest wins this benchmark contest. The smaller BF16 model is pitched for production efficiency, while an NVFP4 version makes its strongest serving claim on Nvidia's Blackwell hardware. Buyers therefore face a portfolio decision, not a clean sweep by one model at every size and cost point.

What the No. 1 rank actually measures

The archived live leaderboard ranks Nemotron-3-Embed-8B-BF16 first, Voyage 4 Large with evolved prompts second and Octen-Embedding-8B third. Nemotron-3-Embed-1B-BF16 is 14th, immediately ahead of Qwen3-Embedding-8B.

The page describes 30 tasks across 22 languages, but its “Mean (Task)” cells are blank for the leading models. Its per-task row also lacks a SWE-bench code-retrieval result for Nvidia's 8B entry, while several competitors have still more empty task cells. The page supplies a rank without exposing in the archived body a like-for-like aggregate that a reader can recalculate across equal task coverage.

Nvidia's model card provides a different, narrower comparison: average NDCG@10 over 16 public RTEB tasks, with model sequence length set to 4,096 tokens.

The 1B BF16 model improves by 10.4 RTEB points over Nvidia's prior 1B baseline within that company comparison. A technical account reports 72.00 for the NVFP4 checkpoint, 0.38 below its BF16 parent. Those are substantial gains for Nvidia's smaller line, but neither smaller checkpoint owns the headline rank.

The lead is also a snapshot. In January, Voyage said Voyage 4 Large had replaced Voyage 3 Large at the top of RTEB. Six months later it is second. The same announcement points to an operational alternative Nvidia does not match within this collection: Voyage 4 models share an embedding space, allowing customers to change model tiers without re-indexing stored vectors. Nvidia's 8B output is 4,096-dimensional and its 1B outputs are 2,048-dimensional; the technical account says a system routing between those tiers needs two indexes.

RTEB's missing private check weakens the generalization claim

RTEB was introduced in beta to counter “teaching to the test” on familiar public datasets. Its original design combined public tasks, which anyone could reproduce, with private tasks evaluated by maintainers to test performance on unseen data.

That private column is now gone. In a January decision, MTEB maintainers said Voyage helped develop RTEB and had direct access to its private evaluation data. Voyage had by then been acquired by MongoDB. The maintainers called that access an “undeniable structural advantage” that undermined trust in the leaderboard, while stressing that no one had alleged misuse.

The maintainers kept the private datasets and said requested evaluations and results remain available. They plan to restore the column after diversifying the private set, ideally with datasets supplied by organizations that do not develop models. Until then, the public rank lacks the benchmark feature originally intended to test generalization beyond public data.

The benchmark's designers identified another limitation at launch: about half of its then-current retrieval datasets had been repurposed from question-answering data. Strong lexical overlap between questions and context can favor keyword matching over semantic understanding. RTEB remains useful and broader than a single-domain test, but its own documentation does not support treating first place as proof of superior retrieval on an unseen enterprise corpus.

Open weights move control to buyers, along with the integration bill

Nemotron-3-Embed-8B-BF16 converts a Ministral-3-8B-Instruct-2512 causal decoder into a bidirectional encoder, averages token representations into a 4,096-dimensional vector and accepts inputs up to 32,768 tokens. Nvidia says it evaluated the model across 34 languages and trained it with more than 50 million public, commercially permissible and synthetic samples.

The maximum context window is a capability limit, not the setting behind the published benchmark table. Nvidia ran the cited RTEB, ViDoRe and MMTEB evaluations at 4,096 tokens. Independent results closer to 32,768 tokens would be needed to show how quality and serving cost behave across the advertised range.

The 1B model was not trained from scratch. Nvidia adapted a 3B parent and used two rounds of structured pruning and distillation to reach 1.14B parameters. The resulting 1B checkpoints produce 2,048-dimensional vectors and target lower-latency serving; the NVFP4 version quantizes weights and activations in linear layers for hardware-accelerated inference.

Nvidia licenses the released model materials under OpenMDW 1.1 for commercial use and publishes adaptation and distillation recipes. That expands deployment control, but it does not remove diligence. The model card says third-party software has separate terms, warns that the model can retrieve the wrong passage and calls for use-case testing. Its privacy subcard says no personal data is known to have been used, but also says user interactions contributed to training and that correction or removal of externally sourced data is not possible.

Availability also varies by route. Nvidia's launch announcement offers weights, NIM, vLLM support and cloud or inference partners. The archived 8B model page, however, says that checkpoint is not deployed by a Hugging Face inference provider. A contemporaneous release account repeats Nvidia's three-tier accuracy, latency and Blackwell positioning but supplies no independent cost or performance test.

Open weights let organizations keep retrieval on infrastructure they control and fine-tune on their own corpus. They also move GPU capacity, vector storage, index migration, operations and validation onto the buyer. That is why an 8B leaderboard win cannot by itself resolve the choice between self-hosting, Nvidia's NIM stack and a hosted embedding API.

The efficiency case is promising, but vendor-measured

Nvidia says the 1B NVFP4 checkpoint retains more than 99% of BF16 retrieval accuracy while delivering up to twice the throughput on Blackwell. It also says a Rust-based NIM microservice matches or beats vLLM on GB200 and RTX PRO 6000 GPUs at input lengths of 256 and 1,024. Those tests support a Blackwell deployment case; they do not provide total cost against hosted competitors or non-Nvidia hardware.

The agent-economics experiment is narrower still. Nvidia held a Nemotron 3 Ultra search agent constant, changed the embedding model and measured ViDoRe V3, BRIGHT and BrowseComp-Plus. It says the 8B embedder produced the highest retrieval accuracy and the lowest estimated downstream token cost.

That estimate used Nemotron 3 Ultra input and output counts with a GPT-5.5 pricing formula. It was not a customer invoice and did not isolate embedding inference, latency, vector storage or the cost of serving an 8B model. The experiment supports the mechanism that better retrieval can reduce repeated searches and later reasoning; it does not establish a lower end-to-end bill.

The partner evidence is similarly early. Automation Anywhere, Boomi and You.com supplied favorable comments, while IBM, Mem0, Palantir, ServiceNow, Zep and Zoom were described as evaluating the models. Those relationships show distribution and interest, not independent, long-running comparisons.

The next evidence must connect retrieval quality to total system cost

The next decision is not whether Nemotron 3 Embed won this RTEB snapshot; it did. The unresolved choice is whether the 8B leader, the smaller Nvidia tiers or a hosted alternative produces the best result on a buyer's own data after every system cost is counted.

A useful production test should hold the agent, corpus and vector database constant, then measure retrieval quality, end-to-end task success, latency, accelerator memory, embedding storage and serving cost together. It should also count the re-indexing or dual-index burden when model dimensions differ, and test whether Nvidia's published gains survive on data none of the model developers could inspect.

Two benchmark questions remain decisive. RTEB needs a credible, diversified private comparison restored, and Nvidia needs reproducible results at longer inputs rather than only the 4,096-token setting behind its published table. Buyers handling sensitive data also need a clearer account of the model card's user-interaction training disclosure and the limits on data correction.

Until those gaps close, Nemotron 3 Embed gives Nvidia a real retrieval win and a path from open weights to NIM and Blackwell. This RTEB snapshot identifies a quality leader. It does not decide who should pay the infrastructure cost, accept a lower-ranked smaller model, maintain multiple indexes or buy retrieval as an API.