This weblog is collectively written by Md Rahman, Arkaprabho Ghosh, Navin Bilwar, and Desh Shukla.
Government abstract
Cisco IT just lately evaluated fine-tuning embedding fashions utilizing NVIDIA Nemotron RAG fine-tuning recipe as a part of an effort to enhance retrieval accuracy for domain-specific enterprise knowledge. The target was to not redesign present retrieval-augmented technology (RAG) methods, however to grasp whether or not focused embedding fine-tuning might materially enhance semantic search high quality with cheap effort and quick turnaround. By way of this experiment, Cisco was in a position to validate firsthand that embedding fine-tuning, mixed with artificial knowledge technology, can ship measurable accuracy positive factors inside a short while body. The experiment additionally demonstrated sturdy time-to-value, enabling speedy iteration and clear efficiency indicators with out lengthy coaching cycles or intensive handbook labeling. The diminished turnaround of just a few days to grasp the quick advantages was a key end result of this collaboration.
The embedding mannequin coaching and analysis workflow was executed on Cisco AI PODs operating Cisco UCS 885A infrastructure powered by NVIDIA HGX platform.
Drawback assertion
Previous to conducting this experiment, Cisco had performed related embedding fine-tuning experiments utilizing earlier technology fashions and smaller scale infrastructure. These prior efforts required important handbook tuning of hyperparameters corresponding to batch measurement and variety of epochs, and outcomes have been usually tough to stabilize. Iteration cycles have been lengthy, making it difficult to discover totally different configurations or scale experiments. Regardless of some localized enhancements, key phrase search remained obligatory for a lot of domain-specific retrieval situations. There was additionally no standardized, end-to-end workflow that engineering groups might execute rapidly and consider persistently throughout runs. Usually, these efforts would take weeks to months of handbook effort for unsure positive factors.
How the positive‑tuning went and time to worth
On this experiment, Cisco used the NVIDIA NeMo Retriever embedding finetuning recipe, leveraging artificial knowledge technology to provide coaching indicators from present corpora. The recipe runs by 5 distinct levels: artificial knowledge technology (SDG), knowledge preparation with hard-negative mining, contrastive fine-tuning, BEIR analysis, and ONNX mannequin export. The workflow was in a position to run end-to-end efficiently. All experiments ran on a single NVIDIA H200 143 GB GPU hosted inside Cisco AI Pods constructed on Cisco UCS 885A methods. Finetuning runs accomplished inside hours of coaching time, enabling speedy experimentation throughout a number of dataset sizes and configurations. Using artificial knowledge technology eradicated the necessity for handbook labeling, considerably lowering overhead. This strategy allowed Cisco to iterate rapidly, observe efficiency developments early, and validate whether or not embedding fine-tuning was price additional funding. The general time-to-value was considerably shorter than earlier efforts, with significant insights gained after solely a small variety of runs.
The five-stage pipeline structure:
Timings based mostly on ~925 paperwork / ~9,200 QA pairs / ~7,800 coaching examples on a single NVIDIA H200 GPU operating on Cisco AI Pods with Cisco UCS 885A infrastructure. Precise period scales with knowledge quantity.
Accuracy positive factors noticed
Throughout a number of experiments, the outcomes confirmed constant, measurable enhancements. Positive-tuning the NVIDIA 1-billion-parameter NV-EmbedQA mannequin on artificial domain-specific knowledge yielded positive factors throughout all retrieval metrics, with NDCG@1 positive factors of +7.1 to +7.3 absolute factors (+9.9% to +11.1% relative). Recall@10 improved by as much as +6.8 factors (+8.5%), and MAP@10 by as much as +6.5 factors (+9.7%). Utilizing an on-premise 120B-parameter LLM for artificial knowledge technology, all the pipeline ran with zero exterior API prices and with the information staying utterly on prem ensured knowledge privateness. These positive factors held at the same time as dataset measurement elevated and retrieval duties turned tougher. Importantly, enhancements have been noticed on domain-specific queries that beforehand carried out poorly with base embedding fashions. Whereas these outcomes symbolize an preliminary baseline relatively than a completely optimized end result, they supplied sturdy affirmation that embedding fine-tuning can materially enhance retrieval high quality for enterprise-specific knowledge.
Abstract of experiments
Desk 1. Retrieval efficiency comparability between the bottom embedding mannequin and the contrastively fine-tuned mannequin throughout two dataset sizes (334 and 925 paperwork). Positive-tuning persistently improves rating high quality throughout all BEIR analysis metrics.
Key Observations:
- Positive-tuning persistently improved retrieval high quality throughout all metrics.
- NDCG@1 confirmed the biggest enchancment in top-level relevance.
- Positive factors have been secure throughout the 2 dataset sizes (334 and 925 paperwork).
- Improved Recall@10 and Map@10 positive factors indicative of higher protection and rating than the bottom embedding mannequin.
What shocked us
Probably the most surprising discovering was how rapidly the recipe delivered actionable outcomes. Inside a couple of days of beginning the experiment, we had measurable accuracy enhancements — a stark distinction to earlier efforts that took weeks to months. The artificial knowledge technology strategy produced coaching indicators of enough high quality to drive significant positive factors with no single manually labeled instance. We have been additionally shocked by how properly the enhancements generalized throughout question varieties, together with the rare-token identifier queries that had traditionally been the weakest level for semantic search.
Subsequent steps with engagement
Constructing on these outcomes, Cisco will proceed working with NVIDIA to systematically push accuracy additional. The following part of labor will focus on:
- Utilizing a hard and fast analysis set throughout runs in order that metrics will likely be straight comparable
- Tuning the educational price (attempting default, half, and double) and growing epochs from 3 to five
- Scaling coaching knowledge to ~100K QA pairs to seek out the saturation level for the area
- Utilizing a bigger or higher-quality LLM for artificial knowledge technology to enhance QA pair constancy
- Making use of 10% warmup with cosine decay for extra secure convergence
- Rising hard-negative mining from 5 to 10 negatives per question for a stronger contrastive sign
- Refining artificial knowledge technology prompts to raised emphasize uncommon and domain-specific phrases — bug IDs, product identifiers, firmware variations — the place base fashions battle most
- Exploring chunk-aware coaching: utilizing actual doc chunks from a manufacturing vector database because the retrieval corpus, producing questions in opposition to these chunks through the LLM, and mapping every query to its constructive chunk and hard-negative chunks — coaching the mannequin on the identical knowledge distribution it will encounter in manufacturing, the place solutions could also be buried in longer textual content and chunking methods will range
Long term, the engagement will broaden to incorporate re-ranker fine-tuning and broader retrieval optimization as a part of a full end-to-end RAG enchancment effort.
Worth of the fine-tuning embedding mannequin
This experiment helps that leveraging a fine-tuning embedding mannequin can speed up time to manufacturing by offering a validated, end-to-end fine-tuning workflow that delivers measurable enhancements in days relatively than months. The concepts and findings from this work are actively shaping the recipe’s evolution, whereas Cisco positive factors early entry to a maturing pipeline that shortens the trail from experimentation to manufacturing. The work additionally demonstrates how Cisco AI Pods based mostly on Cisco UCS 885A methods and NVIDIA H200 GPUs can present an efficient enterprise infrastructure basis for speedy embedding mannequin adaptation.
Key fine-tuning embedding mannequin advantages for companies
- Defend proprietary knowledge (on-premises execution)
- Cut back assist prices (quicker decision, fewer escalations)
- No cloud API dependency (zero exterior prices)
- Quick time-to-value (full end-to-end pipeline — all 5 levels together with SDG, mining, coaching, analysis, and export — completes in 2-5 hours on a single GPU)
Key fine-tuning embedding mannequin advantages for builders
- No handbook annotation required (artificial knowledge technology)
- Modular, hackable structure (5 distinct levels: SDG → Knowledge Prep → Positive-Tune → Consider → Export)
- Manufacturing-ready outputs (ONNX export)
- Constructed-in analysis (BEIR — Benchmarking Info Retrieval — framework)
- Exhausting adverse mining included (computerized high quality enhance)
Get began
The fine-tuning recipe for Llama Nemotron Embed 1B mannequin is on the market now as a whole, production-ready pipeline. Whether or not you’re constructing enterprise search, RAG functions, or domain-specific retrieval methods, this recipe offers a transparent path from uncooked paperwork to deployed, domain-adapted embeddings.
Able to fine-tune your individual embedding mannequin?
👉 Discover the Nemotron Embed Positive-Tuning Recipe on GitHub
