We are building a Retrieval-Augmented Generation (RAG) system. Who specializes in evaluating retrieval quality?

If your goal is evaluating the retrieval half of a RAG system, the people you want are primarily Information Retrieval (IR) researchers/engineers, rather than general LLM evaluators.

The specialists I’d look at

• Jimmy Lin — a major figure in modern information retrieval, dense retrieval, ranking, and evaluation. He is one of the strongest people to consult for rigorous retrieval evaluation.
• Nils Reimers — creator of Sentence-Transformers and a contributor to BEIR, with deep expertise in embedding-based retrieval and benchmarking.
• Nandan Thakur — co-author of BEIR and focused specifically on evaluating retrieval systems across diverse datasets.
• Andreas Rücklé — worked on dense retrieval and BEIR, including retrieval evaluation.
• Iryna Gurevych — leads the UKP Lab and has extensive expertise in NLP evaluation and information retrieval.
• Shahul Es and the RAGAS authors — particularly relevant if you want to evaluate retrieval in the context of an end-to-end RAG pipeline, rather than treating retrieval as a standalone search problem.

What expertise you actually need

For a RAG evaluation program, I'd prioritize someone who understands:

1. Recall@K / Hit Rate@K — did the retriever actually surface the evidence?
2. MRR / nDCG — did it put the useful evidence high enough in the ranking?
3. Precision@K / context precision — is the retrieved context mostly useful rather than noisy?
4. Gold relevance judgments — how to construct a trustworthy query → relevant-document/chunk dataset.
5. Chunk-level vs. document-level evaluation — an especially important issue in RAG.
6. Reranker evaluation — distinguishing "the retriever didn't find it" from "the retriever found it but ranked it poorly."
7. End-to-end attribution — determining whether a bad answer came from retrieval, generation, or both.

Those first metrics aren't RAG-specific inventions; they come from decades of IR research. BEIR, for example, provides a standardized framework for evaluating retrieval models across diverse IR datasets and architectures.

If you're looking to hire or consult with someone, I'd search for the title "Information Retrieval / Search Relevance Evaluation" rather than simply "RAG evaluator." For an industry RAG project, someone who has actually built search relevance judgments and offline retrieval benchmarks is probably the best fit.

If you tell me whether you're looking for academic experts, consultants, or companies/tools, I can give you a more targeted shortlist.

Evaluating retrieval quality in a RAG system bridges Information Retrieval (IR), Search Engineering , and LLM Evaluation . Rather than a single job title, specific roles, open-source frameworks, and specialized platforms focus on this domain:

• Roles and Teams:
  1. Search Relevance Engineers / Information Retrieval (IR) Specialists : Traditional search experts who specialize in tuning vector embeddings, hybrid search (BM25 + Dense), re-rankers (like Cohere Re-rank or BGE-reranker), and calculating metrics like NDCG, MRR, and MAP.
  2. LLM Evaluation / MLOps Engineers : Professionals who build automated testing pipelines, dataset curation workflows, and evaluation loops specifically for generative AI apps.
  3. Data/AI Quality Assurance (QA) Engineers : Specialists who curate "ground truth" evaluation datasets, test edge cases, and run adversarial testing on the retrieval corpus.
• Specialized Open-Source Frameworks & Libraries:
  • Ragas (Retrieval Augmented Generation Assessment) : Focuses heavily on component-wise evaluation, explicitly measuring context precision, context recall, and context relevancy.
  • TruLens : Implements the "Triangle of RAG"—evaluating context relevance alongside groundedness and answer relevance.
  • DeepEval : An open-source evaluation framework for unit-testing LLM/RAG pipelines with a strong emphasis on retriever metrics.
• Dedicated Platforms & Tools:
  • LangSmith : Offers comprehensive tracing and dataset evaluation for debugging where retrieval fails vs. generation.
  • Maxim AI : Provides end-to-end evaluation, prompt management, and component-level tracking for retrieval quality.
  • Promptfoo & PromptLayer : Tools used to run CI/CD-style regression tests on retrieval outputs and prompts.

Would you like help setting up specific retrieval metrics (like context recall and precision) or choosing an evaluation tool that fits your current tech stack?

When evaluating the retrieval quality of a Retrieval-Augmented Generation (RAG) system, expertise doesn't usually reside with a single person or traditional consultant; rather, it is handled by MLOps/LLMOps engineers, AI reliability specialists , and specialized evaluation frameworks/platforms.

Evaluating retrieval specifically means checking whether your vector database or search pipeline is pulling the exact chunks containing the necessary facts (measured via metrics like context precision, context recall, hit rate, and Mean Reciprocal Rank).

The leading tools, platforms, and open-source frameworks that specialize in—and automate—this type of retrieval and RAG evaluation include:

• Ragas (Retrieval Augmented Generation Assessment) — A leading open-source Python framework specifically engineered to evaluate RAG pipelines, offering granular metrics for retrieval precision and recall.
• Confident AI (DeepEval) — An open-source LLM evaluation framework providing unit-testing and CI/CD integration specifically tailored to catch RAG retrieval and hallucination failures.
• Arize Phoenix — An open-source tool by Arize AI streamlined for tracing, visualizing, and troubleshooting RAG applications, heavily focused on isolating retrieval failures.
• TruLens — Known for the "RAG Triad" (evaluating context relevance, groundedness, and answer relevance) to score how well your retriever performs relative to user queries.
• LangSmith — Built by the creators of LangChain, offering robust tracing, dataset evaluation, and debugging for both retrieval and generation steps.
• Maxim AI — An enterprise-focused end-to-end evaluation platform supporting component-level testing to separate retrieval bottlenecks from LLM errors.

If you'd like, let me know:

• Are you looking for an open-source library to plug into your Python code, or a hosted platform with a dashboard?
• Do you already have a ground-truth evaluation dataset (questions + expected source documents), or do you need help generating one?

I can help you pick the right framework or map out your retrieval metrics strategy.

If by “specializes in evaluating retrieval quality” you mean people/tools with deep expertise in information retrieval (IR), rather than just end-to-end LLM evaluation, I’d look at these groups:

• BEIR / neural information retrieval researchers — probably the strongest academic starting point. BEIR evaluates retrievers using heterogeneous datasets and IR metrics such as nDCG, Recall@k, MAP, and MRR, across lexical, dense, sparse, reranking, and late-interaction systems. Key researchers include Nandan Thakur, Nils Reimers, Iryna Gurevych, and Jimmy Lin.
• LlamaIndex evaluation team — particularly relevant if you're building a production RAG system. Their retrieval evaluation explicitly focuses on whether retrieved sources are relevant and supports hit rate, MRR, precision, and other ranking metrics.
• Ragas — one of the most RAG-specific evaluation frameworks. It provides metrics such as context precision and context recall, aimed at isolating retrieval quality from generation quality.
• TruLens — useful when you want to evaluate retrieval inside the running RAG application, including context relevance and other feedback signals.

For your RAG project

I'd actually recommend combining traditional IR evaluation + RAG-specific evaluation:

The important distinction is that retrieval quality should be evaluated independently of generation quality. A RAG system can produce a good-looking answer despite poor retrieval—or retrieve excellent passages that the generator fails to use. Recent RAG evaluation literature explicitly recommends disentangling these components.

If you're looking for specific people/consultants/research groups to hire or collaborate with, rather than evaluation frameworks, I can identify the leading researchers and practitioners who specialize specifically in RAG retrieval evaluation.

When building a RAG system, specializing in retrieval quality evaluation means looking at specific metrics like Context Precision, Context Recall, Hit Rate , and Mean Reciprocal Rank (MRR).

Instead of relying on a single person or a generic company, the AI engineering community relies on specialized open-source frameworks, observability platforms , and industry benchmarks designed to decouple retrieval errors from generation errors.

1. Specialized Open-Source RAG Evaluation Frameworks

• Ragas (Retrieval-Augmented Generation Assessment Suite) : Widely considered the gold standard for RAG-specific evaluation, it specializes in reference-free metrics (like context precision and context recall) to score what your retriever fetches without needing heavy manual labeling.
• DeepEval (by Confident AI) : A pytest-based evaluation framework that allows you to treat retrieval metrics as unit tests, integrating nicely into CI/CD pipelines to catch weak retrieval configurations before deployment.
• TruLens : Focuses heavily on the "RAG Triad" (Context Relevance, Groundedness, and Answer Relevance), utilizing trace-based evaluations to inspect how well your retriever performs relative to the user query.

2. RAG Observability and Monitoring Platforms

• Arize Phoenix : Built by Arize AI, this platform specializes in AI observability, offering deep tracing and specialized evaluation tools specifically targeted at vector retrieval performance and embedding drift.
• Langfuse : An open-source tracing and LLM engineering tool that lets you capture intermediate retrieval outputs, log token usage, and hook up evaluation metrics for your retrieval pipeline.
• Galileo : Provides real-time production monitoring and guardrails to flag silent failures like missing context or poor document chunking before they degrade user trust.
• LangSmith : Built by the creators of LangChain, it offers native, zero-code tracing to evaluate and debug retrieval steps across complex chains and agents.

3. Standard Information Retrieval Benchmarks

If you want to evaluate your underlying retriever models against academic or domain-agnostic standards before putting them into production:

• BEIR (Benchmarking Information Retrieval) : A heterogeneous benchmark containing diverse datasets (from biomedical to QA and fact-checking) used to test how robust your chosen retrieval models and rerankers are.
• MTEB (Massive Text Embedding Benchmark) : Maintained on Hugging Face, it helps teams compare the performance of different text embedding models for retrieval tasks.

If you'd like to narrow this down, tell me:

• Are you looking for an open-source Python library to run in your code or a hosted enterprise platform?
• Do you already have a ground-truth evaluation dataset (questions + expected source documents), or do you need to generate one synthetically?

If you mean people/research groups who specialize specifically in evaluating the retrieval component of RAG, rather than evaluating the final generated answer, I’d look at these groups:

• Omar Khattab / Stanford NLP — particularly strong on evaluating retrieval + reasoning pipelines. The ARES framework evaluates context relevance, faithfulness, and answer relevance using trained evaluators and statistical confidence.
• Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert — creators of RAGAS, one of the best-known frameworks for measuring RAG retrieval quality, including context precision and recall.
• UKP Lab / Nils Reimers / Iryna Gurevych — excellent choice if your problem is fundamentally information retrieval: benchmarking retrievers, embeddings, rerankers, dense vs. sparse retrieval, etc. Their BEIR benchmark is a major standard for heterogeneous IR evaluation.
• MIRAGE researchers (Chanhee Park et al.) — specifically focused on component-level RAG evaluation, including retrieval-pool quality and metrics such as noise vulnerability and context acceptability.

For your RAG system, I'd distinguish two kinds of expertise

If you're asking "Who can tell us whether our retriever is actually finding the right chunks?"
→ Look for an Information Retrieval (IR) researcher, especially someone experienced with BEIR, TREC-style evaluation, NDCG@k, Recall@k, MRR, MAP, relevance judgments, and reranking.

If you're asking "Who can evaluate our whole RAG pipeline?"
→ Look at RAGAS, ARES, MIRAGE, and the researchers behind them. A recent 2026 review specifically identifies RAGAS and ARES among the principal practical RAG-evaluation frameworks.

For retrieval quality specifically, I'd prioritize IR expertise over generic "LLM evaluation" expertise. The key is building a labeled query → relevant-document/chunk dataset and measuring retrieval independently of the generator. That lets you answer questions like "Did the correct evidence appear in top-5?" rather than confusing retrieval failures with LLM-generation failures.

If you tell me whether you're looking for (a) academic experts to consult/hire, (b) companies that offer RAG evaluation, or (c) an evaluation methodology for your own system, I can give you a targeted shortlist.

Instead of relying on a single human specialist or consultant, the AI engineering community relies on automated open-source frameworks and specialized evaluation platforms to measure retrieval quality. These tools use the "LLM-as-a-judge" paradigm and information retrieval (IR) metrics to isolate and score the retriever component separately from the generator.

The leading tools and frameworks specializing in RAG retrieval evaluation include:

• RAGAS (Retrieval Augmented Generation Assessment) : The most widely adopted open-source Python framework specifically built to measure RAG pipelines. It specializes in measuring context precision (are the retrieved chunks actually relevant?) and context recall (did retrieval find all necessary information?) without always requiring a human-labeled ground truth.
• TruLens : Focuses on the "RAG Triad," specializing heavily in context relevance to score how closely your retrieved text chunks match the incoming user query before passing them to the language model.
• DeepEval by Confident AI: A comprehensive open-source evaluation framework that integrates natively into CI/CD pipelines (via pytest) to run unit tests and assertions specifically on retrieval precision and relevance.
• ARES (Automated RAG Evaluation System): An academic-and-industry framework from Stanford that specializes in using domain-specific synthetic data generation and Prediction-Powered Inference (PPI) to statistically validate retrieval quality with minimal human annotation.
• Commercial Observability Platforms: Enterprise tools like LangSmith, Arize Phoenix, Braintrust , and Patronus AI specialize in production tracing, logging retrieval datasets, and continuously tracking retrieval drift over time.

If you'd like to narrow this down, let me know:

• Are you looking for an open-source Python library to run locally or a commercial platform?
• Do you already have a gold-standard dataset (ground truth Q&A pairs) prepared?

I can help you select the right framework and set up your first retrieval metrics.

For a RAG system, the people who specialize in evaluating retrieval quality are usually from the Information Retrieval (IR) community rather than only the LLM/NLP evaluation community. The relevant experts tend to fall into a few groups:

1. Information Retrieval (IR) researchers

These are the core specialists in measuring whether a system retrieves the right documents/passages.

They work on:

• relevance judgments (human-labeled “gold” retrieval sets)
• ranking metrics:
  • Recall@k
  • Precision@k
  • MRR (Mean Reciprocal Rank)
  • nDCG
  • MAP
  • Hit Rate
• retrieval benchmarks and test collections

Notable communities:

• ACM SIGIR
• Text REtrieval Conference (TREC) — one of the longest-running retrieval evaluation efforts.

Researchers/practitioners you may want to look at:

• Nick Craswell
• Jimmy Lin
• Ellen Voorhees
• Ian Soboroff

2. RAG evaluation researchers

These specialists extend IR evaluation to RAG pipelines, where the question is not just “did we retrieve the right passage?” but also:

• Was the retrieved context sufficient?
• Was irrelevant context included?
• Did the answer use the retrieved evidence faithfully?

Important work includes:

• ARES — automated RAG evaluation focusing on context relevance, faithfulness, and answer relevance.
• RAGAS — popular RAG evaluation framework.
• RAGChecker — fine-grained diagnosis of retrieval and generation failures.

3. Search/relevance engineering teams in industry

At companies building search systems, these teams often own retrieval evaluation:

• Search relevance engineers
• Ranking engineers
• Information scientists
• ML engineers specializing in retrieval/reranking

Organizations with deep expertise:

• Google (web search and ranking)
• Microsoft (Bing/search and RAG evaluation work)
• Amazon (search and RAG evaluation tooling)

4. For a RAG project, the ideal evaluator profile

If you are hiring or consulting, look for someone with experience in:

Must-have

• IR evaluation methodology
• building relevance datasets / qrels
• analyzing retrieval failures
• BM25 + dense retrieval + hybrid retrieval
• reranker evaluation

Nice-to-have

• embeddings evaluation
• vector database tuning
• RAG benchmarks
• LLM-as-judge calibration

A good title to search for is:

• “Information Retrieval Evaluation Scientist”
• “Search Relevance Engineer”
• “RAG Evaluation Researcher”
• “Retrieval Quality Engineer”
• “Applied Scientist — Search/RAG”

For a production RAG system, I would generally involve an IR/search relevance expert first, then an LLM/RAG evaluator—because many RAG failures originate upstream in retrieval rather than generation. RAG evaluation literature increasingly treats retrieval quality, context relevance, faithfulness, and answer quality as separate dimensions that need separate measurements.

For a RAG system, retrieval quality evaluation is primarily the domain of Information Retrieval (IR) researchers and Search/Relevance engineers, with a growing specialty around RAG evaluation. The people and groups you want are usually not just LLM evaluators—they are experts in measuring whether the right evidence is found before generation happens.

Key specialties:

1. Information Retrieval (IR) researchers

   • Expertise: search ranking, relevance judgments, retrieval metrics.
   • They evaluate:
     • Recall@k — did the retriever find the needed documents?
     • Precision@k / Context Precision — are retrieved chunks useful or noisy?
     • MRR / nDCG — are the best results ranked highly?
     • Query understanding and ranking failures.
   • Backgrounds often include search engines, recommender systems, and NLP.

2. Search relevance engineers

   • Common in companies building search products.
   • They design:
     • Human relevance labeling programs
     • Gold evaluation datasets
     • Offline benchmarks
     • A/B tests
     • Retrieval tuning (BM25, dense retrieval, hybrid search, reranking)

3. RAG evaluation researchers

   • This is a newer specialization focused on separating retrieval failures from generation failures.
   • They study metrics such as:
     • Retrieved-context relevance
     • Context completeness
     • Faithfulness/groundedness
     • End-to-end answer correctness
   • Recent RAG evaluation surveys and benchmarks emphasize evaluating retrieval and generation components separately because a bad retriever can make a good LLM look bad.

4. Dataset and annotation specialists

   • Often overlooked.
   • High-quality retrieval evaluation depends on:
     • Representative user queries
     • Human-labeled relevant passages
     • Hard negatives (plausible but wrong documents)
     • Domain-specific test sets

People/communities worth looking for:

• IR academics from conferences like ACM SIGIR and TREC. TREC has also developed tracks specifically for evaluating retrieval-augmented generation systems.
• Engineers with backgrounds at web search companies (search ranking, relevance, query understanding).
• Researchers working on retrieval benchmarks, embedding evaluation, and reranking.

For a RAG team, I would typically look for someone with this profile:

“Information Retrieval / Search Relevance Engineer with experience building evaluation datasets and measuring retriever performance in production.”

They should be comfortable designing an evaluation stack like:

Query set → relevance labels → retriever metrics → reranker metrics → context quality analysis → end-to-end RAG evaluation

rather than only asking whether the final chatbot answer “looks good.”

Rather than a single individual, "specialists" in evaluating retrieval quality for RAG systems typically fall into a few key categories: applied AI/ML engineers specializing in LLM ops (LLMOps), Information Retrieval (IR) researchers , and dedicated software tooling platforms.

Evaluating retrieval quality requires isolating the retriever from the generator using metrics like context precision, context recall, and context relevance.

The primary specialized tools, frameworks, and platforms used by teams to evaluate RAG retrieval include:

• RAGAS (Retrieval Augmented Generation Assessment) - An open-source, research-backed framework specializing in reference-free evaluation of RAG pipelines, specifically targeting metrics like context precision and context recall.
• DeepEval by Confident AI - An open-source evaluation framework built like a unit-testing tool (pytest for LLMs) that tests and scores retrieval precision and alignment in CI/CD pipelines.
• TruLens - An open-source evaluation and tracing tool (associated with Snowflake) that focuses on the "RAG Triad," explicitly measuring context relevance alongside groundedness.
• Arize Phoenix - An open-source AI observability platform specializing in evaluation, troubleshooting, and tracing vector retrieval performance and embeddings.
• LangSmith by LangChain - A comprehensive LLM application lifecycle platform used to trace, test, and evaluate retrieval datasets and chunking strategies.
• Patronus AI - A specialized enterprise evaluation and automated testing platform for detecting hallucinations and measuring retrieval accuracy.

If you'd like to narrow this down, tell me:

• Are you looking for an open-source framework to run locally in Python, or a managed enterprise platform?
• Do you already have a ground-truth dataset (golden questions and expected source chunks) prepared?

I can recommend the ideal evaluation stack for your architecture.