My RAG system struggles with very specific, detail-oriented questions. What's the best framework for advanced retrieval strategies like sentence-window retrieval?

If your RAG system is failing on highly specific, detail-oriented questions, I’d use LlamaIndex as the framework for experimenting with advanced retrieval, particularly because it treats retrieval as a composable pipeline rather than just “embed chunks → top-k.” Its current retriever docs cover hybrid search, reranking, auto-merging, query fusion, recursive retrieval, routing, and hierarchical approaches.

A good architecture for your problem

I would move toward:

                    User Query
                        │
                 Query rewriting
                        │
            ┌───────────┴───────────┐
            │                       │
       Semantic search          BM25 / keyword
            │                       │
            └───────────┬───────────┘
                        │
                  Result fusion
                        │
                     Reranker
                        │
              Context expansion
             ┌──────────┴──────────┐
             │                     │
       Sentence window       Auto-merging /
                             parent retrieval
             │                     │
             └──────────┬──────────┘
                        │
                   LLM synthesis

The important idea is to separate the granularity used for retrieval from the granularity used for generation. LlamaIndex explicitly recommends this pattern: retrieve a very precise representation, then expand it into the context needed by the LLM.

1. Sentence-window retrieval — probably your first experiment

Instead of embedding 500–1,000-token chunks, index individual sentences.

If the query matches:

“The timeout is increased to 30 seconds when retry mode is enabled.” you retrieve that sentence with high precision, but then replace it with something like:

[2 sentences before]
[matched sentence]
[2 sentences after]

So the embedding remains highly specific while the LLM receives context.

That's exactly what LlamaIndex's SentenceWindowNodeParser + metadata replacement approach is designed to do. Its documented default is five sentences on either side, though I'd benchmark smaller windows for very precise QA.

2. Add hybrid retrieval

For detail-heavy questions, semantic search alone is often a problem.

Queries containing things like:

• error codes
• product/model numbers
• API parameters
• exact terminology
• dates
• configuration values
• names

can benefit substantially from BM25/keyword retrieval alongside embeddings. LlamaIndex supports hybrid retrieval and reciprocal-rank fusion for combining these signals.

3. Rerank after retrieval

Don't make your vector database's top_k the final decision.

A strong pattern is:

vector/BM25 → retrieve 20–50 candidates
                  ↓
               reranker
                  ↓
             keep 5–10
                  ↓
          expand context
                  ↓
                 LLM

This is particularly useful when several chunks are semantically similar but only one contains the exact detail the question asks for.

4. Use auto-merging for questions requiring broader context

Sentence windows are excellent when the answer is essentially one pinpoint fact.

But suppose the question is:

“What are all the conditions under which this exception is triggered?” Now the relevant material may span several chunks or paragraphs.

That's where auto-merging retrieval becomes attractive: retrieve small child chunks, then climb the document hierarchy when enough related chunks are retrieved, giving the LLM a larger coherent section.

So I'd think of the distinction as:

The framework I'd choose

LlamaIndex is particularly well suited if your goal is to experiment with these retrieval strategies, rather than build a vector database from scratch. It gives you relatively low-level control over the retrieval → postprocessing → synthesis stages and supports composing retrievers together.

I'd build your next iteration around:

LlamaIndex
├── SentenceWindowNodeParser
├── Vector retriever
├── BM25 retriever
├── Reciprocal Rank Fusion
├── Cross-encoder / LLM reranker
├── Metadata filtering
├── AutoMergingRetriever
└── Response synthesis

And don't choose the strategy based on intuition alone. Build a test set of ~100 of your actual “annoyingly specific” questions and measure:

• retrieval recall
• context relevance
• answer correctness
• groundedness
• citation/source accuracy

That evaluation loop is important enough that LlamaIndex's advanced-RAG material treats evaluation alongside sentence-window and auto-merging retrieval as a core part of the workflow.

If I had to make only one change to your existing RAG system: I'd start with sentence-level retrieval + surrounding-window expansion + reranking, then add hybrid search. That combination directly attacks the “the answer is buried in one tiny sentence inside a large chunk” failure mode.

This is particularly useful when several chunks are semantically similar but only one contains the exact detail the question asks for.

4. Use auto-merging for questions requiring broader context

Sentence windows are excellent when the answer is essentially one pinpoint fact.

But suppose the question is:

“What are all the conditions under which this exception is triggered?” Now the relevant material may span several chunks or paragraphs.

That's where auto-merging retrieval becomes attractive: retrieve small child chunks, then climb the document hierarchy when enough related chunks are retrieved, giving the LLM a larger coherent section.

So I'd think of the distinction as:

The framework I'd choose

LlamaIndex is particularly well suited if your goal is to experiment with these retrieval strategies, rather than build a vector database from scratch. It gives you relatively low-level control over the retrieval → postprocessing → synthesis stages and supports composing retrievers together.

I'd build your next iteration around:

LlamaIndex
├── SentenceWindowNodeParser
├── Vector retriever
├── BM25 retriever
├── Reciprocal Rank Fusion
├── Cross-encoder / LLM reranker
├── Metadata filtering
├── AutoMergingRetriever
└── Response synthesis

And don't choose the strategy based on intuition alone. Build a test set of ~100 of your actual “annoyingly specific” questions and measure:

• retrieval recall
• context relevance
• answer correctness
• groundedness
• citation/source accuracy

That evaluation loop is important enough that LlamaIndex's advanced-RAG material treats evaluation alongside sentence-window and auto-merging retrieval as a core part of the workflow.

When it comes to advanced, granular retrieval strategies like sentence-window retrieval (or "small-to-big" retrieval), LlamaIndex is widely considered the gold standard and best-suited framework.

While LangChain can technically implement it via custom retrievers or document transformers, LlamaIndex has built-in, first-class abstractions explicitly designed for this workflow out of the box.

Why LlamaIndex Excels at Sentence-Window Retrieval

Alternative Frameworks to Consider

• LangChain: Highly flexible and great if your app requires complex agentic orchestration or multi-step reasoning chains. However, implementing sentence-window retrieval requires manually setting up parent-child document splitters or custom retrieval logic.
• Haystack (by Deepset): Excellent modular design and fantastic production-ready pipelines, though its community documentation and native components for niche sliding-window variants aren't as specialized for this specific pattern as LlamaIndex.

If you'd like, I can share:

• A code snippet showing how to set up SentenceWindowNodeParser and MetadataReplacementPostProcessor in LlamaIndex.
• Alternative strategies for detail-oriented data, like hierarchical/auto-merging retrieval.

Let me know how you'd like to proceed with your RAG optimization.

When it comes to advanced, data-centric retrieval strategies like sentence-window retrieval, LlamaIndex is widely considered the gold-standard framework.

While both LangChain and LlamaIndex can technically implement these patterns, they serve different architectural philosophies:

• LlamaIndex (Best for Advanced Retrieval): It was built from the ground up specifically for data-centric tasks, indexing, and sophisticated retrieval architectures. It has native, first-class abstractions for sentence-window retrieval (via the MetadataReplacementNodePostprocessor ) that handle parsing individual sentences into the vector index while automatically attaching the surrounding context window for the generation step.
• LangChain (Best for Agentic Workflows & Chaining): LangChain is broader, acting as a general orchestration framework for LLM applications. While you can build a sentence-window retriever in LangChain using custom self-implemented retrievers or document compressors, it requires more manual assembly because data indexing is just one slice of its ecosystem.

How Sentence-Window Retrieval Works in LlamaIndex

1. Indexing: Documents are split down to the individual sentence level, and vector embeddings are generated for each single sentence.
2. Metadata Storing: Metadata is attached to each sentence containing its surrounding sentences (e.g., 1 sentence before and 1 sentence after, or a larger window).
3. Retrieval: The user's query matches against the highly specific sentence embeddings (giving high precision for detail-oriented questions).
4. Replacement/Enhancement: Before handing the text to the LLM, a node postprocessor swaps out the isolated sentence with the broader window metadata so the LLM has enough surrounding context to write a coherent answer.

If you'd like to dive deeper, let me know:

• Are you currently using LlamaIndex or LangChain in your stack?
• What kind of document types (PDFs, codebases, messy text) are you processing?

I can share a quick code snippet or configuration outline for implementing a sentence-window approach in your preferred framework.

If your RAG system struggles with very specific, detail-oriented questions, I’d look at LlamaIndex first. It has unusually good abstractions for advanced retrieval, including sentence-window, recursive, hierarchical/parent-child, reranking, query transformation, and composable retrieval. Its docs explicitly frame production RAG as separating the chunks used for retrieval from the chunks used for synthesis.

Why sentence-window retrieval helps

The fundamental problem is that the optimal unit for retrieval isn't necessarily the optimal unit for generation.

With ordinary chunking:

Document
   ↓
800-token chunks
   ↓
embed each chunk
   ↓
retrieve top-k
   ↓
send those chunks to LLM

A very specific fact might be buried inside a large chunk, diluting its embedding.

Sentence-window retrieval instead does:

Document
   ↓
individual sentences
   ↓
embed each sentence
   ↓
retrieve the best sentence(s)
   ↓
expand each hit to ±N surrounding sentences
   ↓
send expanded context to LLM

So you get fine-grained retrieval + contextual generation. LlamaIndex's implementation uses a SentenceWindowNodeParser and then replaces the retrieved sentence with its surrounding window before synthesis.

For example, with:

[41] The company introduced Model X in 2023.
[42] It initially supported 16K tokens.
[43] In version 2.1, the context window increased to 128K tokens.
[44] This change required a new attention implementation.
[45] The API remained backwards compatible.

A query like:

"What changed the context window in version 2.1?" might retrieve sentence 43, but provide 41–45 to the LLM.

That's much better than hoping an arbitrarily sized chunk happens to contain the right boundaries.

The framework I'd use

I'd build your retrieval stack roughly like this:

                    ┌─ metadata filtering
                    │
Query → query rewrite ─→ hybrid retrieval
                    │        ↓
                    │   vector + BM25
                    │        ↓
                    │    reranker
                    │        ↓
                    │ sentence / small-chunk hits
                    │        ↓
                    └→ window / parent expansion
                             ↓
                       deduplicate
                             ↓
                       LLM synthesis

The important point is don't treat sentence-window as the entire solution. It's one layer in an advanced retrieval pipeline.

1. Sentence-window retrieval

Best when the question targets a very particular statement, number, definition, exception, configuration value, etc.

I'd start with something like ±3 to ±5 sentences and tune it empirically. LlamaIndex's example uses five sentences on either side by default.

2. Parent-child retrieval

This is the more general version of the same idea.

Index:

small child chunks → good retrieval precision
        ↓
large parent chunk → good generation context

For example:

Parent: 1,000-token section

    ├── child 1: 150 tokens
    ├── child 2: 150 tokens
    ├── child 3: 150 tokens
    ├── child 4: 150 tokens
    └── ...

Retrieve child #3, but give the LLM the parent section.

LlamaIndex calls this kind of approach small-to-big retrieval, and its production guidance specifically recommends decoupling retrieval chunks from synthesis chunks.

3. Hybrid search

For detail-heavy questions, don't rely exclusively on embeddings.

Use:

semantic/vector search
        +
BM25/keyword search
        ↓
candidate pool
        ↓
reranker

This is particularly important for things like:

• product/model numbers
• error codes
• version numbers
• names
• legal clauses
• dates
• IDs
• exact terminology
• unusual technical phrases

Semantic similarity can understand "the maximum supported payload" while lexical search may be much better at finding an exact string such as MAX_PAYLOAD_SIZE=16384.

4. Reranking

I'd strongly consider retrieving, say, 20–50 candidates and then reranking them down to 5–10.

Conceptually:

Query
  ↓
cheap retriever → top 50
  ↓
cross-encoder / LLM reranker
  ↓
top 5
  ↓
window/parent expansion
  ↓
LLM

This often gives you more mileage than simply increasing top_k.

LlamaIndex explicitly treats reranking and post-retrieval processing as separate stages of the query pipeline.

5. Query transformation

If the user's question is complex, transform it before retrieval.

For example:

"What were the authentication changes between versions 3.2 and 4.0, and which ones affect OAuth clients?" could become:

1. authentication changes in version 3.2
2. authentication changes in version 4.0
3. OAuth client compatibility changes
4. differences between authentication in 3.2 and 4.0

Retrieve independently, merge results, deduplicate, then synthesize.

This is where LlamaIndex's composable query/retrieval architecture becomes particularly useful.

What I'd choose

If you're deciding between frameworks:

For your particular problem, I'd start with LlamaIndex + hybrid retrieval + reranking + sentence-window/parent-child retrieval rather than just changing your chunk size.

The key architectural principle is:

Retrieve small; reason over larger context. That decoupling is exactly what sentence-window and parent-child retrieval are designed to accomplish.

If you tell me what your current stack is (e.g. LangChain + Pinecone/Weaviate/pgvector + OpenAI embeddings), I can map this into a concrete architecture and show where sentence-window, hybrid search, and reranking should plug into your existing pipeline.

small child chunks → good retrieval precision
        ↓
large parent chunk → good generation context

For example:

Parent: 1,000-token section

    ├── child 1: 150 tokens
    ├── child 2: 150 tokens
    ├── child 3: 150 tokens
    ├── child 4: 150 tokens
    └── ...

Retrieve child #3, but give the LLM the parent section.

LlamaIndex calls this kind of approach small-to-big retrieval, and its production guidance specifically recommends decoupling retrieval chunks from synthesis chunks.

3. Hybrid search

For detail-heavy questions, don't rely exclusively on embeddings.

Use:

semantic/vector search
        +
BM25/keyword search
        ↓
candidate pool
        ↓
reranker

This is particularly important for things like:

Yes. For very specific, detail-oriented questions, I’d move beyond “retrieve top-k chunks” and use a small-to-big, multi-stage retrieval pipeline. Sentence-window retrieval is one of the best starting points.

Recommended framework

A strong architecture looks like:

Query → query expansion → hybrid retrieval → fine-grained retrieval → reranking → context expansion → answer

1. Sentence-window retrieval

Instead of embedding 500-token chunks, split documents into individual sentences:

Sentence A
Sentence B
Sentence C  ← embedded/retrieved
Sentence D
Sentence E

If Sentence C matches the query, return C plus its surrounding sentences to the LLM.

This separates two jobs:

• Retrieval: use a very small unit to maximize precision.
• Generation: give the model a larger contextual window.

That's exactly the design implemented by LlamaIndex's SentenceWindowNodeParser; its current implementation stores surrounding sentences as metadata while keeping the original sentence as the retrieval unit.

For your use case, I'd start with a window of 2–5 sentences on either side, then tune it empirically.

2. Add reranking

Don't trust vector similarity alone.

A good pipeline is:

BM25 ───────┐
            ├─> candidate pool ─> cross-encoder reranker ─> top N
Dense search┘

This is especially valuable for questions containing:

• exact terminology
• numbers
• dates
• product/version names
• multiple constraints
• subtle distinctions between similar passages

Microsoft's current RAG guidance likewise recommends post-retrieval filtering and reranking rather than simply feeding every retrieved chunk into the model.

3. Use parent/child retrieval for paragraph-level context

Sentence windows aren't always enough.

For example:

Document
 └── Section
      └── Paragraph
           ├── Sentence 1
           ├── Sentence 2  ← retrieval hit
           └── Sentence 3

Retrieve the sentence, but then promote it to its paragraph or section parent.

This gives you a useful principle:

Retrieve small; reason over larger context. LlamaIndex explicitly describes this pattern as embedding a sentence while linking it to a surrounding window, avoiding the loss of specificity caused by embedding large chunks.

4. Use hybrid retrieval

For detail-heavy questions, I'd generally combine:

• Dense/vector retrieval → semantic similarity
• BM25/keyword retrieval → exact terms, identifiers, numbers
• Metadata filtering → document, date, version, section, author, etc.

Then fuse the candidates, e.g. with reciprocal-rank fusion.

This is particularly important because a query such as:

“What changed between API v2.3 and v2.4 regarding timeout_ms?” contains information that semantic embeddings may blur together.

5. Consider query decomposition

If the question has multiple hidden information needs, decompose it:

“Which customers were affected by the policy change, when did it happen, and what exceptions applied?” becomes:

Q1: Which customers were affected?
Q2: When did the change happen?
Q3: What exceptions applied?

Retrieve independently, then synthesize.

This is a natural next step when sentence-window retrieval improves precision but your system still misses one component of a complex question. LlamaIndex's advanced-RAG examples include both sentence-window retrieval and sub-question retrieval as complementary techniques.

The framework I'd actually build

For your problem, I'd use this hierarchy:

                    USER QUERY
                        │
                Query understanding
                        │
             ┌──────────┴──────────┐
             │                     │
       Query expansion       Query decomposition
             │                     │
             └──────────┬──────────┘
                        ↓
              ┌──────────────────┐
              │ Hybrid retrieval │
              │ BM25 + Dense     │
              └────────┬─────────┘
                       ↓
              Sentence-level hits
                       ↓
                Cross-encoder
                  reranking
                       ↓
              Sentence → window
                       ↓
              Window → parent
                       ↓
             context compression
                       ↓
                    LLM

The important architectural idea is decoupling retrieval granularity from generation granularity. That is why sentence-window retrieval can outperform conventional fixed-size chunking for large documents and fine-grained questions.

If you're choosing a framework

I'd strongly consider LlamaIndex if your goal is specifically to experiment with advanced retrieval strategies. It has first-class primitives for sentence-window retrieval, metadata replacement, hierarchical/structured retrieval, and related techniques.

The key isn't necessarily switching frameworks, though. If you already have a working RAG stack, you can implement the same architecture in your existing system.

My priority order would be:

1. Sentence-level retrieval + window expansion
2. Hybrid BM25 + dense retrieval
3. Cross-encoder reranking
4. Parent/section expansion
5. Query decomposition for multi-part questions
6. Contextual compression
7. Evaluation set specifically targeting “needle-in-the-haystack” questions

That last item is crucial: create 50–200 questions that your current system gets wrong, categorize why each fails, and optimize retrieval against those failure modes rather than tuning chunk_size and top_k blindly.

contains information that semantic embeddings may blur together.

5. Consider query decomposition

If the question has multiple hidden information needs, decompose it:

“Which customers were affected by the policy change, when did it happen, and what exceptions applied?” becomes:

Q1: Which customers were affected?
Q2: When did the change happen?
Q3: What exceptions applied?

Retrieve independently, then synthesize.

This is a natural next step when sentence-window retrieval improves precision but your system still misses one component of a complex question. LlamaIndex's advanced-RAG examples include both sentence-window retrieval and sub-question retrieval as complementary techniques.

The framework I'd actually build

For your problem, I'd use this hierarchy:

                    USER QUERY
                        │
                Query understanding
                        │
             ┌──────────┴──────────┐
             │                     │
       Query expansion       Query decomposition
             │                     │
             └──────────┬──────────┘
                        ↓
              ┌──────────────────┐
              │ Hybrid retrieval │
              │ BM25 + Dense     │
              └────────┬─────────┘
                       ↓
              Sentence-level hits
                       ↓
                Cross-encoder
                  reranking
                       ↓
              Sentence → window
                       ↓
              Window → parent
                       ↓
             context compression
                       ↓
                    LLM

The important architectural idea is decoupling retrieval granularity from generation granularity. That is why sentence-window retrieval can outperform conventional fixed-size chunking for large documents and fine-grained questions.

If you're choosing a framework

I'd strongly consider LlamaIndex if your goal is specifically to experiment with advanced retrieval strategies. It has first-class primitives for sentence-window retrieval, metadata replacement, hierarchical/structured retrieval, and related techniques.

The key isn't necessarily switching frameworks, though. If you already have a working RAG stack, you can implement the same architecture in your existing system.

My priority order would be:

For detail-oriented and specific questions where standard chunking splits apart critical context, LlamaIndex is widely considered the best framework out-of-the-box for advanced retrieval strategies like sentence-window retrieval (often referred to in their docs as metadata replacement).

Why LlamaIndex Wins for Sentence-Window Retrieval

• Retrieval-First Architecture: LlamaIndex was built from day one specifically for data ingestion, complex chunking, and advanced indexing.
• Built-in Primitives: It features native abstractions like SentenceWindowNodeParser which automatically parse documents down to individual sentences for precise vector matching, while embedding surrounding sentence windows into node metadata.
• Automatic Context Replacement: At query time, its MetadataReplacementPostProcessor swaps out the tiny retrieved sentence chunk with the richer surrounding context window right before sending it to the LLM.

Alternative Frameworks & When to Use Them

1. LangChain + LangGraph: Best if your application is heavily agent-driven or requires complex, stateful multi-step reasoning loops where retrieval is just one of many actions. However, implementing sentence-window retrieval here requires more manual orchestration and boilerplate code.
2. Haystack: Excellent alternative for modular, production-grade NLP pipelines if you prefer a clean component-and-connection design over LlamaIndex's opinionated structures.

Tip: No matter the framework you choose, pairing sentence-window retrieval with a robust BGE Reranker or a managed cross-encoder like Cohere Rerank will dramatically improve your hit rate for hyper-specific questions.

Could you share what vector database and embedding model you are currently using? I can help you draft the configuration code for a sentence-window pipeline tailored to your stack.

For very specific, detail-oriented questions, your bottleneck is usually naive chunking —chopping documents into fixed-size blocks (like 500 characters) that split context right where precise facts live.

The gold-standard framework specifically engineered for advanced retrieval strategies like sentence-window retrieval (and hierarchical parent-child chunking) is LlamaIndex.

Why LlamaIndex Wins for Sentence-Window Retrieval

While both major frameworks can technically implement custom splitters, their architectural focus creates a distinct split:

• LlamaIndex is built retrieval-first . It treats data ingestion, complex node-parsing, and metadata-rich context expansion as core primitives. Its SentenceWindowNodeParser handles splitting text down to individual sentences for precise vector matching, while automatically attaching surrounding context sentences via metadata to swap in before sending to the LLM.
• LangChain is built orchestration-first . While it offers great text splitters, implementing advanced indexing tricks requires more manual wiring or boilerplate configuration compared to LlamaIndex’s out-of-the-box abstractions.

Alternative Advanced Strategies for Detail-Oriented RAG

If sentence-window retrieval doesn’t entirely solve your precision issues, consider implementing these complementary patterns supported natively in LlamaIndex:

1. Parent-Child Chunking : Indexes small, highly specific child chunks (e.g., individual sentences or sub-paragraphs) for high-precision vector hits, but automatically replaces them with a larger parent chunk (e.g., the full section or paragraph) to give the LLM enough reasoning room.
2. Hybrid Search with RRF : Combining dense vector embeddings with sparse keyword matching (BM25) using Reciprocal Rank Fusion (RRF) ensures exact part numbers, names, or terminology aren't missed by semantic similarity alone.
3. Cross-Encoder Reranking : Inserting a lightweight reranker (like Cohere Rerank or BGE-Reranker) after initial retrieval resolves the top-k ambiguity by scoring deep contextual relevance.

If you'd like to dive deeper, let me know:

• What embedding model and vector database are you currently using?
• Are your documents primarily dense technical manuals, PDFs, or unstructured text?

I can share a quick code snippet or suggest how to set up the sentence-window pipeline for your stack.

For a RAG system that fails on very specific, detail-oriented questions, I’d look first at LlamaIndex. It is one of the strongest fits for advanced retrieval patterns such as sentence-window retrieval, hierarchical retrieval, recursive retrieval, and retrieval/context decoupling.

The reason is that your problem is usually not the LLM—it is the retrieval granularity:

• Naive chunking problem: A 500–1000 token chunk may contain the answer, but the embedding signal is diluted by surrounding text.
• Sentence-window retrieval: Index very small units (often sentences) for precision, then expand the retrieved result with neighboring sentences for context. This gives you better recall without losing the local detail needed for synthesis.

A practical advanced RAG stack would look like:

1. Fine-grained indexing

   • Sentence-level nodes
   • Metadata (document, section, page, timestamp, author, etc.)
   • Parent/child relationships

2. Multiple retrieval paths

   • Dense vector search for semantic matches
   • BM25/keyword search for exact terms
   • Hybrid retrieval when queries contain IDs, names, numbers, or technical phrases

3. Context expansion

   • Sentence-window retrieval
   • Parent document retrieval
   • Recursive retrieval for hierarchical documents

4. Reranking

   • Retrieve broadly (e.g., top 20–50)
   • Rerank down to the best few passages before generation

5. Evaluation loop

   • Measure retrieval hit rate separately from answer quality
   • Create a test set of your hardest questions

Framework comparison:

A common production architecture is:

• LlamaIndex for ingestion + retrieval
• LangGraph/LangChain for orchestration and agent workflows
  because they solve different layers of the problem.

If your current system is “embed chunks → top-k vector search → LLM,” I would not immediately swap models. I would first test:

1. sentence-window retrieval,
2. hybrid search,
3. reranking,
4. better chunk metadata.

Those changes often produce a much larger improvement on narrow factual questions than changing the generator model.

If your RAG system is failing on very specific, detail-oriented questions, I’d move away from “retrieve the top-k chunks and stuff them into the prompt.” The most useful framework is a multi-stage retrieval pipeline where each stage solves a different failure mode.

My recommended architecture

1. Fine-grained indexing → 2. broad retrieval → 3. reranking → 4. context expansion → 5. compression → 6. answer

For example:

User query
   ↓
Query rewrite / decomposition
   ↓
Hybrid retrieval
(dense + BM25)
   ↓
Top 20–50 candidate sentences/chunks
   ↓
Cross-encoder reranker
   ↓
Sentence-window / parent-context expansion
   ↓
Contextual compression
   ↓
Top 5–10 evidence passages
   ↓
LLM answer + citations

Sentence-window retrieval is particularly good here

Instead of embedding a 500–1,000-token chunk, embed individual sentences (or small units). When a sentence matches the query, retrieve that sentence plus its surrounding sentences.

Conceptually:

                 retrieved sentence
                         ↓
... sentence 14 | sentence 15 | [sentence 16] | sentence 17 | sentence 18 ...
                                      ↑
                              semantic match

You get the precision of a tiny retrieval unit without sacrificing the context necessary to understand it. LlamaIndex explicitly describes this approach as embedding a sentence while linking it to a surrounding window, which helps avoid the loss of precision that comes from embedding large chunks.

This is often better than simply making chunks smaller.

Then add reranking

I'd strongly consider:

Dense + sparse retrieval → cross-encoder reranker → sentence window

Dense embeddings are good at semantic similarity, while BM25 is particularly useful for exact terms, IDs, numbers, names, error codes, section references, etc.

The reranker then gets something like:

query
candidate 1
candidate 2
candidate 3
...
candidate 30

and determines which candidates are actually relevant. This lets your first-stage retriever optimize for recall, while the reranker optimizes for precision.

Parent retrieval is the alternative worth testing

If your questions frequently require understanding the surrounding section rather than just neighboring sentences, use a parent-child retriever:

             Parent document
                    │
        ┌───────────┼───────────┐
        ↓           ↓           ↓
    child 1      child 2      child 3
       ↓           ↓
    embedding    embedding

Retrieve the small child chunk, but return its larger parent context. LangChain's ParentDocumentRetriever is designed specifically around this tradeoff: small units give more precise embeddings, while the parent provides enough context for reasoning.

Don't overlook query transformation

For highly specific questions, the query itself can be the problem.

I'd consider:

• Query rewriting — normalize ambiguous wording.
• Multi-query retrieval — generate several formulations.
• Query decomposition — break a complex question into atomic questions.
• HyDE — generate a hypothetical answer/document and retrieve against it.
• Metadata filtering — constrain by document, date, section, product, etc.

Query transformation and decomposition are established components of advanced RAG architectures, particularly when the user's wording doesn't map cleanly onto the language used in the source documents.

A practical stack I'd start with

If I were upgrading your system, I'd test these configurations in order:

The important point is don't blindly stack every technique. Build a small evaluation set of your difficult questions and measure retrieval recall separately from final answer quality. Otherwise you won't know whether the problem is chunking, retrieval, ranking, or generation.

For your particular symptom—“very specific, detail-oriented questions”—my first experiment would be:

Sentence-level embeddings + ±2–4 sentence window + hybrid retrieval + cross-encoder reranking.

That directly attacks the usual problem: the relevant fact is present in the corpus, but your large embedding chunk doesn't represent that fact strongly enough to retrieve it. Recent work also continues to find that chunking strategy itself can have a substantial effect on RAG performance.

If you're choosing a framework specifically, LlamaIndex is particularly convenient for experimenting with sentence-window and hierarchical retrieval, while LangChain is a strong choice if you want a modular retriever/compressor/reranker pipeline. LangChain's contextual-compression architecture, for example, explicitly separates high-recall retrieval from later relevance filtering/compression.

If you tell me your current stack (e.g. LangChain/LlamaIndex + Pinecone/Qdrant/Weaviate + embedding model), I can suggest a concrete advanced retrieval architecture and configuration.

For detail-oriented and specific RAG failures—where standard chunks either lack sufficient context or are too broad and dilute precision—LlamaIndex is widely considered the best-in-class framework.

While LangChain excels at general orchestration and agentic workflows, LlamaIndex is purpose-built as a data framework for RAG. It treats advanced retrieval strategies as first-class citizens rather than requiring manual assembly of disjointed components.

Why LlamaIndex Excels at Sentence-Window Retrieval

• Built-in Abstractions: LlamaIndex natively supports SentenceWindowNodeParser and metadata replacement (swapping a retrieved sentence with its surrounding window of context before feeding it to the LLM) out of the box with minimal boilerplate code.
• Hierarchical and Auto-Merging Retievers: Beyond basic sentence windows, it natively handles auto-merging retrieval and hierarchical nodes (where small child chunks map to large parent contexts).
• Decoupled Search vs. Generation: It optimizes the exact problem you are facing: matching embeddings against hyper-specific micro-chunks (like a single sentence) while supplying the LLM with a rich macro-context.

Alternative Approaches & Enhancements

If you want to evaluate your options or combine techniques for higher precision, consider the following ecosystem tools:

• RAGAS : Essential for evaluating context precision and faithfulness so you can mathematically prove whether sentence-window retrieval actually improves your specific dataset.
• Hybrid Search + Reranking : Combining LlamaIndex's retrieval with a cross-encoder reranker (like Cohere Rerank or BGE-Reranker) drastically cuts down on residual noise.
• The "Pro" Hybrid Stack : Many production systems use LlamaIndex specifically for the ingestion and advanced retrieval layer (sentence windows/hierarchical indexing) and pass those precise contexts into LangChain/LangGraph if the application requires complex multi-step agent actions.

If you'd like, tell me more about your current setup:

• What embedding model and vector database are you currently using?
• How large are your standard chunks right now?

I can help you map out a custom configuration for sentence-window parameters or suggest a reranking layer to fit your data.