Data as of Aug 25, 2026 · Based on 38 AI responses from ChatGPT Search and Google AI Mode · See how Parse measures this
Your brand can be here too.
Source | Citation share |
|---|---|
 Best Practices for Implementing RAG Systems in Production | Unstructuredhttps://unstructured.io/insights/rag-systems-best-practices-unstructured-data-pipeline | 2.2% |
Unstructured Data Platform for GenAIhttps://unstructured.io/ | 2.0% |
What Matters for LLM Data Ingestion and Preprocessinghttps://unstructured.io/blog/understanding-what-matters-for-llm-ingestion-and-preprocessing | 2.0% |
Ingesting Unstructured Data at Scale: Best Practices | Unstructuredhttps://unstructured.io/insights/unstructured-data-ingestion-at-scale-enterprise-best-practices | 2.0% |
 Build an unstructured data pipeline for RAG | Databricks on AWShttps://docs.databricks.com/aws/en/agents/tutorials/ai-cookbook/quality-data-pipeline-rag | 1.8% |
Common Challenges in RAG and How to Solve Them in Production | Unstructuredhttps://unstructured.io/insights/rag-pipeline-challenges-from-data-ingestion-to-retrieval | 1.8% |
Best AI for Unstructured Data - LlamaIndexhttps://www.llamaindex.ai/insights/best-ai-for-unstructured-data | 1.8% |
Build an unstructured data pipeline for RAG | Databricks on AWShttps://docs.databricks.com/aws/en/generative-ai/tutorials/ai-cookbook/quality-data-pipeline-rag | 1.3% |
 Top 10 document parsing services for RAG pipelines and LLM ...https://vstorm.co/llamaindex/top-10-document-parsing-services-for-rag-pipelines-and-llm-applications/ | 1.1% |
 From PDF to RAG-Ready: Evaluating Document Conversion Frameworks for Domain-Specific Question Answeringhttps://arxiv.org/abs/2604.04948 | 0.9% |
The "best" pipeline for turning unstructured documents (PDFs, scans, Word docs, slides) into LLM-ready data is no longer a static, brittle script running regex or basic OCR. The modern gold standard is an Agentic, Layout-Aware Parsing Pipeline that preserves semantic structure (tables, headings, reading order) rather than just dumping raw text.
A production-grade architecture typically follows a 5-stage blueprint:
- Top tools include [LlamaParse](https://google.com/goto?url=CAESYQHrOzAVYRZe_5vY5caIbiKfAr5VrkUjTlVyvxwFoDglQ2OoR__SOATZAKEyoIqIvKro7mp1zfFyC_Kj_8dJqMDfDZO3KTosJtqnLl0RGqaE2edC3SiI8pTUNfWZ4utaSWo) (purpose-built for LlamaIndex workflows and markdown conversion), Unstructured.io (great for open-source/enterprise orchestration), and [Reducto](https://google.com/goto?url=CAESXAHrOzAVaSMLLqzQ6DtlZTHtN5KB2kdxlRbQMuPndsqiDjkmSNa3-WLqxaGvk9fbMJuN8XS65fgXA9P5VISvt-vBOHLutowtKYtYQt20jEm6qPrny7mToo0AZZP0) for complex document layouts.[](https://google.com/goto?url=CAESXAHrOzAVaSMLLqzQ6DtlZTHtN5KB2kdxlRbQMuPndsqiDjkmSNa3-WLqxaGvk9fbMJuN8XS65fgXA9P5VISvt-vBOHLutowtKYtYQt20jEm6qPrny7mToo0AZZP0) [[1]](https://google.com/goto?url=CAESXAHrOzAVaSMLLqzQ6DtlZTHtN5KB2kdxlRbQMuPndsqiDjkmSNa3-WLqxaGvk9fbMJuN8XS65fgXA9P5VISvt-vBOHLutowtKYtYQt20jEm6qPrny7mToo0AZZP0)[[2]](https://google.com/goto?url=CAESYQHrOzAVYRZe_5vY5caIbiKfAr5VrkUjTlVyvxwFoDglQ2OoR__SOATZAKEyoIqIvKro7mp1zfFyC_Kj_8dJqMDfDZO3KTosJtqnLl0RGqaE2edC3SiI8pTUNfWZ4utaSWo)[[3]](https://google.com/goto?url=CAESdgHrOzAV5S1pKNEX-MNqFv3QHRtFk4PGcfaM0ZwxJR5NAPs2BMWF3MODj7kGDYPVJBFEWQj_bIQfEjOeUSudVyxPymMa3JnEVGc4OxRLfrYaGfZxfe4jjX1Sre4W1Ff4KpJJw2OhJQ-myWr4qCjTriIeDtchNEc)[[4]](https://google.com/goto?url=CAEScwHrOzAVWg5xn7FlmxjA8FgqqPqwPY-GXc9Ah9wbvr6ATx1ArjTbOVjzjHmhYoqrQOF3Pi6D90CHwmC3jskLsokRQx4t-6yYP413brXX0U1h8DhoUi8n-FR70R6LIc6SlczaLD0xN6t7KWrirPsd8d6wBag)[[5]](https://google.com/goto?url=CAESZQHrOzAVPMM1CyGdc0nIWyaLgeR973-vD75XHRwZpSxWAic8O-iDMxJjHFKM062nco2t2ccncUp3TTVo3tyFXAlpEPqo3wuAXdY4aOtTT0op-2U7RgT8BPxmjVZnCl0D6_R75b4j)
If you want to tailor this, tell me:
I can give you a specific code template or tool stack recommendation.
A strong unstructured-document → LLM-ready data pipeline is less about “putting documents into a vector database” and more about preserving meaning, structure, provenance, and retrieval quality. A production-grade pipeline usually looks like this:
Raw documents
↓
Ingestion + provenance tracking
↓
Parsing + layout extraction
↓
Cleaning + normalization
↓
Structure-aware chunking
↓
Metadata enrichment
↓
Embeddings + indexing
↓
Retrieval optimization
↓
Evaluation + monitoring
↓
LLM/RAG application
Goal: Never lose traceability.
Store:
A common mistake is immediately converting everything into plain text. Keep the raw artifact so you can reprocess when parsers or chunking strategies improve.
Goal: Convert files into structured elements, not just text blobs.
Good output:
{
"type": "section",
"title": "Refund Policy",
"content": "...",
"page": 12,
"source": "policy.pdf"
}
Handle each format differently:
Flattening a complex PDF into a single string is one of the fastest ways to damage retrieval quality.
Typical transformations:
Do not aggressively summarize at this stage. You want a faithful representation of the source.
Chunking is probably the highest-impact step.
Weak:
Every 500 tokens → new chunk
Better:
Document
├── Chapter
│ ├── Section
│ │ ├── Paragraph group
│ │ └── Table
Useful chunk strategies:
| Document type | Chunking approach |
|---|---|
| Policies/manuals | Heading-aware |
| Legal docs | Section/clause-aware |
| Research papers | Section + citation-aware |
| Support docs | FAQ/question-aware |
| Code | Function/class-aware |
Each chunk should be:
Also attach context:
{
"text": "Employees receive 20 days PTO...",
"metadata": {
"document": "employee_handbook",
"section": "Benefits > PTO",
"page": 42
}
}
Structure-aware chunking generally outperforms arbitrary splitting because retrieval depends on meaningful boundaries.
Metadata makes retrieval much better than embeddings alone.
Useful fields:
{
"title": "...",
"author": "...",
"department": "HR",
"date": "2026-01-10",
"document_type": "policy",
"security_level": "internal",
"entities": ["Acme Corp", "California"]
}
Use metadata for:
Example:
Search only HR policies from 2025 onward that the user can access.
Typical storage model:
Chunk
├── Text
├── Embedding vector
├── Metadata
└── Source pointer
Use:
For enterprise systems, hybrid retrieval often works better because users ask questions containing both concepts and exact identifiers (product IDs, policy numbers, error codes).
A mature pipeline often adds:
First retrieve:
Top 100 chunks
Then rerank:
Best 5 chunks → LLM context
Convert:
"How much time off do I get?" into:
"Find employee paid time off policy entitlement rules."
Store:
Measure retrieval separately from generation.
Track:
Many “LLM accuracy” problems are actually document-processing problems upstream.
A common architecture:
Sources
├── SharePoint
├── Google Drive
├── S3
├── Websites
└── Databases
↓
Document processors
├── PDF parser
├── OCR
├── HTML extractor
└── DOCX parser
↓
Canonical document store
(JSON + metadata)
↓
Chunking pipeline
↓
Embedding service
↓
Search layer
├── Vector DB
├── Keyword index
└── Metadata filters
↓
RAG / Agent application
A good rule of thumb: make the LLM see the same information a careful human would see when opening the document, but in a form optimized for search and context assembly.
A common architecture:
Sources
├── SharePoint
├── Google Drive
├── S3
├── Websites
└── Databases
↓
Document processors
├── PDF parser
├── OCR
├── HTML extractor
└── DOCX parser
↓
Canonical document store
(JSON + metadata)
↓
Chunking pipeline
↓
Embedding service
↓
Search layer
├── Vector DB
├── Keyword index
└── Metadata filters
↓
RAG / Agent application
Store:
Measure retrieval separately from generation.
Track:
Many “LLM accuracy” problems are actually document-processing problems upstream.
A common architecture:
The best pipeline is not “PDF → text → embeddings.” It’s a format-aware, lossless-ish transformation pipeline that preserves document structure and provenance all the way through retrieval.
A strong production architecture looks like this:
Raw documents
↓
1. Ingest + fingerprint
↓
2. Format detection / routing
↓
3. Parse + OCR + layout analysis
↓
4. Structural normalization
↓
5. Clean + validate
↓
6. Semantic chunking
↓
7. Metadata + provenance enrichment
↓
8. Embeddings + lexical index
↓
9. Retrieval / reranking
↓
10. LLM
Put every source file into immutable object storage and assign:
document_idThe hash lets you make ingestion idempotent: unchanged documents don't get processed again.
Keep the original file forever if possible. It is your audit/debugging escape hatch.
Don't use one parser for everything.
For example:
Modern document-processing systems increasingly use different extraction strategies depending on document complexity. For example, Unstructured's current pipeline can route PDF pages differently depending on whether they contain embedded text, images, or tables.
The important principle is:
Extract structure before destroying structure. Don't flatten a 40-page report into one giant string and try to reconstruct its hierarchy afterward.
This is probably the most important architectural decision.
Instead of making your downstream pipeline consume arbitrary parser output, normalize everything into something like:
{
"document_id": "abc123",
"elements": [
{
"type": "heading",
"text": "3. Pricing",
"level": 2,
"page": 7
},
{
"type": "paragraph",
"text": "The enterprise plan...",
"page": 7
},
{
"type": "table",
"html": "...",
"page": 8
}
]
}
Useful element types include:
This gives you a document AST, rather than treating documents as bags of strings.
Unstructured's partitioning model follows essentially this philosophy: raw files are transformed into semantic elements such as titles, narrative text, and list items before chunking.
This is where many RAG pipelines quietly fail.
A table shouldn't become:
"Q1 10 Q2 15 Q3 17..." Preserve its relationships.
For example:
Table:
title: Revenue by region
columns: [Region, Q1, Q2, Q3]
rows:
- [US, 10M, 12M, 14M]
- [EU, 7M, 8M, 11M]
I'd generally retain both:
Likewise, images should retain:
That lets you answer questions like "What does Figure 4 show?" rather than losing the figure entirely.
Remove obvious garbage:
But don't aggressively summarize or rewrite the source here.
You want:
raw → normalized
not:
raw → LLM paraphrase → normalized
The latter introduces information loss and hallucinated content before you've even reached retrieval.
Keep the original extracted text alongside the cleaned version so you can debug extraction errors.
This is the second biggest quality lever.
Bad:
every 500 tokens
Better:
document
└── section
├── subsection
│ ├── paragraph
│ └── paragraph
└── subsection
Then combine adjacent semantic units until you hit your target size.
Heading-aware chunking is particularly useful because it prevents a chunk from arbitrarily crossing section boundaries. Unstructured explicitly supports this approach, including preserving title/section and optionally page boundaries.
A good starting point is roughly 300–800 tokens per chunk, but don't treat that as a universal optimum. Optimize it against your actual retrieval benchmark.
Also, don't automatically add 20% overlap everywhere. If your chunks already respect semantic boundaries, unnecessary overlap can actually add noise; Unstructured makes the same distinction between splitting oversized elements and applying overlap to otherwise coherent chunks.
Every chunk should know where it came from.
For example:
{
"chunk_id": "abc123-017",
"document_id": "abc123",
"text": "...",
"title": "2026 Annual Report",
"section": "Risk Factors",
"heading_path": [
"Annual Report",
"Risk Factors",
"Market Risk"
],
"page_start": 47,
"page_end": 48,
"source_uri": "...",
"document_version": "2026-02-14",
"language": "en",
"access_groups": ["finance"],
"element_types": ["paragraph"],
"parser_version": "v3"
}
This metadata is useful for much more than citations.
It enables:
For serious systems, I'd avoid vector-only retrieval.
Use something like:
┌── vector index
chunks ──────────┤
└── BM25/full-text index
↓
candidate fusion
↓
reranker
↓
top chunks
Vector search catches semantic matches.
Lexical search catches exact things such as:
Then use a reranker to select the best candidates.
Treat these as two different pipelines.
documents
→ parse
→ normalize
→ enrich
→ chunk
→ embed
→ index
query
→ query understanding
→ hybrid retrieval
→ reranking
→ context assembly
→ LLM
→ citations
This separation makes reprocessing and evaluation dramatically easier. Production RAG architectures commonly make this distinction explicitly.
This is the piece I'd prioritize most if you're building this for production.
Create ~100–500 representative questions with known relevant source passages.
Then measure independently:
Extraction
Retrieval
Generation
Then change one pipeline component at a time.
Otherwise you'll end up endlessly tweaking chunk sizes and embeddings without knowing which change actually helped.
If I were building one today, I'd use:
Object storage
↓
Queue / workflow engine
↓
Format-aware parsing + OCR/VLM
↓
Canonical document representation
↓
Cleaning + structural validation
↓
Heading/semantic chunking
↓
Metadata + ACL enrichment
↓
┌──────────────┬───────────────┐
↓ ↓
Embeddings BM25/full text
└──────────────┴───────────────┘
↓
Hybrid retrieval
↓
Reranker
↓
LLM
For the document-transformation layer specifically, docs.unstructured.io are a good reference implementation of the partition → enrich → chunk philosophy. Their current documentation also explicitly covers OCR/VLM, tables, images, chunking, and metadata.
If you remember only one thing:
Don't make "text" your canonical data model. Make the document structure your canonical model, and generate text chunks from that structure.
That one decision makes tables, citations, hierarchical retrieval, document updates, debugging, and multimodal documents substantially easier.
This metadata is useful for much more than citations.
It enables:
For serious systems, I'd avoid vector-only retrieval.
Use something like:
┌── vector index
chunks ──────────┤
└── BM25/full-text index
↓
candidate fusion
↓
reranker
↓
top chunks
Vector search catches semantic matches.
Lexical search catches exact things such as:
Then use a reranker to select the best candidates.
Treat these as two different pipelines.
documents
→ parse
→ normalize
→ enrich
→ chunk
→ embed
→ index
query
→ query understanding
→ hybrid retrieval
→ reranking
→ context assembly
→ LLM
→ citations
This separation makes reprocessing and evaluation dramatically easier. Production RAG architectures commonly make this distinction explicitly.
The best modern pipeline for turning unstructured documents (PDFs, images, Word docs) into LLM-ready data combines Unstructured (via the Unstructured Platform ) or Docling for parsing, coupled with an orchestration framework like LlamaIndex or LangChain for semantic chunking and embedding.
An ideal end-to-end architecture relies on a structured four-stage workflow.
Recommended Pipeline Stages
| Pipeline Stage | Top Tools / Libraries | Primary Function |
|---|---|---|
| Ingestion | Unstructured Connectors , Airbyte | Pull data from disparate enterprise storage |
| Parsing | Docling, Azure AI Vision, Unstructured API | Extract text, layout, and HTML tables |
| Orchestration | LlamaIndex , LangChain | Manage chunking strategies and pipelines |
| Storage | Pinecone, Astra DB, Milvus | Index and host vector embeddings |
If you want to tailor this further, tell me:
I can provide a tailored code snippet or specific library recommendation.
The best pipeline is not “PDF → text → embeddings.” For production LLM systems, treat document processing as a data-engineering pipeline whose primary job is to preserve meaning, provenance, structure, and permissions.
A strong default architecture is:
Raw documents
↓
1. Ingest + fingerprint
↓
2. Format-aware parsing / OCR
↓
3. Structure normalization
↓
4. Cleaning + deduplication
↓
5. Semantic extraction / enrichment
↓
6. Structure-aware chunking
↓
7. Metadata + provenance
↓
8. Validation / quality gates
↓
9. Embeddings + lexical index
↓
10. Retrieval + reranking
↓
LLM context
Keep the raw file immutable. Give every document a stable ID and version, and record things like:
This makes incremental processing, deletion, auditing, and reprocessing possible. Production guidance increasingly treats stable document/chunk IDs and source pointers as part of the ingestion contract.
This is probably the most important step.
For a PDF, you want something closer to:
{
"type": "section",
"heading": "Termination",
"page": 14,
"children": [
{"type": "paragraph", "text": "..."},
{"type": "table", "cells": [...]}
]
}
rather than:
TERMINATION ... some text ... 14 ... another column ...
Preserve headings, paragraphs, lists, tables, captions, page numbers, reading order, links, and images where applicable. Scanned documents need OCR, but OCR by itself isn't enough for complicated layouts because reading order and tables can still be wrong.
For heterogeneous corpora, I'd strongly favor a format-aware parser/partitioning layer such as Unstructured rather than maintaining dozens of bespoke PDF/DOCX/PPTX parsers yourself.
Don't immediately turn everything into chunks.
Create a canonical document model:
Document
├── metadata
├── sections
│ ├── heading
│ ├── paragraphs
│ ├── lists
│ ├── tables
│ └── figures
└── provenance
This intermediate representation becomes the source of truth from which you can generate different downstream representations.
That's valuable because today's application might use RAG, while tomorrow's might need structured extraction, summarization, knowledge graphs, or fine-tuning.
Typical transformations:
Be conservative with transformations that alter actual wording. Keep the original parsed representation so you can always trace a generated chunk back to its source. Deduplication and filtering are explicitly called out as important preprocessing steps in current RAG pipeline guidance.
Add metadata that will later help retrieval:
{
"document_id": "...",
"version": 7,
"source": "sharepoint",
"page": 14,
"section_path": [
"Employment Agreement",
"Termination",
"Notice"
],
"document_type": "contract",
"language": "en",
"access_groups": ["legal", "hr"]
}
The section path is particularly useful. A chunk saying “30 days” is much more useful when its context says:
Employment Agreement → Termination → Notice
Metadata also enables filtering by tenant, permissions, date, document type, etc.
This is where many otherwise-good pipelines go wrong.
Don't start with:
every 500 tokens
Instead:
Structure first → semantic boundaries second → token limit third.
For example:
H1: Termination
H2: Termination for Cause
paragraphs...
H2: Termination Without Cause
paragraphs...
H2: Notice Requirements
paragraphs...
Each coherent section becomes a candidate chunk, then you split only if it exceeds your target size.
The optimal chunk size depends on the retrieval task: factual lookup tends to benefit from smaller focused chunks, while explanatory questions may benefit from larger contextual chunks. Current guidance consistently treats chunking as a retrieval design problem, rather than merely a context-window problem.
For difficult documents, I'd use different strategies:
| Document | Preferred strategy |
|---|---|
| Contracts | hierarchical/section-based |
| Manuals | heading + subsection |
| Research papers | section/paragraph |
| Web pages | DOM/heading based |
| Emails | thread/message based |
| Tables | table-aware |
| Scanned forms | layout + field aware |
| Uniform logs | fixed/token based |
A good chunk record might look like:
{
"chunk_id": "doc123:v7:c042",
"document_id": "doc123",
"text": "...",
"section_path": ["Termination", "Notice"],
"page_start": 14,
"page_end": 15,
"source_uri": "...",
"document_version": 7,
"acl": ["legal"],
"parser_version": "2.4.1"
}
That last field—parser version—is surprisingly important. If you change your parser, you want to know which chunks were generated with which version.
Have automated quality gates such as:
parse succeeded?
↓
expected page count?
↓
text extraction non-empty?
↓
tables reasonable?
↓
OCR confidence acceptable?
↓
duplicate detected?
↓
chunk size within bounds?
↓
required metadata present?
↓
ACL present?
↓
publish to index
For high-value documents, route suspicious extraction to human review rather than silently indexing garbage. Validation at multiple pipeline stages is a useful production pattern because failures in parsing/extraction otherwise propagate downstream.
For retrieval, I'd generally start with hybrid search:
┌─ semantic/vector ─┐
query ───────────┤ ├─→ candidate set
└─ keyword/BM25 ────┘
↓
reranker
↓
top chunks
↓
LLM
Dense search handles paraphrases; lexical search is much better for exact identifiers, product codes, names, acronyms, and numbers. A reranker then improves precision over the initial candidate pool.
I would architect two distinct systems:
Offline/indexing path
sources → parse → normalize → enrich → chunk → validate
→ embed → index
Online/query path
question → query processing → hybrid retrieval
→ filtering → reranking → context assembly → LLM
This separation makes the system much easier to debug. When an answer is wrong, you can ask:
Rather than treating everything as "the LLM hallucinated."
If I were building this today, I'd aim for:
Storage
Processing
Retrieval
Operations
The key conceptual shift is:
Your LLM-ready representation should be an evidence layer, not just a pile of text chunks.
Preserve the original document → preserve its structure → create traceable semantic units → attach metadata and permissions → validate them → then create whatever indexes the application needs.
That approach also lets you change embedding models, chunking strategies, or vector databases without having to re-parse the original documents.
Turning unstructured documents (PDFs, scans, Word docs, images) into clean, LLM-ready data requires a robust parsing and ingestion pipeline . The current gold standard relies on a modular, multi-step pipeline rather than a single off-the-shelf parser.
Here is the breakdown of the best-practice pipeline for 2026:
If you want to tailor this, let me know:
I can give you a specific tech stack recommendation.
The best pipeline is not “PDF → text → embeddings.” Treat document processing as a data-engineering pipeline whose goal is to preserve meaning, structure, provenance, and retrieval boundaries.
A strong production architecture looks like this:
Raw documents
│
├── PDF / DOCX / PPTX / XLSX / HTML / scans / email
▼
1. Ingest + fingerprint
│
▼
2. Parse + OCR + layout analysis
│
▼
3. Canonical document representation
│ ├── text
│ ├── headings
│ ├── tables
│ ├── figures/images
│ ├── pages / bounding boxes
│ └── provenance
▼
4. Clean + normalize
│
▼
5. Semantic structure extraction
│ ├── sections
│ ├── titles
│ ├── entities
│ └── document metadata
▼
6. Structure-aware chunking
│
▼
7. Quality checks
│
├── OCR confidence
├── empty/broken chunks
├── table integrity
└── source ↔ chunk traceability
▼
8. Enrichment
│ ├── metadata
│ ├── summaries
│ └── embeddings
▼
9. Hybrid indexes
│ ├── vector
│ ├── keyword
│ └── metadata/filter index
▼
10. Retrieval → reranking → LLM
This is probably the most important design decision.
For PDFs especially, don't extract raw text and then blindly split every N characters. You want to recover reading order, headings, tables, lists, page boundaries, etc.
Docling is a particularly good open-source option: its current pipelines support layout analysis, OCR, table structure extraction, and different pipelines for ordinary PDFs versus complex/scanned documents.
For example:
The result should be a canonical intermediate representation, rather than immediately throwing everything into a vector database.
I'd represent every piece of content roughly like:
{
"document_id": "abc123",
"source": "s3://bucket/report.pdf",
"page": 17,
"section_path": [
"Annual Report",
"Financial Statements",
"Revenue"
],
"content_type": "table",
"text": "...",
"bbox": [x1, y1, x2, y2],
"parent_id": "...",
"metadata": {
"date": "2026-03-31",
"department": "finance"
}
}
That structure becomes extremely valuable later. You can tell the LLM where an answer came from, filter retrieval by metadata, reconstruct surrounding context, and display the original page.
Frameworks such as LlamaIndex explicitly model this as Documents → Nodes, with metadata and relationships preserved through transformations.
Do things like:
But don't aggressively “clean” the source. Keep the original representation immutable and create a cleaned derivative.
You want:
raw → parsed → normalized → enriched → indexed
rather than overwriting the source at every stage.
A good hierarchy is:
Document
└── Section
└── Subsection
└── Paragraph / table / list
└── Retrieval chunk
Start a chunk at natural boundaries. Keep its heading hierarchy attached to it.
For example, instead of:
"The company had revenue of $4.2B..."
make the retrieval unit:
Annual Report
> Financial Results
> Revenue
The company had revenue of $4.2B...
This gives the embedding considerably more context.
LlamaIndex similarly treats chunking as a transformation step and supports multiple structural/token/sentence-based splitters.
Don't obsess over a universal chunk size. Start around 300–800 tokens for ordinary prose and tune against your retrieval evaluation set. Tables, code, legal clauses, and technical documentation often deserve different treatment.
This is where many document RAG systems fall apart.
Don't turn:
| Year | Revenue | Margin |
|---|---|---|
| 2025 | $10M | 20% |
| 2026 | $14M | 25% |
into a mangled paragraph.
Preserve:
Then optionally create a textual representation specifically for retrieval.
For numerical questions, you may eventually want structured extraction into a database in addition to RAG.
Useful metadata includes:
Metadata isn't just decoration—it enables retrieval filters and makes your system auditable.
A particularly good production pattern is:
┌── raw/canonical document
│
document ───────────┼── retrieval chunks
│
├── summaries
│
├── embeddings
│
└── structured facts
Don't force one representation to serve every purpose.
For example:
Then retrieval can combine them.
Hash every source document and track processing versions:
document_id
source_hash
parser_version
chunker_version
embedding_model
processed_at
If a document hasn't changed, don't process it again.
If only your embedding model changes, you shouldn't have to redo OCR.
A pipeline framework such as LlamaIndex's ingestion pipeline supports caching and document/hash-based deduplication, which is the right general architecture.
This is the part teams often skip.
Create a test set of questions such as:
Question
→ expected document
→ expected page/section
→ expected chunk(s)
Then measure:
You should be able to change your parser/chunker and answer:
"Did retrieval get better or worse?"
without relying on vibes.
For a new system today, I'd start with:
Documents → Docling → canonical JSON/Markdown → custom normalization → semantic chunking → metadata enrichment → embeddings + BM25 → reranker → LLM
Docling is attractive if you want a local/open-source parsing layer; its current tooling can export Markdown, JSON, HTML, and chunks and has a native hybrid chunking approach.
For orchestration/indexing, LlamaIndex is a reasonable choice because its ingestion abstraction already maps naturally onto load → transform → index, with metadata, chunking, embeddings, caching, and document management.
If parsing quality is the bottleneck and you're happy with a managed service, a managed parser such as LlamaParse can be substituted for the parsing stage; its current API exposes different parsing tiers ranging from rule-based to AI-powered parsing.
Preserve information first; optimize it for LLMs second.
Your canonical representation should be richer than what you ultimately put into the prompt. Once you throw away layout, table structure, page numbers, provenance, and hierarchy, it's surprisingly difficult to reconstruct them later.
If you tell me whether you're building RAG, fine-tuning/training data, an agent over enterprise documents, or a general-purpose document warehouse, I can give you a concrete production architecture and component choices for that use case.
Turning messy, unstructured documents (PDFs, scans, Word docs, images) into clean, LLM-ready data requires a robust multi-stage pipeline. The industry-standard approach moves through ingestion, parsing, chunking, enrichment, and storage.
Here is the blueprint for the best modern pipeline:
H1 or H2 header) or embedding distance shifts, ensuring a single chunk contains a complete thought or section.Recommended Open-Source & Enterprise Tools
unstructured-io ) designed specifically for ingesting and preprocessing docs for RAG.If you'd like, let me know:
I can tailor a specific tech stack and code pattern for your project.
The best general-purpose pipeline is not “PDF → text → chunk → embeddings.” Treat document ingestion as a document-understanding pipeline that preserves structure, provenance, and modality all the way through.
A strong architecture looks like this:
┌── native text extraction
├── OCR
Documents ───────┼── layout / reading-order analysis
├── table extraction
└── image / figure understanding
↓
Canonical document model
↓
normalize + validate + enrich
↓
hierarchy-aware chunking
↓
metadata + provenance attachment
↓
┌───────────────┴────────────────┐
↓ ↓
lexical index vector index
│ │
└──────────────┬─────────────────┘
↓
LLM / RAG layer
Your intermediate representation should know that something is a heading, paragraph, table, caption, list, footnote, figure, etc., rather than producing one giant string.
For example, Docling produces a unified structured document representation and can export Markdown/JSON while handling PDFs, DOCX, PPTX, spreadsheets, images and OCR.
Unstructured is another good choice, particularly when you want a broader ingestion/partitioning/chunking pipeline. Its partitioning explicitly produces semantic document elements rather than merely splitting extracted text.
My default today: Docling for a document-centric pipeline; Unstructured if its connectors/ecosystem fit your sources better.
Don't send every page through an expensive vision model.
Pass 1 — cheap/deterministic
Pass 2 — targeted OCR
Pass 3 — targeted VLM
Docling, for example, has separate standard and VLM PDF pipelines and recommends choosing between them based on document complexity and latency requirements.
This staged approach is usually much cheaper than “VLM every page.”
I'd store something approximately like:
{
"document_id": "...",
"version": "...",
"source": {
"uri": "...",
"hash": "...",
"ingested_at": "..."
},
"elements": [
{
"id": "...",
"type": "heading",
"text": "3.2 Pricing",
"page": 17,
"bbox": [x1, y1, x2, y2],
"parent": "section-3"
},
{
"id": "...",
"type": "table",
"page": 18,
"section": "3.2 Pricing",
"data": {}
}
]
}
Don't throw away the source representation after generating Markdown.
Keep:
That metadata becomes incredibly valuable for citations, debugging, incremental reprocessing and auditability.
Do the boring cleanup here:
Don't aggressively “clean” the text. A transformation that makes text prettier but destroys structure is a net loss for RAG.
This is probably the most important part after parsing.
Avoid:
every 500 tokens
as your default strategy.
Instead:
Document
└── Chapter
└── Section
├── paragraph
├── paragraph
├── table
└── subsection
Then construct chunks from coherent elements while respecting boundaries.
Unstructured's chunking model follows this philosophy: it chunks already-identified document elements and generally keeps whole semantic elements together, only splitting an individual element when necessary.
A useful chunk might therefore carry:
{
"text": "...",
"title": "Pricing",
"section_path": [
"Product Guide",
"Pricing",
"Enterprise"
],
"page_start": 18,
"page_end": 19,
"document_id": "...",
"element_ids": ["..."]
}
Recent evaluation work also points in this direction: hierarchy-aware splitting and metadata enrichment can matter more to downstream QA accuracy than the particular PDF conversion framework you choose.
This is a common failure mode.
Don't blindly convert:
table → Markdown → embedding
Keep the structured table representation and generate a retrieval-friendly textual representation.
For example:
Table: Enterprise pricing
Region: US
Plan: Enterprise
Monthly price: $499
Seats: 100
...
That gives you something the embedding model can retrieve while retaining the original structured table for precise answering.
Same principle for charts: retain the original figure plus a generated description/structured extraction when useful.
For serious RAG, I wouldn't rely on one embedding per chunk.
A good retrieval layer can combine:
Then:
query
↓
hybrid retrieval
↓
top 50
↓
reranker
↓
top 5–10
↓
LLM
This is particularly important for things like product names, part numbers, legal clauses and exact terminology, where lexical matching can outperform semantic similarity.
This is where production pipelines differ from demos.
For every document, measure things like:
text extraction coverage
OCR confidence
table extraction success
page count consistency
duplicate content
missing headings
reading-order anomalies
And keep a small human-reviewed benchmark:
Document → expected elements → expected questions/answers
Then evaluate the entire pipeline, not merely OCR accuracy:
parse → chunk → retrieve → answer
The goal is ultimately answer accuracy / citation accuracy, not “our Markdown looks nice.”
Hash the original document and preferably individual pages/elements.
When version 17 of a 500-page manual changes on page 327:
don't:
500 pages → OCR → parse → embed → index
do:
hash comparison
↓
changed pages/elements
↓
reprocess
↓
rechunk affected sections
↓
replace affected index entries
Keep old versions if you need historical/audit queries.
For a new system I'd start with:
Ingestion → Docling
Canonical format → structured JSON + Markdown rendering
OCR/VLM → tiered fallback based on page complexity
Chunking → heading/section-aware + element-aware
Metadata → source/page/section/bbox/version/element IDs
Retrieval → hybrid BM25 + embeddings
Reranking → cross-encoder/reranker
Storage → object store for originals + relational/document DB for canonical representation + vector/lexical indexes
Evaluation → golden QA set + retrieval recall + answer/citation accuracy
The key architectural principle is:
Never turn a rich document into plain text earlier than necessary.
Preserve structure → tables → hierarchy → provenance → modality, and only create the flattened representations needed by each downstream component.
If you're building this for a particular corpus—PDF-heavy legal docs, financial filings, technical manuals, emails/Office files, or millions of mixed documents—the optimal pipeline changes substantially.
Turning messy, unstructured documents (PDFs, scanned images, Word docs, messy tables) into clean, LLM-ready data requires a robust, multi-stage parsing and ingestion pipeline.
The modern consensus leans heavily toward multimodal parsing (using vision-language models or advanced layout-aware engines) rather than traditional text-scraping regex.
Here is the best end-to-end pipeline architecture:
If you'd like to narrow this down, tell me:
I can map out a specific toolset and sample architecture for your use case.
