Claude Code with LlamaIndex: RAG Pipelines That Actually Work
Why LlamaIndex without CLAUDE.md produces bad retrieval
LlamaIndex is the most opinionated of the RAG frameworks. Where LangChain offers ten ways to do everything, LlamaIndex offers one or two that are battle-tested for production. This is a strength when you understand the framework's conventions and a problem when you do not, because Claude Code without context will generate code that ignores the conventions and reaches for primitives that work but produce poor retrieval results.
The most common LlamaIndex mistakes Claude makes by default: building indexes inline in API handlers instead of using the ingestion pipeline pattern, picking the default chunk size of 1024 tokens for every document type, treating VectorStoreIndex and SummaryIndex as interchangeable, calling index.query() without configuring a node post-processor for reranking, persisting indexes to disk with pickle instead of StorageContext, and skipping the query engine layer entirely by hitting the retriever directly.
This guide covers the CLAUDE.md configuration that locks Claude Code into LlamaIndex's correct model: an ingestion pipeline that runs once and persists, a vector store backed by your choice of provider, query engines with reranking, and the boundary between LlamaIndex objects that Claude will mix up. If you are building the broader application around LlamaIndex, Claude Code with TypeScript covers the type system for the FastAPI or Next.js layer that sits in front, and Claude Code with Postgres covers the database choices for storing both vectors (with pgvector) and metadata.
The LlamaIndex CLAUDE.md template
The CLAUDE.md at your project root needs to declare which LlamaIndex version you are using, the LLM and embedding model, the vector store backend, the ingestion pipeline structure, and the query engine pattern. Most Claude mistakes come from missing one of these declarations.
# LlamaIndex RAG rules
## Stack
- llama-index ^0.10.x (post-namespace split, packages are llama-index-core, llama-index-llms-anthropic, etc.)
- llama-index-llms-anthropic for Claude as the LLM
- llama-index-embeddings-openai for embeddings (text-embedding-3-large by default)
- llama-index-vector-stores-postgres for pgvector OR llama-index-vector-stores-qdrant for managed
- Python 3.11.x, pydantic ^2.x
## Environment variables
- ANTHROPIC_API_KEY in .env
- OPENAI_API_KEY in .env (for embeddings only, NOT for the LLM)
- DATABASE_URL or QDRANT_URL depending on vector store
- LLAMA_CLOUD_API_KEY if using LlamaCloud for parsing
## Project structure
- src/rag/ingestion.py Ingestion pipeline (runs once per document)
- src/rag/index.py Index loading and querying
- src/rag/settings.py Global LlamaIndex Settings configuration
- src/rag/postprocessors.py Custom node post-processors
- data/raw/ Source documents
- storage/ Persisted StorageContext
## Settings (configure ONCE, in src/rag/settings.py)
from llama_index.core import Settings
from llama_index.llms.anthropic import Anthropic
from llama_index.embeddings.openai import OpenAIEmbedding
Settings.llm = Anthropic(model="claude-opus-4-7")
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-large")
Settings.chunk_size = 512 # NOT the default 1024 for most use cases
Settings.chunk_overlap = 50
## Hard rules
- NEVER build an index inline in an API handler, use the ingestion pipeline
- NEVER use the default chunk_size of 1024 without justifying it for your document type
- NEVER use VectorStoreIndex for summarisation tasks, use SummaryIndex
- NEVER skip the node post-processor stage, ALWAYS rerank
- NEVER persist indexes with pickle, use StorageContext.persist()
- NEVER call retriever.retrieve() directly in production, go through query_engine.query()
- ALWAYS configure Settings ONCE at module import, never per-query
Three rules in this template prevent the majority of failures Claude generates without them.
The ingestion pipeline rule is the most important architecturally. Claude trained on quickstart docs generates code like index = VectorStoreIndex.from_documents(documents) inside a FastAPI endpoint, which means the entire index is rebuilt on every request. The ingestion pipeline pattern separates the one-time work of chunking, embedding, and storing from the per-request work of querying. Without this rule, every API call re-embeds the corpus.
The chunk size rule sets a sensible default. LlamaIndex's default of 1024 tokens is calibrated for general document retrieval. For most RAG use cases, especially code, technical documentation, and customer support content, 512 tokens with 50 tokens of overlap produces tighter, more relevant retrievals. Claude will use the default unless told otherwise.
The reranking rule addresses the biggest quality difference between a naive RAG pipeline and a production one. Retrieving the top 10 chunks by vector similarity and feeding all 10 to the LLM produces noisy answers. Retrieving the top 20 and reranking with a cross-encoder to keep the top 5 produces significantly better answers. Claude will not add a reranker by default because it does not appear in the quickstart.
Install and Settings configuration
LlamaIndex 0.10 split the monolithic package into modular packages. Install the core plus the integrations you need:
pip install \
llama-index-core \
llama-index-llms-anthropic \
llama-index-embeddings-openai \
llama-index-vector-stores-postgres \
llama-index-readers-file \
llama-index-postprocessor-cohere-rerank
Configure the global Settings in one place, imported by every module that uses LlamaIndex:
# src/rag/settings.py
import os
from llama_index.core import Settings
from llama_index.llms.anthropic import Anthropic
from llama_index.embeddings.openai import OpenAIEmbedding
def configure_llama_index() -> None:
"""Configure global LlamaIndex Settings. Call once at app startup."""
if not os.getenv("ANTHROPIC_API_KEY"):
raise RuntimeError("ANTHROPIC_API_KEY is not set")
if not os.getenv("OPENAI_API_KEY"):
raise RuntimeError("OPENAI_API_KEY is not set")
Settings.llm = Anthropic(
model="claude-opus-4-7",
max_tokens=4096,
temperature=0.0,
)
Settings.embed_model = OpenAIEmbedding(
model="text-embedding-3-large",
dimensions=1024,
)
Settings.chunk_size = 512
Settings.chunk_overlap = 50
The dimensions=1024 parameter on the embedding model deserves a note. OpenAI's text-embedding-3-large natively produces 3072-dimensional vectors, but most vector stores work better with smaller dimensions, and OpenAI supports dimensionality reduction at the API level. Picking 1024 (a third of the default) preserves over 95% of retrieval quality while reducing storage and search cost by two thirds. Claude does not know this trade-off and will use the default unless told.
Add an embeddings section to CLAUDE.md:
## Embeddings
- OpenAIEmbedding(model="text-embedding-3-large", dimensions=1024)
- dimensions=1024 reduces storage by 66% with minimal quality loss
- NEVER mix embedding models across index builds (existing vectors become incompatible)
- NEVER swap text-embedding-3-large for ada-002 unless cost is the primary constraint
- For multilingual: text-embedding-3-large handles 100+ languages without separate models
The ingestion pipeline
LlamaIndex's IngestionPipeline is the production pattern for converting raw documents into indexed nodes. It runs once per document, chunks and embeds in a single pass, deduplicates against an existing store, and persists results.
# src/rag/ingestion.py
from pathlib import Path
from llama_index.core import SimpleDirectoryReader
from llama_index.core.ingestion import IngestionPipeline
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.extractors import TitleExtractor, QuestionsAnsweredExtractor
from llama_index.vector_stores.postgres import PGVectorStore
from src.rag.settings import configure_llama_index
def build_ingestion_pipeline(connection_string: str) -> IngestionPipeline:
configure_llama_index()
vector_store = PGVectorStore.from_params(
database="ragdb",
host="localhost",
password="...",
port=5432,
user="rag",
table_name="documents",
embed_dim=1024,
)
pipeline = IngestionPipeline(
transformations=[
SentenceSplitter(chunk_size=512, chunk_overlap=50),
TitleExtractor(nodes=3),
QuestionsAnsweredExtractor(questions=2),
],
vector_store=vector_store,
)
return pipeline
def ingest_directory(source_dir: Path, connection_string: str) -> int:
documents = SimpleDirectoryReader(
input_dir=str(source_dir),
recursive=True,
required_exts=[".md", ".txt", ".pdf"],
).load_data()
pipeline = build_ingestion_pipeline(connection_string)
nodes = pipeline.run(documents=documents, show_progress=True)
return len(nodes)
A few details Claude tends to skip without guidance.
SentenceSplitter splits on sentence boundaries rather than character boundaries, which produces cleaner chunks. Claude defaults to TokenTextSplitter because it appears first in the docs, but SentenceSplitter is the better default for natural language.
The TitleExtractor and QuestionsAnsweredExtractor are metadata extractors that run during ingestion and attach generated metadata to each node. The questions extractor generates hypothetical questions that each chunk would answer, which improves retrieval because user queries match the generated questions more closely than they match the raw chunk text.
IngestionPipeline deduplicates against the vector store by default. Running the pipeline twice on the same documents produces no duplicate nodes. Claude often writes its own deduplication logic on top, which is redundant and slower.
Add an ingestion section to CLAUDE.md:
## Ingestion pipeline
- Use IngestionPipeline with transformations array
- Default transformations:
SentenceSplitter (NOT TokenTextSplitter for natural language)
TitleExtractor(nodes=3)
QuestionsAnsweredExtractor(questions=2)
- IngestionPipeline.run() deduplicates automatically against vector store
- NEVER write custom deduplication on top
- show_progress=True for CLI runs, False for production scripts
- For PDFs with complex layouts, use LlamaParse (LLAMA_CLOUD_API_KEY) instead of SimpleDirectoryReader
Choosing the right index type
LlamaIndex provides several index types, each optimised for different access patterns. The two Claude treats as interchangeable are VectorStoreIndex and SummaryIndex, but they answer different questions.
| Index | Best for | Behaviour |
|---|---|---|
VectorStoreIndex |
"Find me information about X" | Semantic search, returns top_k chunks |
SummaryIndex |
"Summarise the whole corpus" | Reads every chunk, synthesises full answer |
KeywordTableIndex |
"Find chunks mentioning X" | Lexical keyword matching |
TreeIndex |
"Hierarchical Q&A" | Builds a tree of summaries |
KnowledgeGraphIndex |
"Find relationships between entities" | Extracts triples, queries the graph |
For 90% of RAG use cases, VectorStoreIndex is correct. For document summarisation where you need to consider every chunk, SummaryIndex is correct. Claude will use VectorStoreIndex for summarisation requests, which produces partial summaries based on whichever 5 chunks were retrieved.
# src/rag/index.py
from llama_index.core import VectorStoreIndex, StorageContext
from llama_index.vector_stores.postgres import PGVectorStore
from src.rag.settings import configure_llama_index
def load_index(connection_params: dict) -> VectorStoreIndex:
configure_llama_index()
vector_store = PGVectorStore.from_params(**connection_params)
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_vector_store(
vector_store=vector_store,
storage_context=storage_context,
)
return index
The from_vector_store method loads an existing index from a populated vector store without rebuilding it. Claude often uses VectorStoreIndex.from_documents() everywhere, which rebuilds the index from scratch every time. For a production query API, from_vector_store is the right entry point.
Add an index types section to CLAUDE.md:
## Index types
- VectorStoreIndex: default for "find information about X" queries (95% of RAG)
- SummaryIndex: use ONLY for "summarise the whole document" queries
- NEVER use VectorStoreIndex for whole-corpus summarisation, retrieval will miss chunks
- Load existing indexes with from_vector_store(), NOT from_documents()
- from_documents() is for initial build only, called from ingestion pipeline
Get Claudify. The LlamaIndex CLAUDE.md template lands in your repo with ingestion patterns, index type rules, and reranking pre-configured.
Query engines with reranking
A naive query against VectorStoreIndex retrieves the top 5 chunks and feeds them to the LLM. A production query retrieves the top 20, reranks with a cross-encoder, keeps the top 5, and then feeds those to the LLM. The quality difference is significant.
# src/rag/query.py
from llama_index.core import VectorStoreIndex
from llama_index.core.postprocessor import SimilarityPostprocessor
from llama_index.postprocessor.cohere_rerank import CohereRerank
from src.rag.index import load_index
def build_query_engine(index: VectorStoreIndex):
reranker = CohereRerank(
api_key="...",
top_n=5,
model="rerank-english-v3.0",
)
similarity_filter = SimilarityPostprocessor(similarity_cutoff=0.7)
query_engine = index.as_query_engine(
similarity_top_k=20,
node_postprocessors=[similarity_filter, reranker],
response_mode="compact",
)
return query_engine
def answer(question: str) -> str:
index = load_index({"database": "ragdb", "host": "localhost", "port": 5432})
query_engine = build_query_engine(index)
response = query_engine.query(question)
return str(response)
The pipeline here:
- Retrieve top 20 chunks by vector similarity (
similarity_top_k=20) - Filter out chunks below 0.7 similarity (
SimilarityPostprocessor) - Rerank the remaining chunks with Cohere's reranker, keep top 5 (
CohereRerank(top_n=5)) - Synthesise an answer using
compactmode, which packs as many chunks as fit into a single LLM call
The response_mode controls how multiple retrieved chunks are combined:
| Mode | Behaviour |
|---|---|
compact |
Pack chunks into as few LLM calls as fit in context (default) |
refine |
Process chunks sequentially, refining the answer (one LLM call per chunk) |
tree_summarize |
Hierarchical summary (good for many chunks) |
simple_summarize |
Concatenate and summarise once |
For Claude with a 1M token context, compact is almost always the right choice because the model can hold all retrieved chunks at once. For older models or aggressive cost optimisation, refine may be preferable. Claude defaults to refine because that is the LlamaIndex default, which produces unnecessary LLM calls.
Add a query engine section to CLAUDE.md:
## Query engines
- ALWAYS use index.as_query_engine() with node_postprocessors
- similarity_top_k=20, NOT the default 2 (retrieve more, rerank, keep best)
- node_postprocessors=[SimilarityPostprocessor(0.7), CohereRerank(top_n=5)]
- response_mode="compact" for Claude (1M context), NOT "refine"
- Reranker is non-negotiable for production quality
- Use Cohere rerank-english-v3.0 (best price/performance) or rerank-multilingual-v3.0
Chat engines for multi-turn conversations
Query engines are stateless. For conversational interfaces, use as_chat_engine() which maintains conversation history.
# src/rag/chat.py
from llama_index.core import VectorStoreIndex
from llama_index.core.chat_engine.types import ChatMode
from llama_index.core.memory import ChatMemoryBuffer
from src.rag.index import load_index
from src.rag.postprocessors import default_postprocessors
def build_chat_engine(index: VectorStoreIndex):
memory = ChatMemoryBuffer.from_defaults(token_limit=3000)
chat_engine = index.as_chat_engine(
chat_mode=ChatMode.CONDENSE_PLUS_CONTEXT,
memory=memory,
similarity_top_k=20,
node_postprocessors=default_postprocessors(),
system_prompt=(
"You are a helpful assistant. Use the provided context to answer questions. "
"If the context does not contain the answer, say so."
),
)
return chat_engine
The CONDENSE_PLUS_CONTEXT chat mode does the right thing for most RAG chat applications: rewrites the follow-up question to be self-contained, retrieves chunks based on the rewritten question, then answers using those chunks. Claude tends to use the default ChatMode.BEST which is a runtime decision and produces inconsistent behaviour.
## Chat engines
- index.as_chat_engine(chat_mode=ChatMode.CONDENSE_PLUS_CONTEXT)
- NEVER use ChatMode.BEST in production (non-deterministic mode selection)
- Pass ChatMemoryBuffer.from_defaults(token_limit=3000) for memory
- Include node_postprocessors same as query engine
- ALWAYS set a system_prompt that explains the assistant's role and the fallback behaviour
Persistence with StorageContext
Indexes have two persistent components: the vector store (embeddings) and the doc store (chunks plus metadata). The vector store lives in your chosen backend (Postgres, Qdrant, Pinecone, etc.). The doc store can live in the vector store, in Redis, in a local file, or in cloud storage.
For most setups, persisting the entire StorageContext to a directory is the simplest pattern:
from llama_index.core import StorageContext, load_index_from_storage
# Persist after ingestion
storage_context.persist(persist_dir="./storage")
# Load on application startup
storage_context = StorageContext.from_defaults(persist_dir="./storage")
index = load_index_from_storage(storage_context)
For production with a managed vector store, the vector store handles its own persistence and only the doc store needs explicit handling. Most teams use the same Postgres database for both:
from llama_index.storage.docstore.postgres import PostgresDocumentStore
from llama_index.vector_stores.postgres import PGVectorStore
vector_store = PGVectorStore.from_params(...)
docstore = PostgresDocumentStore.from_uri(uri="postgresql://...", table_name="docstore")
storage_context = StorageContext.from_defaults(
vector_store=vector_store,
docstore=docstore,
)
Claude often falls back to pickle for persistence because it is the Python default for serialising arbitrary objects. This breaks the moment you change LlamaIndex versions or move between machines. The StorageContext.persist() pattern uses JSON internally and is version-resilient.
Add a persistence section to CLAUDE.md:
## Persistence
- ALWAYS use StorageContext.persist(persist_dir=...) for filesystem
- For production: vector store + docstore in same Postgres DB
- NEVER use pickle for index persistence (version-fragile)
- Reload with StorageContext.from_defaults(persist_dir=...) then load_index_from_storage()
- For multi-tenant: separate persist_dir per tenant, or namespace within vector store
Common Claude Code mistakes with LlamaIndex
Six patterns Claude generates incorrectly without CLAUDE.md constraints, with the correct replacement for each.
1. Inline index construction
Claude generates: index = VectorStoreIndex.from_documents(documents) inside an API request handler.
Correct pattern: Build the index once via IngestionPipeline, persist with StorageContext, load with from_vector_store() in the handler.
2. Default chunk size
Claude generates: SentenceSplitter() with no arguments, which uses the default 1024 token chunk size.
Correct pattern: SentenceSplitter(chunk_size=512, chunk_overlap=50) for most document types.
3. No reranker
Claude generates: index.as_query_engine(similarity_top_k=5) with no node post-processors.
Correct pattern: Retrieve 20, rerank with Cohere, keep top 5.
4. Wrong response mode
Claude generates: index.as_query_engine(response_mode="refine") which makes one LLM call per chunk.
Correct pattern: response_mode="compact" for Claude's 1M context.
5. Pickle persistence
Claude generates: with open("index.pkl", "wb") as f: pickle.dump(index, f).
Correct pattern: storage_context.persist(persist_dir="./storage").
6. Settings reset per query
Claude generates: a configure_llama_index() call inside every request handler, resetting global settings on every request.
Correct pattern: Call configure_llama_index() once at app startup. The Settings object is a singleton.
Add a common mistakes section to CLAUDE.md with these six pairs. Claude reproduces patterns from CLAUDE.md examples more reliably than from abstract rules.
Permission hooks for LlamaIndex scripts
An ingestion pipeline accumulates scripts: corpus rebuilders, embedding model migrations, index resetters. Some are read-only, some destroy hours of compute.
In .claude/settings.local.json:
{
"permissions": {
"allow": [
"Bash(python -m src.rag.query*)",
"Bash(python -m src.rag.stats*)",
"Bash(python -m src.rag.preview*)"
],
"deny": [
"Bash(python -m src.rag.ingestion*)",
"Bash(python -m src.rag.rebuild_index*)",
"Bash(python -m src.rag.migrate_embeddings*)"
]
}
}
Ingestion and index rebuilds cost real money (embedding API calls scale linearly with corpus size). Migrations recompute embeddings under a different model and invalidate the existing index. Both should require explicit confirmation. For broader patterns on protecting expensive operations, Claude Code with Sentry covers monitoring patterns that catch failed ingestion runs before they silently leave the index empty.
Building RAG that produces good answers
The LlamaIndex CLAUDE.md in this guide produces RAG pipelines where ingestion is a one-time persistent operation, chunks are sized for the document type, every query goes through a reranker, response synthesis uses Claude's full context, the right index type matches the access pattern, and persistence uses StorageContext not pickle.
The underlying principle is the same as any framework integration with Claude Code. LlamaIndex without a CLAUDE.md produces code that runs but retrieves badly: 1024-token chunks that lose precision, no reranking so noise drowns signal, refine mode making sequential LLM calls Claude does not need, and indexes rebuilt on every request because nothing told the model to persist. The CLAUDE.md template removes each failure mode by making the correct pattern the only pattern Claude generates.
For the next layer of the stack, LlamaIndex works alongside Claude Code with Postgres for pgvector storage of embeddings, and Claude Code with FastAPI for the API layer that exposes your query engines to clients.
Get Claudify. Drop a LlamaIndex-aware CLAUDE.md into your project and ship retrieval that actually works.
More like this
Ready to upgrade your Claude Code setup?
Get Claudify
