|$ curl https://forge-ai.dev/api/markdown?path=docs/ai/llamaindex
$cat docs/llamaindex.md
updated Recently·38 min read·published
LlamaIndex
Introduction
LlamaIndex is a data framework for connecting LLMs with external data sources. While LangChain focuses on general-purpose LLM orchestration, LlamaIndex specializes in data ingestion, indexing, and retrieval — making it the go-to choice for RAG applications.
The framework provides a unified interface for loading data from 160+ sources, creating vector and structured indexes, and building query engines that retrieve the most relevant context for LLM generation. Its strength lies in sophisticated retrieval strategies and document management.
Installation
install.sh
Bash
| 1 | # Core package |
| 2 | pip install llama-index |
| 3 | |
| 4 | # Core and LLM integrations |
| 5 | pip install llama-index-core |
| 6 | pip install llama-index-llms-openai |
| 7 | pip install llama-index-embeddings-openai |
| 8 | |
| 9 | # Vector store integrations |
| 10 | pip install llama-index-vector-stores-chroma |
| 11 | pip install llama-index-vector-stores-pinecone |
| 12 | |
| 13 | # Data loaders |
| 14 | pip install llama-index-readers-file |
| 15 | pip install llama-index-readers-web |
Data Connectors (Readers)
Data connectors (called Readers) ingest data from various sources and convert them into LlamaIndex Document objects. Each connector handles parsing, metadata extraction, and chunking.
connectors.py
Python
| 1 | from llama_index.core import SimpleDirectoryReader, VectorStoreIndex |
| 2 | from llama_index.readers.file import PDFReader, CSVReader |
| 3 | from llama_index.readers.web import BeautifulSoupWebReader |
| 4 | |
| 5 | # Load from local directory — auto-discovers file types |
| 6 | documents = SimpleDirectoryReader("./data").load_data() |
| 7 | print(f"Loaded {len(documents)} documents") |
| 8 | |
| 9 | # Load specific file types |
| 10 | pdf_docs = PDFReader().load_data(file=Path("report.pdf")) |
| 11 | csv_docs = CSVReader().load_data(file=Path("data.csv")) |
| 12 | |
| 13 | # Load from URLs |
| 14 | web_reader = BeautifulSoupWebReader() |
| 15 | web_docs = web_reader.load_data(urls=[ |
| 16 | "https://docs.python.org/3/tutorial/", |
| 17 | "https://docs.python.org/3/library/stdtypes.html", |
| 18 | ]) |
| 19 | |
| 20 | # Load from database |
| 21 | from llama_index.readers.database import DatabaseReader |
| 22 | |
| 23 | reader = DatabaseReader( |
| 24 | scheme="postgresql", |
| 25 | host="localhost", |
| 26 | port=5432, |
| 27 | user="user", |
| 28 | pass_="password", |
| 29 | dbname="mydb", |
| 30 | ) |
| 31 | db_docs = reader.query("SELECT content, metadata FROM documents") |
ℹ
info
LlamaIndex has 160+ community-contributed readers. Check the llama-hub registry for your data source. For custom formats, extend BaseReader and implement the load_data method.
Indexes
Indexes organize documents for efficient retrieval. LlamaIndex supports vector indexes, keyword indexes, tree indexes, and knowledge graph indexes — each optimized for different query patterns.
indexes.py
Python
| 1 | from llama_index.core import VectorStoreIndex, Settings |
| 2 | from llama_index.llms.openai import OpenAI |
| 3 | from llama_index.embeddings.openai import OpenAIEmbedding |
| 4 | |
| 5 | # Configure global settings |
| 6 | Settings.llm = OpenAI(model="gpt-4o", temperature=0) |
| 7 | Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small") |
| 8 | |
| 9 | # Vector Store Index — most common, semantic search |
| 10 | documents = SimpleDirectoryReader("./docs").load_data() |
| 11 | vector_index = VectorStoreIndex.from_documents(documents) |
| 12 | |
| 13 | # Persist to disk |
| 14 | vector_index.storage_context.persist(persist_dir="./index_storage") |
| 15 | |
| 16 | # Load from persisted storage |
| 17 | from llama_index.core import StorageContext, load_index_from_storage |
| 18 | storage_context = StorageContext.from_defaults(persist_dir="./index_storage") |
| 19 | loaded_index = load_index_from_storage(storage_context) |
| 20 | |
| 21 | # Tree Index — hierarchical summarization |
| 22 | from llama_index.core import TreeIndex |
| 23 | tree_index = TreeIndex.from_documents(documents) |
| 24 | |
| 25 | # Keyword Table Index — keyword-based retrieval |
| 26 | from llama_index.core import KeywordTableIndex |
| 27 | keyword_index = KeywordTableIndex.from_documents(documents) |
| 28 | |
| 29 | # Knowledge Graph Index — entity-relation extraction |
| 30 | from llama_index.core import KnowledgeGraphIndex |
| 31 | kg_index = KnowledgeGraphIndex.from_documents(documents) |
| Index Type | Best For | Trade-off |
|---|---|---|
| VectorStoreIndex | Semantic search, general Q&A | Best all-around choice |
| TreeIndex | Summarization, hierarchical queries | Slower construction, higher token cost |
| KeywordTableIndex | Exact keyword matching | No semantic understanding |
| KnowledgeGraphIndex | Entity relationships, reasoning | Complex setup, best for structured data |
✓
best practice
Start with VectorStoreIndex for most use cases. It provides the best balance of retrieval quality and setup simplicity. Add tree or keyword indexes later if you need specialized retrieval patterns.
Query Engines
Query engines take an index and a user query, retrieve relevant documents, and generate an answer using the LLM. They are the primary interface for asking questions over your data.
query_engines.py
Python
| 1 | # Basic query engine |
| 2 | query_engine = vector_index.as_query_engine( |
| 3 | similarity_top_k=5, |
| 4 | response_mode="compact", # compact, refine, tree_summarize, simple |
| 5 | ) |
| 6 | |
| 7 | response = query_engine.query("What are the main benefits of microservices?") |
| 8 | print(response.response) |
| 9 | print(f"Source nodes: {len(response.source_nodes)}") |
| 10 | |
| 11 | # Streaming response |
| 12 | streaming_response = query_engine.query("Explain event-driven architecture") |
| 13 | for token in streaming_response.response_gen: |
| 14 | print(token, end="", flush=True) |
| 15 | |
| 16 | # Custom prompt template |
| 17 | from llama_index.core import PromptTemplate |
| 18 | |
| 19 | qa_prompt = PromptTemplate( |
| 20 | template="""Context information is below. |
| 21 | --------------------- |
| 22 | {context_str} |
| 23 | --------------------- |
| 24 | Using only the context above, answer the question: {query_str} |
| 25 | Answer:""" |
| 26 | ) |
| 27 | |
| 28 | custom_engine = vector_index.as_query_engine( |
| 29 | text_qa_template=qa_prompt, |
| 30 | similarity_top_k=3, |
| 31 | ) |
| 32 | |
| 33 | # Query with metadata filtering |
| 34 | from llama_index.core.vector_stores import ExactMatchFilter, MetadataFilters |
| 35 | |
| 36 | filters = MetadataFilters( |
| 37 | filters=[ |
| 38 | ExactMatchFilter(key="category", value="engineering"), |
| 39 | ExactMatchFilter(key="year", value="2026"), |
| 40 | ] |
| 41 | ) |
| 42 | filtered_engine = vector_index.as_query_engine( |
| 43 | filters=filters, |
| 44 | similarity_top_k=5, |
| 45 | ) |
ℹ
info
Use response_mode="compact" for most cases — it concatenates retrieved nodes and sends them in a single LLM call, reducing token usage. Use "refine" when you need iterative detail improvement.
Retrievers
Retrivers handle the retrieval step without generation. They return the most relevant documents for a query, giving you fine-grained control over the retrieval pipeline.
retrievers.py
Python
| 1 | # Base retriever from index |
| 2 | retriever = vector_index.as_retriever( |
| 3 | similarity_top_k=10, |
| 4 | ) |
| 5 | |
| 6 | nodes = retriever.retrieve("What is event sourcing?") |
| 7 | for node in nodes: |
| 8 | print(f"Score: {node.score:.3f} | {node.text[:80]}...") |
| 9 | |
| 10 | # Recursive retriever — follow node relationships |
| 11 | from llama_index.core.retrievers import RecursiveRetriever |
| 12 | |
| 13 | recursive_retriever = RecursiveRetriever( |
| 14 | "vector", |
| 15 | retriever_dict={"vector": vector_retriever}, |
| 16 | node_dict=node_dict, |
| 17 | verbose=True, |
| 18 | ) |
| 19 | |
| 20 | # Auto-merging retriever — collapse similar chunks |
| 21 | from llama_index.retrievers.bm25 import BM25Retriever |
| 22 | |
| 23 | bm25_retriever = BM25Retriever.from_documents(documents, similarity_top_k=5) |
| 24 | |
| 25 | # Hybrid search — combine vector + keyword |
| 26 | from llama_index.core.retrievers import QueryFusionRetriever |
| 27 | |
| 28 | hybrid_retriever = QueryFusionRetriever( |
| 29 | retrievers=[vector_retriever, bm25_retriever], |
| 30 | num_queries=4, |
| 31 | use_async=True, |
| 32 | similarity_top_k=5, |
| 33 | ) |
retrievers_advanced.py
Python
| 1 | # Post-processing — reranking, filtering, transformation |
| 2 | from llama_index.core.postprocessor import ( |
| 3 | SimilarityPostprocessor, |
| 4 | KeywordNodePostprocessor, |
| 5 | EmbeddingRedundancyFilter, |
| 6 | ) |
| 7 | |
| 8 | # Filter by similarity score threshold |
| 9 | score_filter = SimilarityPostprocessor(similarity_cutoff=0.7) |
| 10 | |
| 11 | # Filter by required/avoided keywords |
| 12 | keyword_filter = KeywordNodePostprocessor( |
| 13 | required_keywords=["python", "async"], |
| 14 | avoid_keywords=["deprecated", "legacy"], |
| 15 | ) |
| 16 | |
| 17 | # Remove redundant chunks |
| 18 | dedup_filter = EmbeddingRedundancyFilter( |
| 19 | embedding_model=Settings.embed_model, |
| 20 | similarity_cutoff=0.95, |
| 21 | ) |
| 22 | |
| 23 | # Chain postprocessors |
| 24 | from llama_index.core.postprocessor import MetadataReplacementPostProcessor |
| 25 | |
| 26 | engine_with_rerank = vector_index.as_query_engine( |
| 27 | node_postprocessors=[score_filter, keyword_filter], |
| 28 | similarity_top_k=15, # Retrieve more, filter down |
| 29 | ) |
✓
best practice
Retrieve more documents than you need (top_k=15-20) and apply post-processing filters to narrow down. This two-stage approach (retrieve then rerank) significantly improves retrieval quality.
Chat Engines
Chat engines maintain conversation history and use it to contextualize follow-up questions. Unlike query engines (stateless), chat engines track the full dialogue.
chat_engines.py
Python
| 1 | # Simple chat engine |
| 2 | chat_engine = vector_index.as_chat_engine( |
| 3 | chat_mode="condense_plus_context", |
| 4 | similarity_top_k=5, |
| 5 | verbose=True, |
| 6 | ) |
| 7 | |
| 8 | # Multi-turn conversation |
| 9 | response1 = chat_engine.chat("What is Docker?") |
| 10 | print(response1.response) |
| 11 | |
| 12 | response2 = chat_engine.chat("How do I use it with Kubernetes?") |
| 13 | print(response2.response) # Understands "it" = Docker |
| 14 | |
| 15 | response3 = chat_engine.chat("What about networking?") |
| 16 | # Understands context from full conversation |
| 17 | |
| 18 | # Reset conversation |
| 19 | chat_engine.reset() |
| 20 | |
| 21 | # Available chat modes: |
| 22 | # - "best" — automatic mode selection |
| 23 | # - "condense_plus_context" — condense history + retrieve context |
| 24 | # - "condense_question" — condense follow-up into standalone question |
| 25 | # - "context" — always retrieve fresh context |
| 26 | # - "simple" — no context, just LLM chat |
ℹ
info
"condense_plus_context" is the best default chat mode — it rewrites follow-up questions into standalone queries while also retrieving relevant context, giving you the best of both worlds.
Sub-Question Query Engine
The sub-question query engine decomposes complex questions into simpler sub-questions, each answered by a specialized query engine. This is powerful for questions that span multiple data sources or require multi-step reasoning.
sub_question.py
Python
| 1 | from llama_index.core.query_engine import SubQuestionQueryEngine |
| 2 | from llama_index.core.tools import QueryEngineTool, ToolMetadata |
| 3 | |
| 4 | # Create query engine tools for different data sources |
| 5 | tools = [ |
| 6 | QueryEngineTool( |
| 7 | query_engine=python_docs_engine, |
| 8 | metadata=ToolMetadata( |
| 9 | name="python_docs", |
| 10 | description="Python language documentation and tutorials", |
| 11 | ), |
| 12 | ), |
| 13 | QueryEngineTool( |
| 14 | query_engine=kubernetes_docs_engine, |
| 15 | metadata=ToolMetadata( |
| 16 | name="kubernetes_docs", |
| 17 | description="Kubernetes documentation and guides", |
| 18 | ), |
| 19 | ), |
| 20 | QueryEngineTool( |
| 21 | query_engine=docker_docs_engine, |
| 22 | metadata=ToolMetadata( |
| 23 | name="docker_docs", |
| 24 | description="Docker documentation and tutorials", |
| 25 | ), |
| 26 | ), |
| 27 | ] |
| 28 | |
| 29 | # Sub-question engine automatically decomposes complex queries |
| 30 | sq_engine = SubQuestionQueryEngine.from_defaults( |
| 31 | query_engine_tools=tools, |
| 32 | verbose=True, |
| 33 | ) |
| 34 | |
| 35 | # Complex question — decomposed into sub-questions |
| 36 | response = sq_engine.query( |
| 37 | "Compare the networking models in Docker, Kubernetes, and Python asyncio" |
| 38 | ) |
| 39 | print(response) |
✓
best practice
Sub-question query engines excel at cross-domain questions. Each sub-question is answered independently, then results are synthesized. Use them when a single retrieval pass cannot answer the full question.
Evaluation
LlamaIndex provides built-in evaluation frameworks for measuring retrieval quality, answer correctness, and faithfulness — essential for iterating on your RAG pipeline.
evaluation.py
Python
| 1 | from llama_index.core.evaluation import ( |
| 2 | FaithfulnessEvaluator, |
| 3 | RelevancyEvaluator, |
| 4 | CorrectnessEvaluator, |
| 5 | BatchEvalRunner, |
| 6 | ) |
| 7 | |
| 8 | # Create evaluator |
| 9 | faithfulness_eval = FaithfulnessEvaluator(llm=llm) |
| 10 | relevancy_eval = RelevancyEvaluator(llm=llm) |
| 11 | |
| 12 | # Evaluate a query-response pair |
| 13 | response = query_engine.query("What is event sourcing?") |
| 14 | result = faithfulness_eval.evaluate_response(response) |
| 15 | print(f"Faithful: {result.passing}, Score: {result.score}") |
| 16 | |
| 17 | # Batch evaluation |
| 18 | eval_questions = [ |
| 19 | "What is Docker?", |
| 20 | "How does Kubernetes networking work?", |
| 21 | "Explain CI/CD pipelines", |
| 22 | ] |
| 23 | |
| 24 | eval_results = {} |
| 25 | for q in eval_questions: |
| 26 | response = query_engine.query(q) |
| 27 | eval_results[q] = { |
| 28 | "faithfulness": faithfulness_eval.evaluate_response(response), |
| 29 | "relevancy": relevancy_eval.evaluate_response(response), |
| 30 | } |
| 31 | |
| 32 | # Calculate metrics |
| 33 | avg_faithfulness = sum( |
| 34 | r["faithfulness"].score for r in eval_results.values() |
| 35 | ) / len(eval_results) |
| 36 | print(f"Avg faithfulness: {avg_faithfulness:.2f}") |
ℹ
info
Measure faithfulness (answer grounded in context) and relevancy (context matches question) separately. High faithfulness with low relevancy means your retriever returns irrelevant chunks. Low faithfulness means the LLM hallucinates beyond the context.
$Blueprint — Engineering Documentation·Section ID: AI-LI-01·Revision: 1.0