Ditch Semantic Similarity: Unlock True Document Understanding with PageIndex

Are you tired of inaccurate results when using vector databases with long professional documents? Traditional vector-based RAG (Retrieval-Augmented Generation) struggles with true relevance, often missing the mark when domain expertise and multi-step reasoning are crucial. It's time to embrace reasoning-based retrieval for superior results.

What is PageIndex?

PageIndex is a groundbreaking document indexing system that transforms lengthy PDFs into intelligent, searchable tree structures, optimizing them for reasoning-based RAG. Think of it as creating an LLM-optimized table of contents.

Here's why it matters:

• Relevance, Not Just Similarity: PageIndex enables LLMs to think and reason their way to the most relevant sections, not just the most similar ones. This is critical for professional documents.
• Inspired by AlphaGo: PageIndex utilizes tree search, similar to the AI behind AlphaGo, to perform structured document retrieval, ensuring comprehensive and logical navigation.

Who Benefits from PageIndex?

PageIndex is perfect for:

• Financial reports
• Regulatory filings
• Academic textbooks
• Legal and technical manuals
• Any document exceeding LLM context limits

This makes it indispensable for professionals dealing with dense, complex information.

Key Features that Set PageIndex Apart

• Hierarchical Tree Structure: Enables LLMs to traverse documents logically, providing an intuitive, LLM-optimized table of contents.
• Precise Page Referencing: Every node includes a concise summary and precise start/end page indexes for pinpoint accuracy.
• Chunk-Free Segmentation: Nodes follow the natural structure of the document, eliminating the issues caused by arbitrary chunking methods of other systems.
• Scales to Massive Documents: Designed to handle documents containing hundreds or even thousands of pages.

See PageIndex in Action

Here's a snippet of PageIndex's output formatting the original document:

{
"title": "Financial Stability",
"node_id": "0006",
"start_index": 21,
"end_index": 22,
"summary": "The Federal Reserve...",
"nodes": [
  {
    "title": "Monitoring Financial Vulnerabilities",
    "node_id": "0007",
    "start_index": 22,
    "end_index": 28,
    "summary": "The Federal Reserve's monitoring..."
  },
  {
    "title": "Domestic and International Cooperation and Coordination",
    "node_id": "0008",
    "start_index": 28,
    "end_index": 31,
    "summary": "In 2023, the Federal Reserve collaborated..."
  }
]
}

Get Started with PageIndex

Whether you choose to self-host or use the cloud API, getting started is easy.

Self-Hosting:

1. Install dependencies: pip3 install -r requirements.txt
2. Set your OpenAI API key: Create a .env file with CHATGPT_API_KEY=your_openai_key_here
3. Run PageIndex: python3 run_pageindex.py --pdf_path /path/to/your/document.pdf

Cloud API (Beta):

Don't want to manage your own setup? Try out hosted API:

• Custom OCR model for superior PDF recognition.
• Web dashboard for visual exploration.
• Get 1,000 free pages – Leave your email [here](provided link).

Case Study: Mafin 2.5 – Proof of PageIndex Superiority

Mafin 2.5, a reasoning-based RAG model built on PageIndex, achieved a stunning 98.7% accuracy on the FinanceBench benchmark. This significantly outperforms traditional vector-based RAG systems, showcasing the power of PageIndex's hierarchical indexing for precise content extraction from complex financial documents. This provides a glimpse into the power of reasoning-based RAG.

Build Your Reasoning-Based RAG

PageIndex empowers you to create systems without relying on basic semantic similarity. Perfect for domain-specific applications where nuance and precision are essential.

Preprocessing Workflow Example:

1. Process documents using PageIndex to generate tree structures optimized for reasoning-based retrieval.
2. Store the tree structures and IDs in a database.
3. Store node content in a separate table, indexed by node and tree ID.

Reasoning-Based RAG Framework Example:

1. Query Preprocessing: Analyze the query to understand knowledge requirements.
2. Document Selection: Search for relevant documents and fetch tree structures.
3. Node Selection: Search tree structures to identify relevant nodes.
4. LLM Generation: Fetch node content, format it, and send it to the LLM for contextually informed responses.

Example Prompt for Node Selection

prompt = f"""
You are given a question and a tree structure of a document.
You need to find all nodes that are likely to contain the answer.

Question: { question}

Document tree structure: { structure}

Reply in the following JSON format:
{{
  "thinking": <reasoning about where to look>,
  "node_list": [node_id1, node_id2,...]
}}
"""

Join the Revolution of Reasoning-Based RAG

PageIndex is in early beta, and your input can shape the future. Report issues, ask questions, or contribute directly. And for a more accurate and reliable integration, try the hosted API.

• [Join our Discord](provided link)
• [Leave us a message](provided link)