Unlock PDF Data with MinerU: The Ultimate Open-Source Document Extraction Tool

Tired of wrestling with PDFs? MinerU is a game-changing, open-source tool that converts PDFs into machine-readable formats like Markdown and JSON. This allows for effortless data extraction and integration into your projects. Say goodbye to manual copy-pasting and hello to streamlined workflows!

MinerU: Revolutionizing Scientific Document Parsing

Born from the pre-training process of InternLM, MinerU tackles the challenge of symbol conversion in scientific literature. It's designed to contribute to technological advancement in the age of large language models. While still evolving, MinerU offers a powerful alternative to commercial solutions.

• Problem: Existing tools struggle with complex layouts and specialized symbols in academic papers.
• Solution: MinerU excels at preserving document structure while accurately extracting text and formulas.
• Benefit: Frees up researchers and developers to focus on analysis rather than tedious data wrangling.

Key Features That Set MinerU Apart

MinerU boasts a robust set of features designed to tackle the most challenging PDF extraction tasks:

• Semantic Coherence: Automatically removes headers, footers, and page numbers for clean data.
• Intelligent Layout Handling: Outputs text in a human-readable order, even in complex, multi-column documents.
• Structure Preservation: Retains headings, paragraphs, lists, and other structural elements.
• Comprehensive Content Extraction: Extracts images, tables, formulas (in LaTeX), and associated descriptions.
• OCR & Language Support: Detects scanned and garbled PDFs with OCR support for 84 languages.
• Flexible Output formats: Outputs in multimodal and NLP Markdown, reading-order-sorted JSON, and rich intermediate formats.
• Visualization Tools: Includes layout and span visualization for quality control.
• Cross-Platform Compatibility: Runs on CPU, GPU (CUDA), NPU(CANN), and MPS across Windows, Linux, and Mac.
• Hardware Acceleration: Accelerate your pdf extraction process using GPU(CUDA)/NPU(CANN)/MPS.

Major Performance Boosts and Updates in Recent Releases

MinerU is constantly evolving, with recent updates focusing on speed, accuracy, and usability. Check out these key improvements:

• Batch Processing: Significantly improved parsing speed for multiple PDF files, especially small ones.
• Memory Optimization: Reduced GPU memory usage, enabling operation with as little as 6GB of VRAM.
• Enhanced Formula Parsing: Fixed line break issues with the updated mfr model.
• Paddle-Free Architecture: Replaced PaddleOCR with paddleocr2torch to resolve conflicts and thread safety issues.
• Real-Time Progress Bar: Added a progress bar for improved user experience during parsing.
• Compatibility improvements: Resolved dependency incompatibilities and extended Torch version compatibility.
• Offline Deployment: Offline deployment process optimized; no internet connection required after successful deployment to download any model files.

Get Started with MinerU: Three Easy Approaches

MinerU offers multiple ways to get started, catering to different needs and environments:

1. Online Demo: Instantly test MinerU without installation.
2. Quick CPU Demo: For Windows, Linux, and Mac users, providing an accessible entry point.
   • Install magic-pdf via conda and pip.
   • Download Model Weight Files (refer to documentation).
   • Modify the Configuration File (magic-pdf.json) to customize features like table recognition.
3. Accelerated Inference (CUDA/CANN/MPS): Unleash the full power of MinerU with hardware acceleration.

Advanced PDF Parsing: Leveraging GPU Acceleration

Take advantage of GPU acceleration (CUDA) for blazing-fast parsing speeds if your system supports it. Follow the detailed guides for Ubuntu and Windows for a smooth setup. For Apple silicon users, MPS acceleration can be enabled in configuration for similar speed boosts.

Quick Deployment with Docker

Simplify deployment with Docker, pre-configured for GPU acceleration. Ensure your GPU has at least 6GB of VRAM.

Configuration and Usage

Customize MinerU behavior via the magic-pdf.json configuration file. Enable or disable features like formula and table recognition. Use the command line interface or the Python API for seamless integration into your projects.

Derived Projects

Explore community-built projects based on MinerU, offering additional functionality and user interfaces.

Areas for Future Development

MinerU continues to evolve, with planned enhancements including:

• Improved reading order determination.
• Enhanced list and index recognition.
• Code block recognition.
• Chemical formula and geometric shape recognition.

Known Limitations

While powerful, MinerU has some limitations:

• Reading order issues in extremely complex layouts.
• Lack of support for vertical text.
• Potential errors in complex table recognition.

Consult the FAQ for solutions to common issues.

Acknowledgments

MinerU leverages a number of open-source projects for its functionality, including PyMuPDF, DocLayout-YOLO, RapidTable, and PaddleOCR. The team expresses gratitude to all contributors.

Begin extracting data like never before and experience the power of a tool developed to streamline your workflow!