Paper2Code: Magically Turn Machine Learning Research Papers into Functional Code Repositories

Tired of manually implementing complex algorithms from research papers? Paper2Code automates the process, turning scientific documents into ready-to-use code. This innovative tool significantly accelerates your machine learning projects, saving you valuable time and resources.

What is Paper2Code and How Can it Help You?

Paper2Code is a multi-agent, Large Language Model (LLM) system designed to generate code repositories directly from research papers. It employs a three-stage approach, carefully planning, analyzing, and generating code to produce faithful, high-quality implementations.

• Stop Re-inventing the Wheel: Paper2Code handles the heavy lifting of translating research into code, allowing you to focus on high-level tasks.
• Improve your Machine Learning Workflow: Fast track your experiment by quickly generating and testing implemented papers.
• Access Cutting-Edge Research: Implement the latest advancements by automating code generation.

Quick Start: Generating Code with Paper2Code

Ready to see Paper2Code in action? Here's how to get started quickly, using the "Attention Is All You Need" paper as an example. Whether you prefer using OpenAI's API or open-source models, the setup is straightforward.

Option 1: Using OpenAI API

This option leverages the power of OpenAI's models to generate code.

1. Install the OpenAI Package: pip install openai
2. Set Your API Key: export OPENAI_API_KEY="<YOUR_OPENAI_API_KEY>"
3. Run the Script:

   cd scripts
   bash run.sh
   

   Copy
   (Estimated cost for using o3-mini: $0.50–$0.70.)

Option 2: Using Open Source Models with vLLM

This option uses open-source models, offering an alternative to OpenAI's API.

1. Install vLLM: pip install vllm (Refer to the official vLLM repository if you encounter any installation issues.)
2. Run the Script:

   cd scripts
   bash run_llm.sh
   

   Copy
   (The default model is deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct.)

Understanding the Output Folder Structure

After running Paper2Code, the generated files are organized into a clear structure. Key directories include:

outputs/
├── Transformer/
│   ├── analyzing_artifacts/
│   ├── coding_artifacts/
│   └── planning_artifacts/
└── Transformer_repo/ # Final output repository

Detailed Setup Instructions: Preparing Your Environment for Paper2Code

Before diving in, ensure your environment is properly configured. This involves installing the necessary packages and converting your PDF papers into a structured JSON format.

1. Environment Setup

Install the required packages based on your chosen method:

• For OpenAI API: pip install openai
• For Open-Source Models: pip install vllm

Refer to the official vLLM repository if you encounter any issues during installation.

2. Converting PDF to JSON

Paper2Code requires the input paper to be in JSON format. Use the s2orc-doc2json repository for this conversion.

1. Clone the Repository:

   git clone https://github.com/allenai/s2orc-doc2json.git
   

   Copy
2. Run the PDF Processing Service:

   cd ./s2orc-doc2json/grobid-0.7.3
   ./gradlew run
   

   Copy
3. Convert Your PDF:

   mkdir -p ./s2orc-doc2json/output_dir/paper_coder
   python ./s2orc-doc2json/doc2json/grobid2json/process_pdf.py \
    -i ${PDF_PATH} \
    -t ./s2orc-doc2json/temp_dir/ \
    -o ./s2orc-doc2json/output_dir/paper_coder
   

   Copy

Running Paper2Code on Your Own Papers

After setting up the environment and converting your research paper to JSON, you can run Paper2Code on your own papers. Remember to modify the environment variables accordingly to point to your specific paper location. Using PaperCoder on a local pdf requires correct paths.

1. Using OpenAI API:

   export OPENAI_API_KEY="<YOUR_OPENAI_API_KEY>"
   cd scripts
   bash run.sh
   

   Copy
   (Estimated cost for using o3-mini: $0.50–$0.70.)
2. Using Open Source Models with vLLM:

   cd scripts
   bash run_llm.sh
   

   Copy
   (The default model is deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct.)

Exploring the Paper2Code Benchmark Datasets

The Paper2Code benchmark dataset includes detailed information about the benchmark dataset in data/paper2code, giving insight into how the models and code generation performs. Refer to Section 4.1, "Paper2Code Benchmark," in the paper for an in-depth look.

Evaluating Generated Repositories with a Model-Based Approach

Paper2Code offers a model-based approach to evaluate the quality of generated repositories which supports both reference-based and reference-free settings. The system critiques key implementation components, assigns severity levels, and provides a correctness score from 1 to 5.

• Reference-Free Evaluation: Evaluates without comparing to existing code.
• Reference-Based Evaluation: Compares against an existing "gold" repository.

Setting Up the Evaluation Environment

Before evaluation, ensure you have the necessary packages installed and your OpenAI API key configured:

1. Install Tiktoken: pip install tiktoken
2. Set Your API Key: export OPENAI_API_KEY="<YOUR_OPENAI_API_KEY>"

Performing Reference-Free Evaluation

cd codes/
python eval.py \
 --paper_name Transformer \
 --pdf_json_path ../examples/Transformer_cleaned.json \
 --data_dir ../data \
 --output_dir ../outputs/Transformer \
 --target_repo_dir ../outputs/Transformer_repo \
 --eval_result_dir ../results \
 --eval_type ref_free \
 --generated_n 8 \
 --papercoder

Performing Reference-Based Evaluation

cd codes/
python eval.py \
 --paper_name Transformer \
 --pdf_json_path ../examples/Transformer_cleaned.json \
 --data_dir ../data \
 --output_dir ../outputs/Transformer \
 --target_repo_dir ../outputs/Transformer_repo \
 --gold_repo_dir ../examples/Transformer_gold_repo \
 --eval_result_dir ../results \
 --eval_type ref_based \
 --generated_n 8 \
 --papercoder

Example Evaluation Output

========================================
🌟 Evaluation Summary 🌟
📄 Paper name: Transformer
🧪 Evaluation type: ref_based
📁 Target repo directory: ../outputs/Transformer_repo
📊 Evaluation result:
 📈 Score: 4.5000
 ✅ Valid: 8/8
========================================
🌟 Usage Summary 🌟
[Evaluation] Transformer - ref_based
🛠️ Model: o3-mini
📥 Input tokens: 44318 (Cost: $0.04874980)
📦 Cached input tokens: 0 (Cost: $0.00000000)
📤 Output tokens: 26310 (Cost: $0.11576400)
💵 Current total cost: $0.16451380
🪙 Accumulated total cost so far: $0.16451380
============================================

Automate Your Research with Paper2Code Today

Paper2Code offers a powerful way to bridge the gap between research papers and functional code. By automating code generation, it saves time, reduces errors, and simplifies the implementation of complex algorithms. Start using Paper2Code now to accelerate your machine learning projects and stay at the forefront of innovation.