Paper2Code: Automatically Generate Code from Research Papers

Tired of manually implementing algorithms from research papers? Paper2Code automates the process of turning scientific papers into functional code repositories, saving you time and effort. This article dives into how Paper2Code can streamline your research workflow.

What is Paper2Code and How Does it Work?

PaperCoder is a multi-agent LLM system designed to convert research papers, especially in machine learning, into runnable code. The system uses a three-stage pipeline:

• Planning: The system analyzes the paper and creates a plan for code generation.
• Analysis: In-depth examination of the paper's methodology, data structures, and algorithms.
• Code Generation: Creation of the code repository based on the analysis, with specialized agents handling each stage.

This automated process helps researchers and developers quickly implement and test novel ideas presented in academic literature.

Quick Start with Paper2Code

Ready to try Paper2Code? Here's how to get started:

Using OpenAI API:

This method uses the OpenAI API (specifically o3-mini) to generate code. Note that this incurs costs.

1. Install the OpenAI package:

   pip install openai
   

   Copy
2. Set your OpenAI API key:

   export OPENAI_API_KEY="<YOUR_OPENAI_API_KEY>"
   

   Copy
3. Run the script:

   cd scripts
   bash run.sh
   

   Copy

Using Open Source Models with vLLM:

This method uses open-source models such as deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct, through vLLM.

1. Install vLLM:

   pip install vllm
   

   Copy
2. Run the script:

   cd scripts
   bash run_llm.sh
   

   Copy

Consult the official vLLM repository if you run into an installation issue.

Detailed Setup Instructions

Environment Setup

• For OpenAI API: pip install openai
• For open-source models: pip install vllm

Converting PDFs to JSON

To process your own papers, convert them to JSON format using s2orc-doc2json.

1. Clone the repository:

   git clone https://github.com/allenai/s2orc-doc2json.git
   

   Copy
2. Run the PDF processing service:

   cd ./s2orc-doc2json/grobid-0.7.3
   ./gradlew run
   

   Copy
3. Convert your PDF:

   mkdir -p ./s2orc-doc2json/output_dir/paper_coder
   python ./s2orc-doc2json/doc2json/grobid2json/process_pdf.py \
    -i ${PDF_PATH} \
    -t ./s2orc-doc2json/temp_dir/ \
    -o ./s2orc-doc2json/output_dir/paper_coder
   

   Copy

Running Paper2Code on Your Paper

Modify environment variables to point to your own paper. Use the same commands as in the Quick Start section, replacing the example paper with your converted JSON file.

Paper2Code Benchmark Datasets

You can find the Paper2Code benchmark dataset description in data/paper2code. Section 4.1, "Paper2Code Benchmark" in the research paper provides a comprehensive overview.

Evaluating Repositories Generated by PaperCoder

PaperCoder uses a model-based approach to evaluate the quality of the generated code repositories, supporting reference-based and reference-free settings. The model checks crucial parts of the implementation. For more details, see Section 4.3.1 of the paper.

Reference-Free Evaluation:

cd codes/
python eval.py \
--paper_name Transformer \
--pdf_json_path ../examples/Transformer_cleaned.json \
--data_dir ../data \
--output_dir ../outputs/Transformer \
--target_repo_dir ../outputs/Transformer_repo \
--eval_result_dir ../results \
--eval_type ref_free \
--generated_n 8 \
--papercoder

Reference-Based Evaluation:

cd codes/
python eval.py \
--paper_name Transformer \
--pdf_json_path ../examples/Transformer_cleaned.json \
--data_dir ../data \
--output_dir ../outputs/Transformer \
--target_repo_dir ../outputs/Transformer_repo \
--gold_repo_dir ../examples/Transformer_gold_repo \
--eval_result_dir ../results \
--eval_type ref_based \
--generated_n 8 \
--papercoder

The tool helps verify the output quality and identify where improvements can be made. This evaluation ensures that the generated repositories are functional and faithful to the original research.