Unlock Unprecedented Reasoning: Introducing LUFFY, the Reinforcement Learning Game-Changer

Tired of AI models that struggle with complex reasoning? LUFFY, a revolutionary reinforcement learning framework, is here to bridge the gap between zero-RL and imitation learning. By intelligently incorporating off-policy reasoning traces, LUFFY achieves state-of-the-art results, especially in out-of-distribution generalization. Discover how LUFFY can transform your AI projects!

Supercharge Your AI with Off-Policy Guidance

LUFFY leverages external reasoning traces to guide the learning process, effectively bootstrapping from the insights of stronger models like DeepSeek-R1. This off-policy guidance is a key differentiator, allowing LUFFY to learn faster and achieve superior results.

• Leverage insights from superior models.
• Accelerate learning through targeted guidance.
• Overcome the limitations of zero-RL methods.

Dynamic Learning: Explore AND Imitate

LUFFY doesn't just imitate; it learns when to explore and when to leverage existing knowledge. This dynamic balance allows the model to adapt and improve throughout the training process, leading to more robust and versatile AI.

How does LUFFY strike this balance?

LUFFY intelligently combines on-policy rollouts with off-policy demonstrations during advantage estimation. This hybrid approach allows LUFFY to learn from both its own experiences and the wisdom of expert traces.

Policy Shaping: Emphasize Crucial Actions

Traditional policy gradients often overlook critical, low-probability actions. LUFFY addresses this with policy shaping via regularized importance sampling, emphasizing those vital actions often ignored. This focus leads to better generalization and a more nuanced understanding of complex tasks.

• Identify and prioritize critical actions.
• Improve generalization by focusing on overlooked elements.
• Enhance model understanding of underlying problem dynamics.

Getting Started with LUFFY: Installation Made Easy

Ready to experience the power of LUFFY? The installation process is straightforward and well-documented.

1. Environment Setup::

   conda create -n luffy python=3.10
   conda activate luffy
   

   Copy
2. Install Dependencies::

   cd luffy
   pip install -r requirements.txt
   pip install -e.
   cd verl
   pip install -e.
   

   Copy

Having trouble with flash-attn? Try this version:

wget https://github.com/Dao-AILab/flash-attention/releases/download/v2.7.3/flash_attn-2.7.3+cu12torch2.4cxx11abiFALSE-cp310-cp310-linux_x86_64.whl
pip install flash_attn-2.7.3+cu12torch2.4cxx11abiFALSE-cp310-cp310-linux_x86_64.whl

Training LUFFY: Unleash Its Potential

Follow these simple steps to train LUFFY on your data (or use the provided example):

1. Prepare the Data::

   cd data
   python prepare_train.py
   

   Copy
2. Run the Training Script::

   cd exp_scripts
   bash train.sh
   

   Copy

LUFFY in Action: Inference Examples

Once trained, using LUFFY for inference is just as easy. Explore the examples provided to get a feel for how LUFFY can be integrated into your projects.

Outperforming the Competition: Evaluation Results

LUFFY shines in evaluations, achieving state-of-the-art results among zero-RL methods on competition-level benchmarks. It consistently outperforms both on-policy RL and imitation learning (SFT), demonstrating its superior generalization capabilities and adaptability.

Generalization to Out-of-Distribution Tasks

Specifically, LUFFY shows impressive generalization abilities on out-of-distribution tasks, with an average gain of over +6.2 on ARC-C, GPQA, and MMLU-Pro.

Reproduce Our Results: Step-by-Step Guide

Want to verify LUFFY's performance yourself? Follow these steps to reproduce our results:

ROOT=YOUR_ROOT_PATH
DATA=$ROOT/data/valid.all.parquet
OUTPUT_DIR=./results/
mkdir -p $OUTPUT_DIR

# Configure model path and name
MODEL_PATH=Elliott/LUFFY-Qwen-Math-7B-Zero
MODEL_NAME=luffy

# Set the template
if [ $MODEL_NAME == "eurus-2-7b-prime-zero" ]; then
    TEMPLATE=prime
elif [ $MODEL_NAME == "simple-rl-zero" ]; then
    TEMPLATE=qwen
else
    TEMPLATE=own
fi

# Run evaluation
CUDA_VISIBLE_DEVICES=0,1,2,3 python eval_scripts/generate_vllm.py \
    --model_path $MODEL_PATH \
    --input_file $DATA \
    --remove_system True \
    --output_file $OUTPUT_DIR/$MODEL_NAME.jsonl \
    --template $TEMPLATE > $OUTPUT_DIR/$MODEL_NAME.log

Ready to Transform Your AI?

LUFFY is more than just a framework; it's a catalyst for AI innovation. By combining the best of zero-RL and imitation learning, LUFFY empowers you to build more intelligent, adaptable, and robust AI models. Unlock the next level of AI performance with LUFFY today!