Unleash Your Inner Composer with NotaGen: AI-Powered Sheet Music Generation

Are you ready to revolutionize how music is created? NotaGen is a groundbreaking AI model that uses Large Language Models (LLMs) to generate high-quality classical sheet music. Whether you're a seasoned composer or just starting, NotaGen opens up new possibilities for musical expression. Read on to discover how NotaGen can transform your musical journey.

What is NotaGen?

NotaGen is a symbolic music generation model leveraging the power of LLMs. It's designed to produce exceptional classical sheet music through a unique three-stage training process:

1. Pre-training: Trained on a massive dataset of 1.6 million musical pieces.
2. Fine-tuning: Refined on approximately 9,000 classical compositions using period-composer-instrumentation prompts.
3. Reinforcement Learning: Optimized with a novel CLaMP-DPO method, requiring no human annotations or pre-defined rewards.

Why Use NotaGen for Symbolic Music Generation?

• High-Quality Output: Generate professional-grade classical sheet music.
• AI-Powered Creativity: Explore new musical ideas and styles effortlessly.
• No Human Annotation Needed: The CLaMP-DPO method streamlines the reinforcement learning process.

Getting Started with NotaGen

Setting up NotaGen is straightforward. Follow these steps:

1. Environment Setup:

   conda create --name notagen python=3.10
   conda activate notagen
   conda install pytorch==2.3.0 pytorch-cuda=11.8 -c pytorch -c nvidia
   pip install accelerate
   pip install optimum
   pip install -r requirements.txt
   

   Copy

2. Download Model Weights: Choose from pre-trained weights of different scales. Note: Pre-trained weights are not for conditional generation.

   • Fine-tuning: Download weights fine-tuned on ~9k classical pieces.
   • Reinforcement Learning: Access weights optimized with 3 iterations of CLaMP-DPO.

Introducing NotaGen-X: The Enhanced Version

Inspired by Deepseek-R1, NotaGen-X takes symbolic music generation to the next level with optimized training procedures. Improvements include:

• Post-Training Stage: Refining the model with a classical-style subset after pre-training.
• Key Augmentation Removal: Resulting in more reasonable instrument ranges in generated compositions.
• Iterative Refinement: Multiple rounds of post-training, fine-tuning, and reinforcement learning for superior output.

Demo: Experience NotaGen-X in Action

• Online Gradio Demo: Input "Period-Composer-Instrumentation" to generate music, preview audio and PDF scores, and download your creations.
• Local Gradio Demo: Deploy NotaGen-X inference locally (requires 8GB GPU memory).

Community Contributions

• Colab Notebook: Thanks to @deeplearn-art, you can try NotaGen on Google Colab.
• ComfyUI Node: @billwuhao's ComfyUI node automatically converts generated .abc files to .xml, .mp3, and .png formats.

Deep Dive: Training and Customization

Pre-training

Want to pre-train a blank NotaGen model using your own data?

1. Preprocess your data and generate data index files.

2. Modify parameters in pretrain/config.py.

3. Run:

   cd pretrain/
   accelerate launch --multi_gpu --mixed_precision fp16 train-gen.py
   

   Copy

Fine-tuning with NotaGen

Fine-tune NotaGen-large with your desired dataset (requires at least 24GB GPU memory or use NotaGen-small/medium).

1. Configuration:

   • Modify DATA_TRAIN_INDEX_PATH and DATA_EVAL_INDEX_PATH in finetune/config.py.
   • Download pre-trained weights and update PRETRAINED_PATH.
   • Set EXP_TAG to differentiate models.

2. Execution:

   cd finetune/
   CUDA_VISIBLE_DEVICES=0 python train-gen.py
   

   Copy

Reinforcement Learning (CLaMP-DPO) for Enhanced Musicality

Enhance your fine-tuned model using CLaMP-DPO.

1. CLaMP 2 Setup: Download and place CLaMP 2 and M3 model weights in the clamp2/ folder.
2. Extract Ground Truth Features:

   • Modify input_dir and output_dir in clamp2/extract_clamp2.py.

   • Run:

     cd clamp2/
     python extract_clamp2.py
     

     Copy
3. CLaMP-DPO Iteration Example:
   • Inference:

     • Modify INFERENCE_WEIGHTS_PATH and NUM_SAMPLES in inference/config.py.

     • Run:

       cd inference/
       python inference.py
       

       Copy
   • Extract Generated Data Features:
     • Modify input_dir and output_dir in clamp2/extract_clamp2.py.
     • Run the extraction script.
   • Statistics (Optional):
     • Modify parameters in clamp2/statistics.py and run the script.
   • Construct Preference Data:
     • Modify parameters in RL/data.py.
     • Run the script to generate preference data.
   • DPO Training:

     • Modify parameters in RL/config.py.

     • Run:

       cd RL/
       CUDA_VISIBLE_DEVICES=0 python train.py
       

       Copy

Ready to Create Your Symphony?

NotaGen is more than just a model; it's a musical partner. Dive in, experiment, and unlock new symphonies with the power of AI-driven music generation.