Split Large Language Models Across Multiple GPUs: A Practical Guide

Large Language Models (LLMs) excel at tasks like text generation, translation, and conversational AI. However, their massive size (hundreds of billions of parameters) demands substantial GPU resources. Splitting LLMs across multiple GPUs is crucial for efficient operation, addressing memory limitations, and boosting inference speeds. This guide details how to split and load LLMs across multiple GPUs for optimal performance.

Why Spread Your AI Wealth? The Benefits of Multi-GPU LLMs

Modern LLMs such as PaLM and Megatron-like networks often exceed the memory capacity of a single GPU (12GB, 24GB, or even 80GB). Splitting an LLM across multiple GPUs isn't just about fitting the model; it's about unlocking its true potential.

• Memory Scalability: Distributing model parameters eliminates "Out of Memory" (OOM) errors, crucial for large models during training and inference.
• Performance Gains: Parallel computation drastically improves training and inference speeds. Properly implemented, multi-GPU processing accelerates both.

Splitting LLMs is essential for advanced AI, whether performing multi-GPU deep learning on a single machine or distributed training across multiple servers.

Data Parallelism vs. Model Parallelism: Choosing Your Weapon

There are two primary methods for leveraging multiple GPUs with LLMs, each with distinct advantages:

• Data Parallelism: Replicates the entire model on each GPU. Each GPU processes a unique data subset and calculates gradients, which are then synchronized across all GPUs.
• Model Parallelism: Divides the model itself across multiple GPUs, with each GPU handling specific layers or parameters. Distribution can occur at different granularities: tensor, layer, or pipeline stage.

Diving Deeper: Types of Model Parallelism

Model parallelism offers several specialized techniques:

• Tensor Parallelism: Splits individual layer weights across GPUs. During matrix multiplication, each GPU processes a portion of the matrix independently.
• Pipeline Parallelism: Distributes different model layers across GPUs. Each GPU processes a segment of the model, creating a pipeline where data flows from one GPU to the next.
• Sharded Data Parallelism: Combines data parallelism with parameter sharding (each GPU stores only a fraction of the parameters). This reduces memory footprint while maintaining training efficiency.

Conquering the Memory Beast: Optimizing Multi-GPU Performance

GPU memory management is a key bottleneck in multi-GPU systems. Simply slicing the model across GPUs can lead to communication overhead and memory fragmentation. Careful allocation of layers, tensors, and pipeline stages is crucial for efficient multi-GPU inference with LLMs.

Key considerations:

• Batch Size: Larger batches improve GPU utilization but can trigger OOM errors. Use profiling tools (e.g., PyTorch's profiler) to identify memory hotspots.
• Activation Checkpointing: Reduces memory consumption by recomputing forward passes for selected layers during backpropagation.
• Offloading: Transfers unused GPU memory to CPU or NVMe storage. While enabling huge models, this introduces processing delays.

Frameworks to the Rescue: Powering Multi-GPU LLMs

Several open-source frameworks facilitate multi-GPU training and inference for LLMs:

PyTorch DistributedDataParallel (DDP)

DDP synchronizes gradients across multiple GPUs and nodes, a widely used approach for distributed training.

• Ease of Use: DDP handles gradient synchronization automatically.
• Scalability: Highly scalable for large clusters.
• Versatile: Supports both single-node multi-GPU and multi-node setups.

Hugging Face Accelerate

Enables seamless multi-GPU inference with minimal code changes. Automatic model sharding is achieved via the device_map="auto" parameter.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "togethercomputer/LLaMA-2-7B-32K",
    torch_dtype=torch.float16,
    device_map="auto"  # Automatically distributes across available GPUs
)

Accelerate loads the model sequentially, filling each GPU before moving to the next.

Ollama

Ollama offers efficient CPU and GPU inference capabilities. Configure GPU partitioning by setting environment variables like OLLAMA_GPU_COUNT and OLLAMA_GPU_MEMORY_LIMIT.

For example:

export OLLAMA_GPU_COUNT=5
export OLLAMA_GPU_MEMORY_LIMIT=16GB

vLLM (Versatile Large Language Model)

vLLM optimizes LLM inference with a transformer interpreter and PagedAttention for managing the KV cache. It supports distributed inference and serving via tensor parallelism.

from vllm import LLM

# Initialize model with tensor parallelism across 4 GPUs
llm = LLM(model="meta-llama/Llama-2-70b-hf", tensor_parallel_size=4)

# Generate text for multiple prompts in parallel
outputs = llm.generate(["Write a book", "Explain artificial intelligence"])

DeepSpeed

Developed by Microsoft, DeepSpeed optimizes large-scale model training using the Zero Redundancy Optimizer (ZeRO). ZeRO partitions model states across GPUs to eliminate memory redundancy​. DeepSpeed supports CPU and NVMe offloading via ZeRO-Offload and ZeRO-Infinity.

{
  "train_batch_size": 8,
  "fp16": { "enabled": true},
  "zero_optimization": {
    "stage": 3,
    "offload_param": { "device": "cpu"}
  }
}

Megatron-LM

NVIDIA's framework combines tensor parallelism with pipeline parallelism for massive parallel processing. Define tensor_parallel_size (GPU distribution per layer) and pipeline_parallel_size (model stage segmentation). Ideal for training billion-parameter models from scratch.

Scaling Beyond a Single Machine: Distributed Training Across Multiple Machines

Training LLMs across multiple machines requires a distributed environment with multiple GPUs per node. PyTorch's distributed communication backend (NCCL) facilitates seamless connections between processes.

• Master Node Setup: Designate a master node (IP address and port) to manage all other nodes.
• Rank and World Size: Assign each process/node a rank, with the total number of processes representing the world size.
• Launch: Use torch.distributed.launch or torchrun to initiate processes across nodes.
• Network Optimization: Utilize a high-bandwidth network (e.g., InfiniBand) to minimize synchronization overhead.

Distributing LLM Training with PyTorch DDP: a Lean Example

PyTorch DistributedDataParallel (DDP) replicates the model on each GPU and synchronizes gradients to simulate single-device training.

1. Initialize the Process Group

import torch.distributed as dist
dist.init_process_group(backend="nccl")

2. Configure the Device and Wrap the Model with DDP

import os
import torch
local_rank = int(os.environ["LOCAL_RANK"])
torch.cuda.set_device(local_rank)
model = MyModel().to(local_rank)
model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[local_rank])

3. Use a DistributedSampler in the DataLoader

from torch.utils.data import DataLoader, DistributedSampler
sampler = DistributedSampler(train_dataset)
train_loader = DataLoader(train_dataset, batch_size=BATCH_SIZE, sampler=sampler)

4. Implement the Training Loop

DDP handles gradient synchronization automatically. Clean up after training:

dist.destroy_process_group()

5. Launch Training with torchrun

torchrun --nproc_per_node=4 train_ddp.py

Taming the Errors: Common Multi-GPU Debugging Scenarios

Here's a breakdown of common errors in multi-GPU LLM deployment, their causes, and solutions: