Build AlexNet from Scratch with PyTorch: A Step-by-Step Guide

Dive into deep learning by building AlexNet from scratch using PyTorch. This guide provides a practical, hands-on approach to understanding and implementing one of the most influential convolutional neural networks (CNNs) in computer vision history. Learn how to construct each layer, train the network, and evaluate its performance using the CIFAR-10 dataset.

Why Build AlexNet from Scratch?

Understand the inner workings of CNNs and gain practical experience with PyTorch. Learning by doing is the most effective way to master deep learning concepts.

• Solidify Understanding: Build a deep understanding of convolutional neural networks and their architectural components.
• Practical Experience: Gain hands-on experience in implementing and training a state-of-the-art CNN using PyTorch.
• Customization Skills: Learn the skills necessary to modify and adapt existing architectures for specific tasks.

AlexNet Architecture: A Deep Dive

AlexNet, a groundbreaking CNN, achieved state-of-the-art results in image classification. Learn the key components of AlexNet.

• Input: 3-channel images (224x224x3).
• Convolutional Layers: Using kernels of sizes 11x11, 5x5, and 3x3.
• Pooling: Utilizes max pooling for subsampling.
• Activations: ReLU activations are used throughout the network to introduce non-linearity.

Prerequisites: Essential Knowledge

Before diving into the code, ensure you have a baseline understanding of these concepts. A solid grasp of these topics will accelerate your success in implementing AlexNet.

• Neural Networks: Familiarity with layers (input, hidden, output), activation functions, optimization algorithms, and loss functions.
• CNNs: Understanding of convolutional layers, pooling layers, stride, padding, and kernel/filter size.
• Python & PyTorch: Proficiency in Python syntax and the PyTorch library is crucial for understanding the code.

Loading and Preparing the CIFAR-10 Dataset for AlexNet

Use the CIFAR-10 dataset, consisting of 60,000 32x32 color images across 10 classes, to train AlexNet. Proper data pre-processing significantly impacts model performance.

• Dataset Structure: 50,000 training images and 10,000 test images.
• Data Augmentation: To enhance robustness, apply random crops and horizontal flips to the training data.
• Normalization: Normalize the data using pre-calculated mean and standard deviation for each color channel.

Importing Libraries: Setting the Stage

Import necessary libraries such as NumPy, PyTorch, and Torchvision. Set the device (cuda if available, otherwise cpu).

import numpy as np
import torch
import torch.nn as nn
from torchvision import datasets
from torchvision import transforms
from torch.utils.data.sampler import SubsetRandomSampler

# Device configuration
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

Loading the CIFAR-10 Dataset with PyTorch

Use torchvision to load and preprocess the CIFAR-10 dataset efficiently. Employ data loaders for streamlined batch processing.

1. Define Data Loaders: Functions get_train_valid_loader and get_test_loader handle loading and preprocessing.
2. Normalization: Normalize the data using the mean and standard deviation of each color channel.
3. Data Augmentation: Apply transformations such as random cropping and horizontal flipping to the training dataset.
4. Train/Validation Split: Divide the training data into training (90%) and validation (10%) sets.
5. Data Loaders: Use PyTorch's DataLoader to manage batching, shuffling, and loading data efficiently.

def get_train_valid_loader(data_dir,
 batch_size,
 augment,
 random_seed,
 valid_size=0.1,
 shuffle=True):
 # Define transforms
 # Load the dataset
 # Split into Train and Validation
 # Create DataLoaders
   return (train_loader, valid_loader)

def get_test_loader(data_dir, batch_size, shuffle=True):
 # Define transform
 # Load the dataset
 # Create DataLoader
    return data_loader

Building AlexNet from Scratch using PyTorch: Step-by-Step

Define the AlexNet class inheriting from nn.Module. Construct the layers in __init__ and define the forward pass in the forward method.

1. Convolutional Layers: Use nn.Conv2d to define convolutional layers with appropriate kernel sizes, strides, and padding.
2. Max Pooling: Apply nn.MaxPool2d for down-sampling.
3. ReLU Activation: Introduce non-linearity with nn.ReLU.
4. Fully Connected Layers: Use nn.Linear for fully connected layers, along with dropout (nn.Dropout) for regularization.

class AlexNet(nn.Module):
    def __init__(self, num_classes=10):
        super(AlexNet, self).__init__()
        # Define layers
    def forward(self, x):
        # Define forward pass
        return out

Setting Hyperparameters and Initializing the Model for CIFAR-10

Define hyperparameters like the number of epochs, batch size, and learning rate. Instantiate AlexNet and define the loss function and optimizer.

num_classes = 10
num_epochs = 20
batch_size = 64
learning_rate = 0.005

model = AlexNet(num_classes).to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, weight_decay = 0.005, momentum = 0.9)

Training AlexNet: The Learning Process

Iterate through epochs and batches. Perform the forward pass, calculate the loss, and update the model's weights using backpropagation.

1. Move Data to Device: Transfer images and labels to the GPU (if available).
2. Forward Pass: Compute the model's predictions.
3. Calculate Loss: Determine the difference between predictions and actual labels using the cost function.
4. Backward Pass: Update model weights to minimize the loss.
5. Validation: Evaluate the model's accuracy on the validation set after each epoch.

for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):
        # Move tensors to the configured device
        # Forward pass
        # Backward and optimize
        # Print training progress
        
    # Validation

Testing AlexNet: Performance Evaluation

Evaluate the trained model on the test dataset to measure its generalization performance.

with torch.no_grad():
    correct = 0
    total = 0
    for images, labels in test_loader:
        # Move tensors to the configured device
        # Forward pass
        # Calculate predictions
        # Calculate accuracy

    print('Accuracy of the network on the {} test images: {} %'.format(10000, 100 * correct / total))

Conclusion: What You've Achieved Building AlexNet

You've successfully implemented and trained AlexNet from scratch using PyTorch. This hands-on experience has solidified your understanding of CNNs and deep learning principles. From loading data to training and testing, you've gained practical skills that will empower you to tackle a wide range of computer vision tasks.