Cut GitHub Actions CI/CD Costs by Up to 58% & Double Deployment Speed: A Practical Guide

Dive into a cost-effective strategy for running GitHub Actions with self-hosted runners on Amazon EKS. Learn how to leverage auto-scaling for optimized performance and resource utilization. This guide will show you how to deploy and test your own solution.

Why Self-Hosted GitHub Actions Runners on EKS?

Discover significant savings and performance boosts by using self-hosted runners on Kubernetes. This approach offers a robust and scalable infrastructure for CI/CD. Free up resources and increase project velocity by reducing deployment times.

The Power of Self-Hosted Runners on Kubernetes (K8s)

Understand the core concepts of self-hosted runners and runner scale sets. Learn how they work within a Kubernetes environment to manage your CI/CD workloads efficiently. Optimize your infrastructure with properly configured runner scale sets to make the most of your CI/CD pipeline.

Key Concepts Defined:

• Runners: Ephemeral machines (servers, VMs) executing workflow jobs. Use self-hosted for customization, or GitHub-hosted for simplicity.
• Runner Scale Sets: Logical groups of runners with identical configurations. Essential for managing runner capacity.
• Runner Scale Sets Naming: Addressable by name. When you need to specify that name of runner scale set in the runs on: property of GHA to assign that workflow on on a particular runner scale set.
• Endpoints: ARC talks to two endpoints api.github.com and pipelines.actions.githubusercontent.com. Configure your firewall, proxies, and NAT gateway to allow these for ARC controller.

Understanding the ARC Controller

The Actions Runner Controller (ARC) intelligently manages runners. It dynamically allocates runners based on workflow demand, optimizing resource usage and costs. Use the ARC Controller to balance efficiency and responsiveness in your CI/CD pipeline.

ARC Components:

• Controller Manager: Manages resources like the AutoScalingListener, ensuring desired count and state.
• Runner ScaleSet Listener: Decides how many runners to create, with one listener pod per runner scale set.

Terraform Code Explained: EKS Auto-Scaling for GitHub Runners

Step through the infrastructure-as-code (IaC) setup using Terraform. Deploy self-hosted GitHub Actions runners on Amazon EKS with auto-scaling. Learn how the code provisions and manages the necessary AWS resources.

eks-auto-self-hosted-runners/
├── README.md
├── architecture/
├── commit_log.txt
├── scripts/
└── terraform/
    ├── base/
    └── modules/
        ├── arc/
        ├── eks/
        ├── karpenter_config/
        └── vpc/

This repository allows you to:

• Deploy a customized EKS cluster with auto-scaling.
• Setup GitHub Actions Runner Controller (ARC).
• Configure auto-scaling runner sets.
• Supports Docker-in-Docker (DinD) runners.

Essential Steps to Deploying Self-Hosted Runners on EKS Auto Using Terraform

From VPC setup to controller deployment, each step is designed to optimize efficiency and security.

1. VPC Configuration: Creates the virtual network with public and private subnets.
2. EKS Cluster Setup: Provisions the managed Kubernetes service.
3. Karpenter Auto-Scaling: Leverages Karpenter for efficient EC2 spot instance provisioning.
4. ARC Deployment: Deploys the GitHub Actions Runner Controller via helm.
5. GitHub Authentication: Configures authentication using a GitHub App.
6. Namespace Management: Organizes resources into dedicated namespaces.
7. Cleanup Handling: Ensures proper resource cleanup during destroy process with the cleanup script.

Configure EKS Auto's Pre-Installed Karpenter

EKS Auto comes with Karpenter pre-installed, we leverage Karpenter to provision and auto scale Ec2 spot instances from our desired Ec2 type, capacity and configuration for our GHA runner's compute.

Key configuration parameters include:

• Instance types: m7a family with 8 CPUs
• Capacity type: Spot instances for cost savings
• Storage: 300GB with 5000 IOPS
• Availability zones: us-east-1a and us-east-1b

controller config for on demand node

nodeSelector: 
  karpenter.sh/nodepool: system 
tolerations: 
- key: " CriticalAddonsOnly"
  operator: " Exists"

listener config for on demand node

listenerTemplate: 
  spec: 
    nodeSelector: 
      karpenter.sh/nodepool: system 
    tolerations: 
    - key: " CriticalAddonsOnly"
      operator: " Exists"

Verify & Test Your Setup: Ensuring Optimal Performance

Explore practical methods for testing and validating your self-hosted runner setup. Use sample GitHub Actions workflows to confirm the functionality and performance of your deployment. Ensure reliability and efficiency in your CI/CD pipelines using these hands-on test scenarios.

Test Scenarios:

• Simple Test: Ensures basic workflow execution.
• Concurrent Job Test: Evaluates the system's ability to handle multiple jobs simultaneously.
• DinD Job Test: Verifies Docker-in-Docker functionality for jobs needing container builds.

GitHub Large Hosted Runner vs. EKS Auto Runners: A Head-to-Head Comparison

Compare the performance, speed, and cost of GitHub's Large Hosted Runners against running self-hosted runners on EKS Auto. The results highlight the advantages of the EKS Auto solution with up to 58% savings.

Real-World Cost Savings:

• EKS Auto Solution: Approximately $530.93 per month (1 hour of concurrent CI workloads per day).
• GitHub Large Runner: $731.20 per month (1 hour of concurrent CI workloads per day).

Potential Savings: 27%

If you increase to 2 hours of concurrent CI workloads per day then you get savings of 58.6%.

Performance Advantages:

• EKS Auto: Maintained a stable execution time of 1 minute 8 seconds.
• GitHub Large Runners: Showed variability in execution times under concurrent loads.

Architectural Benefits: Aligning with AWS Well-Architected Framework

Discover how this solution aligns with the AWS Well-Architected Framework, and understand, how it ensures robust CI/CD processes.

• Operational Excellence: IaC ensures consistent deployments; auto-scaling eliminates manual management.
• Security: Runners operate in isolated Kubernetes pods with defined security.
• *Reliability: Failed runners are automatically replaced, ensuring workflow reliability.