Slash CI/CD Costs by 27% & Double Deployment Speed: Run GitHub Actions on EKS Auto

Want to optimize your CI/CD pipeline? Discover how running GitHub Actions self-hosted runners on Amazon EKS Auto can significantly cut costs and boost deployment speed. This guide dives deep into a practical, real-world solution tested at scale.

Why Run GitHub Actions Self-Hosted Runners on EKS Auto?

Choosing between GitHub’s Large Hosted Runners and self-hosting on Kubernetes boils down to cost, control, and performance. As a Senior DevOps Engineer, I faced this dilemma when our CI/CD pipelines became increasingly complex. GitHub Actions on default runners were slowing down. Using GitHub Actions self-hosted runners on EKS Auto offered a compelling alternative.

Understanding the Core Concepts: Runners, Scale Sets, and ARC

Before we dive into the setup, let's clarify some key terms:

• Runners: The worker machines that execute your CI/CD jobs. They can be GitHub-hosted or self-hosted runners on your infrastructure.
• Runner Scale Sets: Logical groupings of runners with identical configurations, managed at the repository, organization, or enterprise level.
• Endpoints: ARC communicates with api.github.com and pipelines.actions.githubusercontent.com. Ensure these are allowed through your firewalls and proxies.
• ARC (Actions Runner Controller): The brains of the operation, managing the runners in your Kubernetes cluster.

ARC expertly manages runners, efficiently dynamically scaling them based on the workflow load.

Terraform Code Walkthrough: Building Your Infrastructure

This solution uses Infrastructure as Code (IaC) to deploy auto-scaling GitHub Actions self-hosted runners on Amazon EKS using GitHub's Actions Runner Controller (ARC).

Key Benefits:

• Fully managed EKS cluster with auto-scaling.
• GitHub Actions Runner Controller (ARC) setup.
• Auto-scaling runner sets based on workflow needs.
• Docker-in-Docker (DinD) runner support.

Setting Up Auto-Scaling GitHub Actions Self-Hosted Runners on Amazon EKS: A Step-by-Step Guide

Let's break down the Terraform implementation, which is structured for clarity and maintainability.

Project Structure:

• eks-auto-self-hosted-runners/: Root directory.
• terraform/: Main infrastructure code.
  • base/: Entry point for Terraform deployment.
  • modules/: Reusable Terraform modules.
    • arc/: Actions Runner Controller configuration.
    • eks/: EKS cluster configuration.
    • karpenter_config/: Node auto-scaling configuration.
    • vpc/: Network infrastructure configuration.

1. VPC Configuration

Create a VPC with public and private subnets for network isolation, ensuring proper Kubernetes integration, using a CIDR block of 10.0.0.0/16 across two availability zones.

2. EKS Cluster Setup

Deploy an EKS cluster (Kubernetes v1.31) with worker nodes placed in private subnets for enhanced security.

3. Configuring EKS Auto's Karpenter for Auto-Scaling

Leverage Karpenter, pre-installed with EKS Auto, to provision and auto-scale EC2 Spot instances for cost-effective runner compute.

Important configurations:

• Instance types: m7a family with 8 CPUs.
• Capacity type: Spot instances for cost savings.
• Storage: 300GB with 5000 IOPS.

4. Deploying Actions Runner Controller (ARC)

Deploy ARC using a Helm chart, encompassing the controller (runner lifecycle management) and runner sets (runner configurations).

Ensure Controller and runner scale set pods are deployed on on-demand EC2 instances, with ephemeral runner pods on Spot instances.

5. GitHub Authentication Setup

Authenticate ARC with GitHub using a GitHub App, storing credentials in Kubernetes secrets.

6. Namespace Management

Create dedicated namespaces for ARC components: arc-systems (controller) and arc-runners (runner pods).

7. Cleanup Handling

Implement a cleanup script to remove finalizers from Kubernetes resources during Terraform destroy.

Testing Your Setup: Real-World Examples

I've created three GitHub Actions to test different scenarios.

Simple Test

Ensure your workflow's runs-on parameter specifies the arc-runner-set label.

Concurrent Job Test

Simulate load by running multiple concurrent jobs.

DinD Job Test

Run microservices in containers for end-to-end testing, using a runner set configured for DinD workflows.

GitHub Large Hosted Runner vs. Running Runners on EKS Auto: The Showdown

Let's compare performance, speed, and cost.

Performance & Speed

Our solution has a constant execution time throughout, which ensures overall performance and consistency.

Price

The solution is proven cheaper than GitHub hosted runners, yielding a savings of 27%. Furthermore, with a 2 hour concurrent CI workload per day, the savings become around 58.6%.

Solution Architect Perspective: AWS Well-Architected Framework

This solution aligns with the AWS Well-Architected Framework:

• Operational Excellence: Terraform ensures consistent deployments, and auto-scaling runners eliminate manual management.
• Cost Optimization: Spot instances and on-demand provisioning minimize costs.
• Reliability: Auto-scaling runners with health checks guarantee reliable workflow execution.
• Security: Runners operate in isolated Kubernetes pods with defined security contexts.