Skip to content

Self-Hosted GitLab Runners on EKS

Investigation into hosting GitLab CI runners on the Recro EKS cluster to reduce CI/CD costs by using our own compute instead of GitLab's shared runners.

Status: On Hold

The GitHub-to-GitLab mirroring approach requires a GitLab Premium or Ultimate plan. The free/trial tier does not support pulling from external repositories as a CI trigger. The GitLab trial has been shut down, but the org is retained for future use if Recro migrates to GitLab entirely.

Goal

Run GitLab CI pipelines on Recro-managed EKS infrastructure so that:

  • CI jobs use our existing EKS compute (cost savings over GitLab shared runners)
  • Jobs inherit AWS credentials via IRSA (no static secrets)
  • ArgoCD manages the runner deployment (GitOps pattern)
  • IaC changes (Terraform plan/apply) could run in GitLab CI with full AWS access

Architecture

GitHub (source of truth)
  │
  ▼  mirror / pull
GitLab (CI engine)            ◄── requires Premium/Ultimate
  │
  ▼  dispatches jobs
GitLab Runner (Helm on EKS)
  │
  ▼  IRSA
AWS IAM Role (AdministratorAccess)

Components Built

Component Repository Path / Branch Status
IAM Role (IRSA) recro-aws-iac terraform/modules/eks/iam.tf Merged & applied
EKS Access Entry recro-aws-iac terraform/modules/eks/main.tf Needed
ArgoCD Application eks-gitops argocd/applications/gitlab-runner.yaml On main
Helm values eks-gitops workloads/gitlab-runner/values.yaml On main
Service Account (IRSA) eks-gitops workloads/gitlab-runner/serviceaccount-jobs.yaml On main
CI Dockerfile recro-aws-iac ci/Dockerfile On main
.gitlab-ci.yml recro-aws-iac .gitlab-ci.yml On main

IAM Role

An IRSA role was created for GitLab Runner job pods:

  • Role name: recro-eks-gitlab-runner
  • Trust: EKS OIDC provider, scoped to system:serviceaccount:gitlab-runner:gitlab-runner-jobs
  • Permissions: AdministratorAccess (needed for Terraform apply)

This was merged via PR #8 and applied with the 2026-02-03 tag.

Note: An EKS Access Entry (AmazonEKSClusterAdminPolicy) for this role still needs to be created in main.tf if the runner needs Kubernetes API access for Terraform operations against EKS.

GitLab Runner Deployment

The runner is deployed via ArgoCD using the official gitlab-runner Helm chart (v0.71.0):

  • Namespace: gitlab-runner
  • Executor: Kubernetes (jobs run as pods)
  • Job image: ubuntu:22.04
  • Node selector: node-type: general
  • Service account: gitlab-runner-jobs with IRSA annotation

Manual Setup Required

After ArgoCD creates the namespace, a runner token secret must be created manually:

kubectl create secret generic gitlab-runner-token \
  --namespace gitlab-runner \
  --from-literal=runner-registration-token='' \
  --from-literal=runner-token='glrt-YOUR_TOKEN_HERE'

CI Pipeline Design

A .gitlab-ci.yml was written for the recro-aws-iac repo with two stages:

Stage Trigger Actions
validate Push to main make status, make init, make plan
release Date-based tag (e.g. 2026-02-04) make status, make init, make plan, make apply

Jobs use a custom Docker image (ghcr.io/recro/terraform-ci:latest) with AWS CLI, Terraform 1.9.8, and standard build tools. The Dockerfile lives at ci/Dockerfile in the recro-aws-iac repo.

Blocker: GitLab Licensing

Pulling from a GitHub repository into GitLab (repository mirroring) requires GitLab Premium or Ultimate. On the free tier and trial, this feature is not available, which means:

  • Cannot automatically mirror GitHub repos into GitLab
  • Cannot trigger GitLab CI from GitHub pushes without the mirror feature
  • Manual pushes to GitLab would work but defeats the purpose

Decision

Go ahead and jot down your findings in the cog wiki. Just shut down the trial but keep the org so we can use it later. I still see us moving to GitLab entirely in the future but not right now, that's a heavy lift and lot of effort for low value in the short term.

Action items:

  1. Shut down the GitLab trial (keep the org)
  2. Leave the EKS infrastructure in place (IAM role, ArgoCD app)—low cost when idle
  3. Revisit when/if Recro migrates fully to GitLab

Resuming This Work

If Recro moves to GitLab in the future:

  1. Activate a GitLab Premium plan on the existing org
  2. Configure repository mirroring from GitHub
  3. Add EKS Access Entry for the recro-eks-gitlab-runner IAM role in main.tf (if Terraform needs K8s API access)
  4. Create a runner registration token in GitLab and apply the secret (see Manual Setup above)
  5. ArgoCD will sync the runner deployment automatically
  6. Verify the runner registers and picks up jobs

See Also