Dynamic Jenkins agents on GCP Spot VMs

Why move off an always-on runner

The starting point was a single self-hosted GitHub Actions runner — an always-on 8-vCPU VM. It had the quiet cost problem every fixed runner has: you pay for it 24/7 whether it’s building or idle, and peak concurrency is capped at that one machine, so a second build just waits. CI load is bursty by nature, so most of that 24/7 spend bought nothing.

I wanted the opposite model: capacity that appears when a build is queued and disappears when it finishes, priced near spot rates. GCP Spot VMs are the compute; the interesting engineering is making agents disposable and making preemption a non-event.

How it works

The Jenkins controller is the only long-lived component. Builds come from a GitHub multibranch pipeline (one per branch/PR). When a job is queued, the Google Compute Engine plugin provisions a fresh Spot VM from a pre-baked golden image, the agent attaches, runs the build, and the VM is deleted on completion. No pooled agents, no idle machines.

Two instance templates back the fleet — a smaller default (2 vCPU) and a larger one (4 vCPU) for heavier jobs — both Spot. In a typical month the plugin churns through ~1,400 single-use agents (about 47 a day).

The decision that actually mattered: bake a golden image

The first version installed the toolchain in a boot-time startup script. It worked in testing and failed under real load with the GCE plugin’s most unhelpful error:

Agent failed to connect, even though the launcher didn’t report it.

The cause wasn’t the launcher. The startup script was doing a full toolchain install at boot — slow, and worse, it ran set -euo pipefail, so a single transient apt hiccup aborted the whole thing. Either way the install raced the agent-launch timeout, and the VM came up without ever becoming a usable agent. On Spot capacity, where you want boots measured in seconds, at-boot provisioning is exactly wrong.

The fix was to move all of that out of the critical path into a golden image: boot an Ubuntu builder, run the same provisioning script once, snapshot the disk into an image family, and point the instance templates at the family so the newest image is picked up automatically on the next agent boot. Now agents boot in seconds with zero runtime apt.

One detail earned its keep: the image also pre-warms the Go module cache and compiles golangci-lint ahead of time. Cold Spot agents were hitting the lint step’s timeout on first run while modules downloaded; baking that in removed the last source of flaky first-build failures.

# Re-baking is a script, not a ceremony: provision once, snapshot to a family,
# recreate the templates on the new image, recycle agents.
gcloud compute images create "jenkins-agent-$(date +%Y%m%d-%H%M)" \
  --source-disk "$BUILDER" --source-disk-zone "$ZONE" \
  --family jenkins-agent      # templates track the family → newest image auto-used

Treating preemption as normal

Spot VMs can be reclaimed at any time. The instinct is to guard against it; the better move is to expect it. Because a CI build is a pure function of the commit, a preempted build simply re-queues onto fresh capacity — the developer sees a slightly longer build, never a red one.

pipeline {
  agent { label 'spot' }   // fresh single-use Spot VM per build
  options { retry(2) }      // preemption re-queues onto new capacity
  stages {
    stage('build') { steps { sh './ci/build.sh' } }
    stage('test')  { steps { sh './ci/test.sh'  } }
  }
  // No VM cleanup step — the controller deletes the agent on completion.
}

Worth being honest about the payoff of that design: in the last 30 days there were zero recorded preemptions across ~1,400 agent starts. Spot capacity in the region has been stable, so the retry path rarely fires — but it costs nothing to have, and it means a bad Spot day degrades to slower, never broken.

What it cost, and what it proves

Estimated on a GCP list-price basis: the old always-on 8-vCPU runner billed on the order of a couple hundred USD a month just to exist. The Spot fleet only bills for actual build time — roughly 1,400 builds a month at ~10–23 minutes each, on 2–4 vCPU Spot instances — which lands well under that, a rough 5–10× reduction in CI compute cost. The bigger win is structural: concurrency is no longer capped at one machine, so builds stop queueing behind each other.

The pattern generalizes, too — the same controller also drives AWS Spot agents via EC2 Fleet, so the “disposable agent, expect preemption” model isn’t GCP-specific. None of it is exotic: a Jenkins controller, the GCE plugin, a golden image, and the discipline to make agents disposable.

There’s a shorter companion write-up of the setup — see the build note.