Dynamic Jenkins agents on GCP Spot VMs

Why move off always-on runners

Self-hosted CI runners have a quiet cost problem: you pay for the machine whether it’s building or idle, and your peak concurrency is capped at whatever fleet you pre-provisioned. Teams respond by over-provisioning for the busy hour, then paying for that headroom around the clock. The economics get worse the more the CI usage is bursty — which is exactly the usage pattern of most engineering orgs.

I wanted to prove the opposite model end-to-end: capacity that appears when a build is queued and disappears when it finishes, priced near spot rates. This POC is that model with GCP Spot VMs as the compute, built to survive the one thing everyone worries about — preemption.

How it works

The Jenkins controller is the only long-lived component. When a job is queued, the GCE plugin provisions a Spot VM from a pre-baked image, the agent attaches over SSH/JNLP, runs the build, and the VM is deleted on completion. No pooled agents, no idle machines.

The two decisions that made it viable

Bake the image, don’t provision at boot. The first version installed the toolchain in a startup script. Agents took minutes to become useful, which kills the elastic model — you feel the latency on every single build. Moving the toolchain into a Packer-baked image dropped attach time dramatically and made per-build provisioning feel instant. TODO(hung): insert the before/after attach-time numbers.

Treat preemption as normal. Spot VMs can be reclaimed at any time. The instinct is to guard against it; the better move is to expect it. Because CI jobs are inherently retry-safe (they’re pure functions of the commit), a preempted build simply re-queues onto fresh capacity. The controller never surfaces preemption as a red build — the developer sees a slightly longer build, not a failure.

// Pipeline agents are disposable and stateless by design.
pipeline {
  agent { label 'gcp-spot' }        // provisions a fresh Spot VM per run
  options { retry(2) }              // preemption re-queues onto new capacity
  stages {
    stage('build') { steps { sh './ci/build.sh' } }
    stage('test')  { steps { sh './ci/test.sh'  } }
  }
  // No post-build cleanup of the VM needed — the controller deletes it.
}

What this proves

Elastic, near-spot-priced CI is achievable with boring, public primitives — a Jenkins controller, the GCE plugin, a Packer image, and Terraform to wire it together. The interesting engineering isn’t exotic; it’s the discipline of making agents disposable and making preemption a non-event.

There’s a companion write-up that walks through the POC setup step by step — see the short build note.