Elastic CI on Spot VMs: a build note

The short version: you can get CI capacity that scales to zero when idle and costs near spot rates, using nothing exotic — a Jenkins controller, the GCE plugin, a Packer-baked image, and Terraform. The full case study is over in Dynamic Jenkins agents on GCP Spot VMs; this is the shorter builder’s note.

What I actually stood up

One long-lived Jenkins controller, and everything else disposable. A queued build provisions a fresh Spot VM from a pre-baked image, runs, and the VM is destroyed on completion. There is no pool, no idle machine, no cleanup cron.

pipeline {
  agent { label 'gcp-spot' }   // fresh Spot VM per build
  options { retry(2) }         // preemption re-queues onto new capacity
  stages {
    stage('build') { steps { sh './ci/build.sh' } }
    stage('test')  { steps { sh './ci/test.sh'  } }
  }
}

The one decision that made it fast

Bake the toolchain into the image. My first cut installed dependencies in a startup script, and agents took minutes to become useful — which defeats the whole point, because you feel that latency on every build. Moving the toolchain into a Packer image dropped attach time to under a minute and made per-build provisioning feel instant. TODO(hung): drop the before/after attach numbers here.

Why preemption stopped being scary

Spot VMs get reclaimed. That sounds disqualifying for CI until you notice CI jobs are already retry-safe — a build is a pure function of the commit. So a preempted build just re-queues onto fresh capacity, and the developer sees a slightly longer build rather than a red one. Expecting preemption, instead of guarding against it, is what makes the cost model work.

If you want the architecture, the diagram, and the tradeoffs in full, read the case study.