Elastic CI on Spot VMs: a build note

The short version: you can get CI capacity that scales to zero when idle and costs near spot rates, using nothing exotic — a Jenkins controller, the Google Compute Engine plugin, and a golden VM image. The full case study is over in Dynamic Jenkins agents on GCP Spot VMs; this is the shorter builder’s note.

What I actually run

One long-lived Jenkins controller, and everything else disposable. A queued build provisions a fresh Spot VM from a pre-baked image, runs, and the VM is destroyed on completion. No pool, no idle machine, no cleanup cron — about 1,400 single-use agents a month.

pipeline {
  agent { label 'spot' }   // fresh single-use Spot VM per build
  options { retry(2) }      // preemption re-queues onto new capacity
  stages {
    stage('build') { steps { sh './ci/build.sh' } }
    stage('test')  { steps { sh './ci/test.sh'  } }
  }
}

The one decision that made it fast

Bake the toolchain into the image. My first cut installed dependencies in a boot-time startup script, and it failed under load with the GCE plugin’s least useful error — “Agent failed to connect, even though the launcher didn’t report it.” The install was racing the agent-launch timeout (and aborting on transient apt errors under set -euo pipefail), so the VM came up but never became a usable agent.

Moving everything into a golden image — provision an Ubuntu builder once, snapshot the disk into an image family the templates track — dropped boot to seconds with zero runtime apt. Baking the Go module cache and a pre-compiled linter into the same image killed the last flaky first-build failures.

Why preemption stopped being scary

Spot VMs get reclaimed. That sounds disqualifying for CI until you notice CI jobs are already retry-safe — a build is a pure function of the commit. So a preempted build just re-queues onto fresh capacity, and the developer sees a slightly longer build rather than a red one. In practice it barely fires: zero preemptions across ~1,400 agent starts in the last 30 days — but the retry path means a bad Spot day degrades to slower, never broken.

If you want the architecture, the diagram, and the cost breakdown in full, read the case study.