2026-07-31 · ci · testing · gradle

the green build that ran nothing

spectroscope shipped five releases with no CI. Wiring one up meant confronting a build that reports success in half a second having executed zero tests, and deciding that a tick is not evidence unless a number stands next to it.

A passing tick is a claim like any other, and the cheapest evidence you can attach to it is a count. Until you attach one you are trusting a colour.

The other half of this story is that "works on my machine" has a precise mechanism. A developer machine accumulates state a fresh checkout does not have, and a clean runner is the only thing that reliably takes that state away.

spectroscope went out five times, 0.1.0 through 0.4.1, entirely by hand against the nine steps in docs/RELEASE-PLAYBOOK.md. There was no .github directory. The only automation in the estate was Cloudflare's git build on the website repos, where a push is a deploy. Card 125 opened to fix that, and the first thing it had to deal with was not a workflow at all.

the half second

On this tree, this command is a lie:

./gradlew test

It reports BUILD SUCCESSFUL in about half a second, having executed zero tests. Gradle's up-to-date checks look at unchanged inputs and skip the task whole. Even cleanTest test comes back FROM-CACHE. The behaviour is documented in the project's own CLAUDE.md, and it had already misled a human here at least once.

That fact decides whether a pipeline is worth building at all. A CI that runs ./gradlew test on every commit is actively harmful: it produces a green badge that means nothing, forever, and it does so with more authority than a developer's local run because it looks institutional. So the flags are the whole point, and the workflow says so where somebody editing it will read it. From .github/workflows/gate.yml, lines 50 to 60:

      # --rerun-tasks --no-build-cache is NOT optional. On a warm tree a plain
      # `./gradlew test` reports BUILD SUCCESSFUL in half a second having
      # executed zero tests (up-to-date checks skip the whole task, and even
      # `cleanTest test` comes back FROM-CACHE). These flags force every test
      # to execute on every run. The javadoc legs are here because javadoc
      # otherwise only runs at release, which is the one place a broken doc
      # comment is expensive to discover.
      - name: Java gate
        run: >
          ./gradlew test --rerun-tasks --no-build-cache
          :spectro-core:javadoc :spectro-orchestrator:javadoc

The javadoc legs ride along for the reason the comment gives. Card 87 once orphaned three doc comments from their stream() methods, and nothing noticed until the release gate ran javadoc, which is the most expensive possible moment to find out.

the tick is not the measurement

Comments do not enforce anything. Somebody will eventually simplify that command, so neither job trusts its own exit status alone. Both extract the real test count and fail on zero.

The Java side sums the attributes over every JUnit XML result file and writes the totals into the job summary, so a reviewer sees JUnit: 1162 tests, 0 failures, 0 errors, ... rather than a tick that hides it. The web side reads vitest's summary line out of the captured gate log, and that is where the pipeline nearly acquired its own silent failure.

vitest prints a fixed-format line on pass, Tests 1841 passed (1841). A naive grep for that is right on a laptop and wrong on a runner. Here is the same line pulled out of a real CI log, rendered through cat -v so the escapes are visible:

^[[2m      Tests ^[[22m ^[[1m^[[32m1841 passed^[[39m^[[22m^[[90m (1841)^[[39m

The gate captures that through a pipe, and it still arrives coloured, with the escapes sitting between the word and the digits. An anchored grep for Tests followed by a number never matches it, so the count would have come back empty and the guard would have failed every green run. It was caught before the first push by reproducing against a real CI=true vitest log, and the extraction strips the escapes first:

line="$(sed $'s/\x1b\\[[0-9;]*m//g' "$log" 2>/dev/null | grep -E '^[[:space:]]*Tests[[:space:]]+[0-9]+ passed' | tail -n 1 || true)"
count="$(printf '%s\n' "$line" | grep -oE '[0-9]+' | head -n 1 || true)"

The guard's job is narrow. The count is a reported measurement; the pass/fail authority stays with the gate's exit status. A count cannot tell you the tests are good. It tells you they ran, which happens to be the one lie this tree is capable of telling.

proving the gate by breaking it

An untested test harness is a contradiction, so card 125's acceptance criteria were written as things to break rather than things to observe. There are seven. Four could only be settled on GitHub itself, and those four have now run.

PR #4 carried two tests deliberately flipped, one Java (NotesServerRealProcessTest:74) and one web (format.test.ts). It went red on exactly those two and nothing else, with zero infrastructure errors in either log. main is green with the real counts in the job summary, JUnit 1162 and vitest 1841, and both count guards fail on zero. A deliberately broken doc reference planted at Spectro.java:31 killed the javadoc leg with error: reference not found, which is precisely how card 87's break would have been caught a release earlier.

what a clean machine knew that this one did not

The pipeline's first execution was red in 43 seconds, and it kept finding things into the next morning. Seven defects, not one of which this machine could see.

Two arrived together in the very first run (6842506). native/spectro-pty.c did not compile on Linux: glibc hides kill, sigaction and usleep behind feature-test macros under strict -std=c11, and Darwin exposes them regardless, so every macOS build had been silent about it. The fix is two lines that have to sit above every include:

#define _XOPEN_SOURCE 700
#define _DEFAULT_SOURCE

The second is the better story. spectro-web's drift tests import node:fs, but @types/node was nobody's declared dependency. tsc had been resolving the types sideways out of spectro-desktop/node_modules, a directory that exists on a developer machine and not in a fresh checkout. As the commit message puts it:

Locally even `tsc -b --force` was green off the borrowed types.

That is the shape of the whole category. The local build was not passing by luck, it was passing by borrowing, and no amount of forcing a rebuild could reveal it because the borrowed thing was still there.

Two more were the PTY helper meeting Linux naming. The build script's verify pattern only knew Darwin's /dev/ttysNNN, so a helper whose child correctly answered /dev/pts/0 failed its own build (1afdb68), and SpectroPtyHelperTest awaited the same Darwin-only substring and burned a ten second timeout on a child that had a real terminal (0a6614c). Neither weakened a check: none of the probe's reachable failure strings contain either substring, which was measured rather than assumed.

The fifth was a genuine race in a test, not the server. aBlankModelSwitchTakesTheTargetsDefaultNotTheCarriedModel made two back-to-back WebSocket sendText calls and discarded both futures. The JDK contract completes the second exceptionally while the first is in flight, so on a starved CPU one frame was silently never transmitted. Reproduced 3 times in 300 on a 0.5-CPU container with the exact CI signature, and a dead-port run passed, which ruled out the obvious reachability theory.

The last two came off the same runner over two days and wanted opposite responses. Three ProcessBusWireTest failures were choreography inside the tests. A fourth failure, in a test that had been in the suite since the bus was written, was a real and silent data-loss bug in the message bus. Both surfaced only because a shared two-vCPU runner kept losing the schedule lottery, which is what makes the pair worth its own post.

One red is still unexplained. SpeechRendererTest.abortClearsTheQueueAndStopsThePlayer failed once, on 31 July, and has been green on every run since. Nobody has touched that file. It sits on the list, unfixed and uncounted.

the lesson

All seven hid here for the same kind of reason. This machine has a warm Gradle cache, a sibling node_modules, a libc that hands out the POSIX surface without being asked, and fourteen cores against the runner's two. A fresh checkout on a shared runner has none of that, and every one of them was quietly holding something up.

The pipeline's value was never the badge. It was that a clean environment disagrees with a working one, and the disagreement is where the bugs live. The part that made the disagreement legible was the smallest piece of the whole thing: printing the number next to the tick.