The other half of this story is that "works on my machine" has a precise mechanism. A developer machine accumulates state a fresh checkout does not have, and a clean runner is the only thing that reliably takes that state away.

spectroscope went out five times, 0.1.0 through 0.4.1, entirely by hand against the nine steps in docs/RELEASE-PLAYBOOK.md. There was no .github directory. The only automation in the estate was Cloudflare's git build on the website repos, where a push is a deploy. Card 125 opened to fix that, and the first thing it had to deal with was not a workflow at all.

the half second

On this tree, this command is a lie:

./gradlew test

It reports BUILD SUCCESSFUL in about half a second, having executed zero tests. Gradle's up-to-date checks look at unchanged inputs and skip the task whole. Even cleanTest test comes back FROM-CACHE. The behaviour is documented in the project's own CLAUDE.md, and it had already misled a human here at least once.

That fact decides whether a pipeline is worth building at all. A CI that runs ./gradlew test on every commit is actively harmful: it produces a green badge that means nothing, forever, and it does so with more authority than a developer's local run because it looks institutional. So the flags are the whole point, and the workflow says so where somebody editing it will read it. From .github/workflows/gate.yml, lines 50 to 60:

      # --rerun-tasks --no-build-cache is NOT optional. On a warm tree a plain
      # `./gradlew test` reports BUILD SUCCESSFUL in half a second having
      # executed zero tests (up-to-date checks skip the whole task, and even
      # `cleanTest test` comes back FROM-CACHE). These flags force every test
      # to execute on every run. The javadoc legs are here because javadoc
      # otherwise only runs at release, which is the one place a broken doc
      # comment is expensive to discover.
      - name: Java gate
        run: >
          ./gradlew test --rerun-tasks --no-build-cache
          :spectro-core:javadoc :spectro-orchestrator:javadoc

The javadoc legs ride along for the reason the comment gives. Card 87 once orphaned three doc comments from their stream() methods, and nothing noticed until the release gate ran javadoc, which is the most expensive possible moment to find out.

the tick is not the measurement

Comments do not enforce anything. Somebody will eventually simplify that command, so neither job trusts its own exit status alone. Both extract the real test count and fail on zero.

The Java side sums the attributes over every JUnit XML result file and writes the totals into the job summary, so a reviewer sees JUnit: 1162 tests, 0 failures, 0 errors, ... rather than a tick that hides it. The web side reads vitest's summary line out of the captured gate log, and that is where the pipeline nearly acquired its own silent failure.

vitest prints a fixed-format line on pass, Tests 1841 passed (1841). A naive grep for that is right on a laptop and wrong on a runner. Here is the same line pulled out of a real CI log, rendered through cat -v so the escapes are visible:

^[[2m      Tests ^[[22m ^[[1m^[[32m1841 passed^[[39m^[[22m^[[90m (1841)^[[39m

The gate captures that through a pipe, and it still arrives coloured, with the escapes sitting between the word and the digits. An anchored grep for Tests followed by a number never matches it, so the count would have come back empty and the guard would have failed every green run. It was caught before the first push by reproducing against a real CI=true vitest log, and the extraction strips the escapes first:

line="$(sed $'s/\x1b\\[[0-9;]*m//g' "$log" 2>/dev/null | grep -E '^[[:space:]]*Tests[[:space:]]+[0-9]+ passed' | tail -n 1 || true)"
count="$(printf '%s\n' "$line" | grep -oE '[0-9]+' | head -n 1 || true)"

The guard's job is narrow. The count is a reported measurement; the pass/fail authority stays with the gate's exit status. A count cannot tell you the tests are good. It tells you they ran, which happens to be the one lie this tree is capable of telling.

proving the gate by breaking it

An untested test harness is a contradiction, so card 125's acceptance criteria were written as things to break rather than things to observe. There are seven. Four could only be settled on GitHub itself, and those four have now run.

PR #4 carried two tests deliberately flipped, one Java (NotesServerRealProcessTest:74) and one web (format.test.ts). It went red on exactly those two and nothing else, with zero infrastructure errors in either log. main is green with the real counts in the job summary, JUnit 1162 and vitest 1841, and both count guards fail on zero. A deliberately broken doc reference planted at Spectro.java:31 killed the javadoc leg with error: reference not found, which is precisely how card 87's break would have been caught a release earlier.

what a clean machine knew that this one did not

The pipeline's first execution was red in 43 seconds, and it kept finding things into the next morning. Seven defects, not one of which this machine could see.

Two arrived together in the very first run (6842506). native/spectro-pty.c did not compile on Linux: glibc hides kill, sigaction and usleep behind feature-test macros under strict -std=c11, and Darwin exposes them regardless, so every macOS build had been silent about it. The fix is two lines that have to sit above every include:

#define _XOPEN_SOURCE 700
#define _DEFAULT_SOURCE

The second is the better story. spectro-web's drift tests import node:fs, but @types/node was nobody's declared dependency. tsc had been resolving the types sideways out of spectro-desktop/node_modules, a directory that exists on a developer machine and not in a fresh checkout. As the commit message puts it:

Locally even `tsc -b --force` was green off the borrowed types.

That is the shape of the whole category. The local build was not passing by luck, it was passing by borrowing, and no amount of forcing a rebuild could reveal it because the borrowed thing was still there.

Two more were the PTY helper meeting Linux naming. The build script's verify pattern only knew Darwin's /dev/ttysNNN, so a helper whose child correctly answered /dev/pts/0 failed its own build (1afdb68), and SpectroPtyHelperTest awaited the same Darwin-only substring and burned a ten second timeout on a child that had a real terminal (0a6614c). Neither weakened a check: none of the probe's reachable failure strings contain either substring, which was measured rather than assumed.

The fifth was a genuine race in a test, not the server. aBlankModelSwitchTakesTheTargetsDefaultNotTheCarriedModel made two back-to-back WebSocket sendText calls and discarded both futures. The JDK contract completes the second exceptionally while the first is in flight, so on a starved CPU one frame was silently never transmitted. Reproduced 3 times in 300 on a 0.5-CPU container with the exact CI signature, and a dead-port run passed, which ruled out the obvious reachability theory.

The last two came off the same runner over two days and wanted opposite responses. Three ProcessBusWireTest failures were choreography inside the tests. A fourth failure, in a test that had been in the suite since the bus was written, was a real and silent data-loss bug in the message bus. Both surfaced only because a shared two-vCPU runner kept losing the schedule lottery, which is what makes the pair worth its own post.

One red is still unexplained. SpeechRendererTest.abortClearsTheQueueAndStopsThePlayer failed once, on 31 July, and has been green on every run since. Nobody has touched that file. It sits on the list, unfixed and uncounted.

the lesson

All seven hid here for the same kind of reason. This machine has a warm Gradle cache, a sibling node_modules, a libc that hands out the POSIX surface without being asked, and fourteen cores against the runner's two. A fresh checkout on a shared runner has none of that, and every one of them was quietly holding something up.

The pipeline's value was never the badge. It was that a clean environment disagrees with a working one, and the disagreement is where the bugs live. The part that made the disagreement legible was the smallest piece of the whole thing: printing the number next to the tick.

It stops being right the moment the receiver is allowed to decline a frame. If a receiver can legitimately drop one without consuming it, the high-water mark lets the next frame close over the hole, and the later replay of that hole is indistinguishable from a duplicate.

There is a second story tangled up in the first, about triage. Four things failed on a shared runner inside thirteen hours. Three were test choreography and one was the product, and the fastest way to make the suite green would have buried the one that mattered.

the failure

spectroscope's orchestrator has a process bus: agents publish typed events, a hub fans them out, clients subscribe per topic. ClientCore is the client-side bookkeeping, and its contract is exactly-once delivery to the consumer with announced gaps. Loss is the one direction it is not allowed to fail in. If the hub's replay ring evicts something, the client must raise a BusGap, loudly.

A client can legitimately end up with a frame it does not consume: the hub fans out on a topic while every local handle on that topic has been closed. Dropping that frame is correct, and so is not advancing the cursor past it, since nobody saw it. The bug was what happened next.

It arrived as one red line in a CI log: framesForAFullyUnsubscribedTopicDoNotEatTheReplayOfALaterConsumer, at ProcessBusWireTest:351. That test has been in the suite since the bus was written, and no gate run before that one had ever tripped it. It took a machine with two shared vCPUs to lose the right race. From the commit message on e52b8ac:

The bug (found because a shared 2-vCPU runner kept losing the schedule lottery): a frame fanned out while every local handle was closed is dropped without advancing the cursor - correct - but ClientCore.accept's plain high-water then let the NEXT live frame leapfrog the hole, and the hub's replay of the hole was rejected as redelivery. Permanent, silent frame loss on that client, no BusGap, a wrong resume cursor forever: exactly the loss direction the class's exactly-once contract forbids.

Read that sequence slowly, because every individual step is defensible. Frame 7 arrives with no consumer, so it is dropped and the cursor stays at 6. Frame 8 arrives live and is consumed, so the high-water goes to 8. The hub, doing its job, later replays 7. The client compares 7 against a high-water of 8, concludes redelivery, and discards it. Nothing anywhere is wrong on its own. The client has now lost a frame permanently, will announce no gap, and will send a resume cursor that lies about what it has seen, forever.

the fix is bookkeeping, not policy

The correction does not touch the delivery path's decision rules. It gives the client a second thing to remember: the holes. From spectro-orchestrator/src/main/java/dev/spectroscope/orchestrator/ClientCore.java:

    /** topic → sender → epoch → highest sequence handed to the consumer. */
    private final Map<String, Map<String, Map<Long, Long>>> consumed = new LinkedHashMap<>();
    /** topic → sender → epoch → sequences the wire delivered while NOBODY
     *  here consumed the topic (all handles closed): holes above the cursor.
     *  {@link #accept} must never advance PAST one — the replay that fills
     *  it would be misread as redelivery and the frame lost silently, with
     *  no gap announced. Only that replay ({@link #accept}) or an announced
     *  eviction ({@link #noteGap}) removes a hole. Growth is bounded by the
     *  frames the hub fans out during one consumer-less window: a reconnect
     *  only re-subscribes topics somebody still consumes. */
    private final Map<String, Map<String, Map<Long, NavigableSet<Long>>>> undelivered =
            new LinkedHashMap<>();

The unbounded-growth question is answered in the same comment rather than left for a reviewer to ask, which is the house rule: the set only grows for as long as a consumer-less window lasts, because a reconnect re-subscribes only topics somebody still consumes.

Recording a hole has one subtlety worth showing, which is that not every undelivered frame is a hole:

    void noteUndelivered(BusEnvelope env) {
        Long known = consumed
                .getOrDefault(env.topic(), Map.of())
                .getOrDefault(env.sender(), Map.of())
                .get(env.epoch());
        if (known != null && env.sequence() <= known) {
            return; // redelivered consumed history — not a hole
        }

A frame below the cursor that arrives with no consumer is history being redelivered, and treating it as a hole would wedge the cursor behind something that was consumed long ago. Above the cursor, it is a hole. accept then refuses to advance past an outstanding one, and a hole is cleared by exactly two events: its own replay, or an announced eviction. accept's own javadoc states the new refusal case:

     * @return true when the consumer must see it; false on redelivery, or
     *         when the frame rides above a hole and its own redelivery is
     *         already in flight behind that hole's replay

pinning a race without a sleep

A bug that appears because a two-vCPU runner lost a scheduling lottery is not proved fixed by a suite that passes once. The pin is theCursorNeverLeapfrogsAFrameNobodyConsumed, and it was red 2 of 2 before the fix with the exact CI signature. It is a pin and not a coin flip because a witness forces the ordering: the test proves the state it depends on before it proceeds, instead of waiting a plausible number of milliseconds and hoping. Three ClientCore unit tests cover the bookkeeping directly.

The full gate after the change was JUnit 1162, which is the previous 1158 plus exactly those four new pins, 0 failures, with the orchestrator suite green three consecutive runs. The red and the green were both re-measured independently by an adversarial review pass, because a test that only its author has seen fail is not yet a pin.

the three that were not bugs

The same runner also struck ProcessBusWireTest three more times, at lines 179, 219 and 298: once on 30 July, then twice inside a single run the next morning. Those were choreography. The tests subscribed on one connection and published on another with nothing ordering the two at the hub, so on a starved runner the subscription lost the race to the burst, and a ring with capacity 2 made the shortfall permanent instead of recoverable.

The fix is five lines and one assertion, and it is the same idea as the pin above: prove the state, do not wait for it.

    private static CountDownLatch probeSubscription(ProcessBus witnessSide) {
        CountDownLatch heard = new CountDownLatch(1);
        witnessSide.subscribe(TOPIC, env -> heard.countDown());
        return heard;
    }

Callers send one probe frame and block on it before the real burst:

                assertTrue(subLive.await(10, TimeUnit.SECONDS),
                        "the probe frame proves the hub registered the witness's sub");

Every original assertion in those three tests is byte-identical afterwards. That constraint matters: a flake fix that also edits what the test asserts has quietly become a different test, and you no longer know whether the original claim still holds. All three strikes were predicted by analysis before two of them had ever fired. That prediction is the only reason anyone could be confident they were choreography and not symptoms.

the lesson

Telling the two apart is the whole story. Four failures, one shared cause on the surface (a slow, contended runner), and two completely different diagnoses underneath. The reflex response to a flaky suite is a retry, a longer timeout, or a sleep, and any of the three would have turned all four green. Three of them would have deserved it. The fourth would have gone on losing frames in production with a green suite standing over it.

The property worth carrying elsewhere is simple enough. A high-water mark is a compression of history, and it is only lossless if every frame below it was actually processed. The moment a receiver is allowed to say no, the mark stops being a summary of what happened and becomes a summary of what got far enough, and those two things drift apart silently.

The habit worth stealing is small. When an alert lands, do not patch the line it points at. Identify the taint source class it has found, in this case "an id that becomes a path", then go find every caller that produces one, because the scanner only saw the callers its queries model.

five alerts

The code-scanning page showed five open alerts, all high, all java/path-injection or java/xss. Alerts 1, 2 and 4 were false positives: the containment checks in that code are real, but the query does not model them as barriers. One was a direct-child parent equality check; another was a Files.walk without FOLLOW_LINKS combined with deleteIfExists, which removes a symlink rather than the thing it points at. Each dismissal carries a source-anchored argument instead of a shrug, and each of those three then had the scanner-legible guard shape added on top anyway, on the theory that an alert you have to re-argue every quarter is a cost you keep paying.

Alert 3 was real. SessionStore.readSessionEvents(id) resolved its id straight into the sessions directory with no normalize and no containment check, unlike its guarded siblings deleteSession and the controller's exportSession.

the flow it saw, and the one it did not

CodeQL reached that method through GET /api/sessions/{id}/events. That is a path variable, and the servlet container rejects %2F inside a path segment, so exploitation there is awkward. Judged on that flow alone, this is a tidy-up.

The flow it missed is worse, and it is in the same method's caller set. The websocket reads ?resume=<id>, a query parameter, which nothing pre-filters, and feeds it to loadSession, eventCount and the resume constructor. From the commit message on 3b97b2e:

the websocket's ?resume= feeds that seam with a query parameter no container pre-filters, through loadSession, eventCount AND the resume constructor, whose file every later append goes to. A ?resume=../../x could read and then append outside the store

The escalation is in the last clause. The read is bad; the resume constructor stores the resolved file as the store's append target, so every later event written in that session goes to whatever the traversal chose. A read primitive became a write primitive by being handed to a constructor.

The audience for this is bounded: /ws carries a local-origin fence from card 92, so the caller has to be on the machine. A browser page on a foreign origin is refused; a local page, or a local tool that sends no Origin at all, is not. That is not much comfort, because something running locally is precisely the attacker class the 0.3.0 hardening pass and card 74 were built around. Bounding an audience is a mitigation, not a fix.

one seam, so every caller inherits it

The correction is a single resolution point that every id-to-path conversion goes through. From spectro-core/src/main/java/dev/spectroscope/core/session/SessionStore.java:

    /**
     * Resolves a session id to its JSONL file — the ONE place an id becomes a
     * path. The id is caller input on three surfaces (REST path variable, the
     * websocket's {@code ?resume=}, the CLI's {@code --resume}), so the resolved
     * path must stay inside the store: normalized, under the sessions directory,
     * and a DIRECT child of it. Containment rather than an id-shape regex on
     * purpose — any file that really lives in the store stays addressable; the
     * strict shape check remains the HTTP edge's job.
     *
     * @param id the session id as the caller gave it — never trusted as a path
     * @return the contained session file path (it may not exist)
     * @throws IOException when the id resolves outside the sessions directory
     */
    public static Path sessionFile(String id) throws IOException {
        Path file = SESSIONS_DIR.resolve(id + ".jsonl").normalize();
        if (!file.startsWith(SESSIONS_DIR) || !SESSIONS_DIR.equals(file.getParent())) {
            throw new IOException("Not a session id: " + id);
        }
        return file;
    }

Two checks, on purpose, and the comment says why: startsWith is the guard shape scanners recognise, and parent equality is the stronger rule. Only one of them is load-bearing for correctness. The other is there so the next scanner run does not re-raise this, which is a legitimate reason to write a line of code and worth admitting rather than dressing up.

Choosing containment over a shape regex is deliberate too. A regex on the id would have been shorter, but it would make any session file that really lives in the store, yet predates the current id format, unaddressable. The strict shape check still exists. It just belongs at the HTTP edge, where rejecting odd input early is free.

The constructor now refuses before it touches anything:

        try {
            this.file = sessionFile(this.id);
        } catch (IOException outsideStore) {
            // A resume id is caller input (CLI --resume, the socket's ?resume=):
            // one that resolves outside the store is an argument error, not an
            // I/O failure — refused before anything is created or appended to.
            throw new IllegalArgumentException("Not a session id: " + id, outsideStore);
        }

Two tests went red first, readSessionEventsRefusesATraversalId and theResumeConstructorRefusesATraversalId. At the HTTP edge, /api/sessions/{id}/events picked up the same full-match SESSION_ID shape check that export and delete already had, pinned by SessionEventsShapeTest, and the export now serves through the shared seam with a modeled content type plus nosniff.

the alert that survived being fixed

There is a good coda. Alert 5, the fourth and last of the dismissals, was java/xss on the export response body. Session content is caller-shaped text, so the response was changed to stop being anything a browser would parse: application/x-ndjson, nosniff, and Content-Disposition: attachment.

CodeQL re-flagged it as alert 6, at the new line. Its java/xss query models neither a non-HTML content type nor an attachment disposition as a barrier at all, so the rewrite moved the code without moving the alert. That one was dismissed by hand, with the argument verified against the pushed code rather than the intent: the response never reaches an HTML parsing context, the id in the filename passed the full-match shape check so it carries no quote, CR or LF and cannot split a header, and the endpoint is local-origin fenced. The dismissal comment names the condition that reopens it, which is the response ever becoming renderable.

A dismissal is only honest if it says what would make it wrong. Otherwise it is just a way of turning the light off.

what the gate did not pick up

Unrelated to any alert, a javadoc break turned up on this branch. An ImageCopyController @param had drifted from body to req, which had been keeping :spectro-server:javadoc red on main.

The gate did not find it, and could not have. Its javadoc legs are :spectro-core:javadoc and :spectro-orchestrator:javadoc, two of the five Gradle modules, so spectro-server gets no javadoc coverage on any push at all. This one came out of a local ./gradlew javadoc over everything, which is the less flattering version of the story. The pipeline post argues that javadoc belongs in the gate instead of at the release. On this repo that argument currently covers two modules out of five.

The outcome is a number rather than an adjective. Zero open alerts on the code-scanning page, read back through the API instead of eyeballed on the page. The gate on the fix commit reported JUnit 1166, 0 failures and 1 skip across 173 result files, and ./gradlew javadoc ran clean over all five modules locally. The security tab picked up a SECURITY.md policy the same day. No Dependabot alerts are open either, though that boast is smaller than it sounds: twenty-four have been raised over the life of the repo, twenty-one closed by an actual dependency bump and three auto-dismissed.

the lesson

The scanner was wrong in both directions and useful anyway. It raised four flows that were already contained, which cost an afternoon of argument, and it missed the flow that could write, which no amount of staring at its output would have revealed. What it did do was name a taint source class precisely enough to be followed: an id that becomes a path.

Following that class through every caller is the part that found the real problem, and it is the part the tool cannot do for you, because the tool only knows the callers its queries model. The alert is where the work starts, not where it happens.