2026-07-31 · correctness · orchestrator · ci

the cursor that leapfrogged a hole

A shared CI runner failed four times inside thirteen hours. Three were test choreography. The fourth was permanent, silent frame loss in the message bus, and the reflex that would have made the suite green would also have hidden it.

A high-water mark is the standard way to make an at-least-once stream behave like an exactly-once one. You remember the highest sequence number you handed to the consumer, and you discard anything at or below it on redelivery. It is four lines and it is usually right.

It stops being right the moment the receiver is allowed to decline a frame. If a receiver can legitimately drop one without consuming it, the high-water mark lets the next frame close over the hole, and the later replay of that hole is indistinguishable from a duplicate.

There is a second story tangled up in the first, about triage. Four things failed on a shared runner inside thirteen hours. Three were test choreography and one was the product, and the fastest way to make the suite green would have buried the one that mattered.

the failure

spectroscope's orchestrator has a process bus: agents publish typed events, a hub fans them out, clients subscribe per topic. ClientCore is the client-side bookkeeping, and its contract is exactly-once delivery to the consumer with announced gaps. Loss is the one direction it is not allowed to fail in. If the hub's replay ring evicts something, the client must raise a BusGap, loudly.

A client can legitimately end up with a frame it does not consume: the hub fans out on a topic while every local handle on that topic has been closed. Dropping that frame is correct, and so is not advancing the cursor past it, since nobody saw it. The bug was what happened next.

It arrived as one red line in a CI log: framesForAFullyUnsubscribedTopicDoNotEatTheReplayOfALaterConsumer, at ProcessBusWireTest:351. That test has been in the suite since the bus was written, and no gate run before that one had ever tripped it. It took a machine with two shared vCPUs to lose the right race. From the commit message on e52b8ac:

The bug (found because a shared 2-vCPU runner kept losing the schedule lottery): a frame fanned out while every local handle was closed is dropped without advancing the cursor - correct - but ClientCore.accept's plain high-water then let the NEXT live frame leapfrog the hole, and the hub's replay of the hole was rejected as redelivery. Permanent, silent frame loss on that client, no BusGap, a wrong resume cursor forever: exactly the loss direction the class's exactly-once contract forbids.

Read that sequence slowly, because every individual step is defensible. Frame 7 arrives with no consumer, so it is dropped and the cursor stays at 6. Frame 8 arrives live and is consumed, so the high-water goes to 8. The hub, doing its job, later replays 7. The client compares 7 against a high-water of 8, concludes redelivery, and discards it. Nothing anywhere is wrong on its own. The client has now lost a frame permanently, will announce no gap, and will send a resume cursor that lies about what it has seen, forever.

the fix is bookkeeping, not policy

The correction does not touch the delivery path's decision rules. It gives the client a second thing to remember: the holes. From spectro-orchestrator/src/main/java/dev/spectroscope/orchestrator/ClientCore.java:

    /** topic → sender → epoch → highest sequence handed to the consumer. */
    private final Map<String, Map<String, Map<Long, Long>>> consumed = new LinkedHashMap<>();
    /** topic → sender → epoch → sequences the wire delivered while NOBODY
     *  here consumed the topic (all handles closed): holes above the cursor.
     *  {@link #accept} must never advance PAST one — the replay that fills
     *  it would be misread as redelivery and the frame lost silently, with
     *  no gap announced. Only that replay ({@link #accept}) or an announced
     *  eviction ({@link #noteGap}) removes a hole. Growth is bounded by the
     *  frames the hub fans out during one consumer-less window: a reconnect
     *  only re-subscribes topics somebody still consumes. */
    private final Map<String, Map<String, Map<Long, NavigableSet<Long>>>> undelivered =
            new LinkedHashMap<>();

The unbounded-growth question is answered in the same comment rather than left for a reviewer to ask, which is the house rule: the set only grows for as long as a consumer-less window lasts, because a reconnect re-subscribes only topics somebody still consumes.

Recording a hole has one subtlety worth showing, which is that not every undelivered frame is a hole:

    void noteUndelivered(BusEnvelope env) {
        Long known = consumed
                .getOrDefault(env.topic(), Map.of())
                .getOrDefault(env.sender(), Map.of())
                .get(env.epoch());
        if (known != null && env.sequence() <= known) {
            return; // redelivered consumed history — not a hole
        }

A frame below the cursor that arrives with no consumer is history being redelivered, and treating it as a hole would wedge the cursor behind something that was consumed long ago. Above the cursor, it is a hole. accept then refuses to advance past an outstanding one, and a hole is cleared by exactly two events: its own replay, or an announced eviction. accept's own javadoc states the new refusal case:

     * @return true when the consumer must see it; false on redelivery, or
     *         when the frame rides above a hole and its own redelivery is
     *         already in flight behind that hole's replay

pinning a race without a sleep

A bug that appears because a two-vCPU runner lost a scheduling lottery is not proved fixed by a suite that passes once. The pin is theCursorNeverLeapfrogsAFrameNobodyConsumed, and it was red 2 of 2 before the fix with the exact CI signature. It is a pin and not a coin flip because a witness forces the ordering: the test proves the state it depends on before it proceeds, instead of waiting a plausible number of milliseconds and hoping. Three ClientCore unit tests cover the bookkeeping directly.

The full gate after the change was JUnit 1162, which is the previous 1158 plus exactly those four new pins, 0 failures, with the orchestrator suite green three consecutive runs. The red and the green were both re-measured independently by an adversarial review pass, because a test that only its author has seen fail is not yet a pin.

the three that were not bugs

The same runner also struck ProcessBusWireTest three more times, at lines 179, 219 and 298: once on 30 July, then twice inside a single run the next morning. Those were choreography. The tests subscribed on one connection and published on another with nothing ordering the two at the hub, so on a starved runner the subscription lost the race to the burst, and a ring with capacity 2 made the shortfall permanent instead of recoverable.

The fix is five lines and one assertion, and it is the same idea as the pin above: prove the state, do not wait for it.

    private static CountDownLatch probeSubscription(ProcessBus witnessSide) {
        CountDownLatch heard = new CountDownLatch(1);
        witnessSide.subscribe(TOPIC, env -> heard.countDown());
        return heard;
    }

Callers send one probe frame and block on it before the real burst:

                assertTrue(subLive.await(10, TimeUnit.SECONDS),
                        "the probe frame proves the hub registered the witness's sub");

Every original assertion in those three tests is byte-identical afterwards. That constraint matters: a flake fix that also edits what the test asserts has quietly become a different test, and you no longer know whether the original claim still holds. All three strikes were predicted by analysis before two of them had ever fired. That prediction is the only reason anyone could be confident they were choreography and not symptoms.

the lesson

Telling the two apart is the whole story. Four failures, one shared cause on the surface (a slow, contended runner), and two completely different diagnoses underneath. The reflex response to a flaky suite is a retry, a longer timeout, or a sleep, and any of the three would have turned all four green. Three of them would have deserved it. The fourth would have gone on losing frames in production with a green suite standing over it.

The property worth carrying elsewhere is simple enough. A high-water mark is a compression of history, and it is only lossless if every frame below it was actually processed. The moment a receiver is allowed to say no, the mark stops being a summary of what happened and becomes a summary of what got far enough, and those two things drift apart silently.