UltiCode Outbox and Redis Streams: Making Judge Delivery Recoverable

Evidence status: This article is based on the current fixed-commit outbox, Redis Streams, and reaper source in UltiCode. “Recoverable” here means that durable intent and retry paths are implemented; it does not mean exactly-once processing, unlimited throughput, or production HA has been verified.

The most dangerous queue failure is not “sending returned an error.” It is that the business transaction succeeded, but no reliable record remained of what needed to be sent. In an online judge, that window leaves the user seeing a successful submission while the Judge never receives a job.

When the judge outbox path is enabled, UltiCode puts the submission and judging intent in the same database transaction:

submit @Transactional
    ├── submissions: Pending
    └── judge_outbox: PENDING
             │
             ├── claim: FOR UPDATE SKIP LOCKED
             ├── atomic SET + XADD
             └── judge-workers

An outbox record is a state machine

JudgeOutboxMapper claims records by status and next retry time, using FOR UPDATE SKIP LOCKED so dispatchers do not claim the same row repeatedly. The current dispatcher defaults to batches of 50 and a two-second schedule; these are source defaults, not throughput metrics.

JudgeOutboxDispatcher maps different failures to different state transitions:

Real dispatch is also controlled by shadow flags and a cutover watermark, which matters when distinguishing shadow tasks from real tasks during migration.

Why Redis Streams instead of one XADD

RedissonStreamsJudgeQueueAdapter uses Lua during enqueue to put the deduplication key and XADD in one atomic operation. The deduplication key contains submissionId:generation.

This prevents a specific failure: if the process performs SETNX and then crashes before XADD, the system believes the message was already seen even though the Stream has no actual job. The atomic script cannot eliminate every external-system failure, but it removes this reproducible half-step state.

The consumer group is created from 0-0, so entries that existed before group creation still have a chance to be consumed. When a worker NACKs, the message remains in the Pending Entries List (PEL) instead of being silently deleted.

PEL recovery and observable state

UnackedStreamEntriesReaper scans idle entries every 10 seconds by default, advances at most one entry per sweep, and exposes PEL size, queue delay, oldest-entry age, and DLQ metrics. The current visibility-timeout constant is 1_800_000L, or 30 minutes; it is an implementation parameter, not an SLO.

Reclaimed jobs still pass through the generation/attempt fence in the judge executor. Recoverable delivery does not give an old message permanent permission to write.

This is not exactly-once

The boundary between the database transaction and external Redis operations remains. A more accurate description is:

persistent business intent
    + retryable delivery
    + message-level deduplication
    + fenced result writes
    = explainable at-least-once processing path

This is more reliable than saying “messages never repeat,” because duplicate delivery and duplicate execution can still happen. What the system actually protects is the final state from arbitrary duplicate or stale results.

Applying the model to LLM/Agent systems

Long-running Agent tasks should not rely only on an in-memory queue. User requests, tool-call intent, execution state, and retry reasons need recoverable records; idempotency keys identify duplicate triggers; final writes still need version fences. The window where “the model request succeeded but the tool job was not created” is the same class of problem as the submission/outbox window in an online judge.

Minimal verification path

Read JudgeOutboxDispatcher.java and RedissonStreamsJudgeQueueAdapter.java, then inspect SubmissionOutboxDispatcherIT.

Sources and related documents

• ulticode-engineering-highlights-f801a1076
• UltiCode: Verifiable Boundaries for Online Judging and Agent Engineering