tidemarkIn progress

Stream processors fail quietly. A watermark computed as the max instead of the min fires windows on incomplete data and nothing crashes; the counts are just slightly wrong. tidemark is a distributed stream processing engine in Go built around that fact, on Go 1.25 and currently at Phase 6b. Correctness is the deliverable and throughput is secondary.

Event-time windowing with watermark propagation through the dataflow graph, where each input gate's watermark is the minimum across its channels.

Chandy-Lamport barrier checkpointing with barrier alignment over Pebble-backed keyed operator state, and exactly-once sinks that commit on checkpoint-complete notification rather than during the snapshot.

A chaos suite that injects seeded fault schedules across five Nexmark queries, with every windowed result checked against an independent batch oracle.

A job is a dataflow graph of vertices running as parallel subtasks, and the subtask is the unit of scheduling, state, and failure. Watermarks and barriers travel in-band as stream elements alongside records, which keeps them ordered relative to the data they describe. Sources inject barriers at a fixed element interval rather than on a timer, and every source is a pure function of seed and offset, so a recovering subtask seeks to its checkpointed offset and replays exactly the records it would have read.

Faults are keyed to logical position — elements processed, barriers seen — never to wall-clock time. Go's scheduler is not deterministic, so the fault schedule has to be, and that is what makes a crash and its recovery reproducible from a single seed.

Identical output means the final sorted contents of the sink, not emission order. Ordering after recovery will differ from a clean run, so a test comparing emission order is testing the wrong property.

Published numbers are gated on a machine fingerprint. The benchmark check refuses to compare runs across different machines, which is why the throughput and recovery tables are still empty rather than filled in from a development container.

State size is the part that is measured, because peak entry counts and checkpoint bytes are functions of the workload and the encoding rather than of the hardware. On Nexmark q7 over two million events, checkpoint state is 41.0 bytes per entry at every cardinality and every parallelism, and 33.0 bytes per entry on q5.

Measuring it surfaced a real engine property: source subtasks split the offset space into contiguous ranges and the gate watermark is a minimum, so the same q7 workload at 100,000 auctions holds 26,621 peak entries at parallelism 1 and 2,723,906 at parallelism 4. Any state figure from this engine has to be quoted with its parallelism.

Throughput and recovery latency are both pending measurement on reference hardware, because the development machine could not produce either number honestly.