Kafka Durability Benchmark

Summary

Goblin Core's broker acknowledgement mode buys a precise success boundary: a write reply is not released until Kafka has accepted every mutation in that atomic operation under the broker's configured durability policy. On the single-node Redpanda installation described below, an individually confirmed SET took 966 us at p50, 1.355 ms at p99, and 1.532 ms at p99.9, for 1,079 writes/s. The rare tail was real: p99.99 was 16.7 ms across 100,000 measurements per round.

That cost amortizes well. RESP pipelines reached 51.0K broker-confirmed writes/s at depth 128 and 61.5K/s at depth 512. A 128-write MULTI/EXEC block reached 44.1K writes/s, with its one atomic result withheld until all 128 Kafka records were acknowledged. Every broker-mode workload finished with zero pending records and an acknowledged logical offset equal to the applied offset.

These are medians of three complete rounds, not the best round.

What was tested

One native C++ client talked RESP2 over a Unix-domain socket to one Goblin Core process. The server ran in three modes:

The Kafka producer used idempotence, acks=all, one in-flight request per broker connection, and an explicit linger.ms=0. Every Kafka run used a new topic with one partition and replication factor 1. The runner deleted only its own topics after preserving their final high watermarks.

SET used fixed keys and alternating 16-byte values, so each operation was a real overwrite without growing the Goblin keyspace. Each mode and round issued exactly 1,101,520 writes:

• 2,000 warm-up writes.
• 100,000 sequential SETs, timed one by one. This provides 100,000 observations per round for the p99.99 result.
• About 200,000 SETs at each pipeline depth: 8, 32, 128, and 512.
• About 50,000 SETs inside MULTI/EXEC blocks containing 1, 8, 32, or 128 writes. The server transaction buffer was explicitly set to 64 KiB.

Mode order rotated across rounds to reduce ordering bias. Tables report the median of each run-level result. Pipeline and transaction latency is whole-batch latency; the observation count therefore shrinks as batch size grows. Their tail percentiles should not be confused with the 100,000-observation sequential distribution.

Sequential latency

One SET was sent and its reply consumed before the next SET. All latency columns are microseconds.

Queued mode keeps the common path close to the UDS baseline, but asynchronous producer work still appears in its tail because delivery callbacks run on the single Goblin server thread. Broker mode makes the approximately one-millisecond Redpanda acknowledgement the common path. The source of the roughly 16-28 ms rare pauses was not profiled in this run, so this document does not assign them to Goblin, Redpanda, the NVMe device, or the scheduler.

RESP pipeline throughput

Each pipeline is one client write followed by consumption of every individual SET reply. Throughput is SET operations per second. settled adds the time needed after the last client reply for Kafka's acknowledged logical offset to catch the applied offset.

The queued producer did not hide a large unpaid backlog in this test. Across all queued workloads and rounds, at most 384 records remained after the final client reply and the longest final drain was 3.237 ms. Broker mode had zero pending records at every measured client boundary by construction.

Atomic transaction throughput

The client sent MULTI, N SET commands, and EXEC as one RESP batch, then read the OK, every QUEUED, and the final EXEC array. Throughput counts the SET mutations, not transaction envelopes.

The one-write transaction row includes MULTI/EXEC protocol and transaction bookkeeping, so it should not be compared to raw SET as though they were the same operation. The useful result is the scaling: the broker round trip is amortized over the mutation block without weakening atomicity or releasing EXEC early.

Test system

Goblin and the benchmark client were deliberately pinned. Redpanda retained its deployed affinity rather than being retuned for the benchmark. On this four- socket machine, node 0 to node 2 has the largest reported NUMA distance. The result is therefore representative of this installation, not a claim about the best latency a broker-local placement or current hardware can achieve.

Goblin connected to Redpanda at the host's LAN address, but both processes ran on the same physical server; Linux routed the TCP traffic locally. This benchmark does not include a switch or remote Kafka node.

Interpretation and limits

Broker mode does not invent a durability guarantee stronger than the broker's configuration. This Redpanda topic had one replica, so acks=all acknowledged one broker. It validates the Goblin-to-broker success boundary and local durable configuration, but not availability through a broker-machine failure or the latency of a multi-node quorum.

Goblin applies a write locally before the asynchronous delivery callback releases its reply. Other clients continue making progress while that acknowledgement is pending. The mode guarantees what a successful write reply and downstream firehose release mean; it is not an isolation barrier hiding the tentative local value from concurrent readers.

Crash recovery and snapshot replay are tested separately in External Logging: The First Recovery Test and External Logging: All Persistent Types.

Reproduce it

Build the server and native harness, then run against an existing Redpanda or Kafka broker:

cmake -S . -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DGOBLIN_CORE_ENABLE_KAFKA=ON \
  -DGOBLIN_CORE_BUILD_BENCHMARKS=ON
cmake --build build \
  --target goblin_core_server goblin_core_kafka_ack_benchmark -j

BROKERS=127.0.0.1:9092 \
GOBLIN_BINARY="$PWD/build/goblin-core" \
BENCHMARK_BINARY="$PWD/build/goblin_core_kafka_ack_benchmark" \
OUTPUT_DIR=/path/on/local/storage/kafka-ack-results \
benchmarks/kafka_ack_benchmark.sh

The runner records raw CSV, machine and broker metadata, per-process logs, and topic high watermarks. For this run the raw CSV SHA-256 was 32ebb9d6c31e041377eed1f40bccaa1ac5972f459b3097d3d256d2a3774ee2d8.