Libfabric provider matrix over 100 Gb/s

Goblin Core's fastest locally available network path completed Redis-shaped operations in 6.39 microseconds at the arithmetic-average median. That was SBE over libfabric FI_EP_RDM using verbs;ofi_rxm and the automatic inject path. The same eight operations averaged 90.33 microseconds over ordinary kernel TCP with RESP2: 14.1x higher latency and 14.0x fewer sequential round trips per second.

Libfabric's software tcp provider also mattered. RESP2 over that provider averaged 39.59 microseconds at p50, 2.28x lower than RESP2 over the kernel socket path. This is a provider and software-path result, not a bandwidth result; every workload deliberately used pipeline depth one.

Scope: this qualifies the Goblin code path intended for AWS EFA, but it is not an AWS EFA hardware result. Native provider efa was unavailable on the local machines. The measured providers were tcp and verbs;ofi_rxm on a pair of 14-year-old, four-socket Intel Xeon E5-4657L v2 systems over a direct 100 Gb/s Mellanox ConnectX-5 Ethernet link. The native AWS EFA fast follow is now published.

Complete ranking

Modes are ranked by the arithmetic average of the eight per-operation medians. The average p99.99 is likewise an average of reported per-operation percentiles, not a merged distribution. Sequential operations per second is the reciprocal of each operation's observed mean, averaged across operations.

SBE over verbs;ofi_rxm led the aggregate result with either send path. Automatic injection was 4.2-4.4% faster at the average median than forced fi_send on that provider, while their average p99.99 results differed by less than one percent. On the software tcp provider, the two send policies were effectively tied.

The forced-send result is important beyond its ranking: every frame used fi_send, transmit buffers were reclaimed through the TX completion queue, receives used fi_recv, and receive completions came through fi_cq_readfrom. The faster automatic mode uses fi_inject for an eligible small frame and falls back to that same fi_send path for a larger frame.

Operation latency

Each cell below is p50 / p99.99 in microseconds. Pub/Sub measures the full path from issuing PUBLISH, through its acknowledgement, to validated receipt by one literal-channel subscriber.

Automatic inject path

Kernel TCP is included as the socket baseline; "automatic" applies only to the libfabric columns.

The tcp/SBE automatic Pub/Sub cell was slower than its RESP2 neighbor and than the forced-SBE result. It is retained as measured rather than replaced by an inferred value.

Forced fi_send path

The raw matrix on GitHub contains min, p50, p75, p90, p95, p99, p99.9, p99.99, maximum, mean, and sequential operations per second for all 80 distributions. Its neighboring README records the build identity and artifact layout.

Tail behavior

On verbs;ofi_rxm, operation-level p99.99 ranged from 24.92 to 54.27 microseconds across both wire formats and send paths. Isolated maxima between 0.74 and 1.00 milliseconds remained beyond p99.99. Kernel TCP produced the largest single observation, 2.36 milliseconds during GET.

The original RDM qualification had a much larger and repeatable tail step. It was not caused by out-of-order delivery. Instrumented depth-one runs observed 1,760,003 replies on the publisher connection and 220,001 replies on the subscriber connection without one sequestered frame, duplicate, reorder-window overflow, sequence gap, or pending reordered message. Server-side request counters were likewise clean.

The stall was local to Goblin's bootstrap listener. The polled server called nonblocking accept() on every spin-loop pass. On Linux, an unsuccessful accept() can allocate and then retire socket, file, and inode objects before returning EAGAIN. Function tracing showed repeated RCU softirq batches that reclaimed those objects, with individual batches lasting roughly 0.4-1.1 milliseconds.

Goblin now checks listener readiness first and only drains accept() when a bootstrap connection is waiting. Before the fix, one trace contained 4,119 RCU softirq runs over 100 microseconds. During a two-second active post-fix trace, all 59 softirq runs were below 100 microseconds and the maximum was 20.6 microseconds. The immediate control run cut average SBE/RDM p99.99 from 857.47 to 29.47 microseconds; this independent full matrix reproduced it at 30.83 microseconds.

The preserved pre-fix results and traces are in benchmarks/libfabric-2026-07-26.

Method

• Both hosts are four-socket Intel Xeon E5-4657L v2 systems running Linux 5.15.0-186.
• Both endpoints use a Mellanox ConnectX-5 (firmware 16.35.8008) negotiated at 100,000 Mb/s full duplex over a direct-attach copper link.
• The NIC is on NUMA node 1 on both hosts. Both Goblin and the benchmark client were pinned to CPU 5 and memory node 1.
• The Release build used AWS libfabric 2.4.0amzn5.0. RDM used FI_EP_RDM, FI_MSG | FI_SOURCE, fi_send/fi_recv, and completion queues through providers tcp and verbs;ofi_rxm.
• Each provider/send combination used a separate server run. This also kept Goblin's strict-priority polled libfabric target out of the kernel TCP reference.
• Every workload used pipeline depth 1, a 128 KiB client work buffer, 20,000 warmups, and 200,000 measured operations. Thus p99.99 represents 20 observations per distribution.
• Libfabric heartbeats were disabled during measurement, avoiding idle heartbeat traffic while both clients continuously generated work.
• The libfabric client used 16 transmit slots, 16 receive slots, a 128 KiB maximum record, and a bounded 64-message/1 MiB reorder window.
• SET alternated two same-length 16-byte values on an existing key. GET was a hit. HSET updated one existing field with the same alternating values and HGET was a hit. ZADD alternated scores 1 and 2 for an existing member and ZSCORE was a hit.
• Pub/Sub used separate publisher and subscriber connections and one literal channel. Timing began before PUBLISH and ended only after the subscriber received and validated the payload.
• RESP2 and SBE replies were validated on every warmup and measured operation.

Depth-one operations per second measure sequential round trips, not saturated throughput. The 100 Gb/s link is a latency path in this test, not a bandwidth target.

Reproduction

The complete cross-host matrix is automated:

SAMPLES=200000 WARMUP=20000 \
  bash benchmarks/libfabric_provider_matrix.sh

An automatic-inject server for the fastest local provider:

build/goblin-core \
  --libfabric 'verbs;ofi_rxm' 10.100.0.1 17401 \
  --enable-sbe \
  --no-auth-libfabric \
  --efa-heartbeat-timeout-ms 0 \
  --cpu 5 \
  --numa 1

Add --libfabric-force-send to route every frame through fi_send and its TX completion queue. The benchmark executable selects the same client-side mode with the final auto or send argument:

numactl --physcpubind=5 --membind=1 \
  build/goblin_core_libfabric_latency_benchmark \
  fabric-sbe 'verbs;ofi_rxm' 10.100.0.1 17401 10.100.0.2 \
  verbs-sbe-send 200000 20000 send