XLIO Ultra Command Latency

Summary

This benchmark compares seven small Goblin Core operations over four transport paths: a local shared-memory RESP ring, ordinary kernel TCP across a direct 100 Gb/s Ethernet link, a native XLIO Ultra server with an ordinary kernel TCP client, and native XLIO Ultra on both ends. Every path uses the same C++ RESP2 probe, pipeline depth one, fixtures, operation order, warmup, and sample count.

With native XLIO on both endpoints, median cross-host round trips were 4.98-8.37 microseconds. That is a conservative 5.3-9.1x improvement over ordinary kernel TCP on this hardware, while the asymmetric run demonstrates that an ordinary TCP client can still talk to the accelerated server. The local shared-memory RESP ring remains the fastest path at 0.87-2.77 microseconds.

Test paths

The benchmark seeds one string, a ten-field hash, and a ten-member sorted set. SET, HSET, and ZADD then overwrite an existing value with the same bytes; GET, HGET, and ZSCORE read those entries. PING measures the smallest complete command path.

Each operation has 100,000 warmup round trips followed by 1,000,000 measured round trips. There is exactly one request in flight, so the numbers are synchronous command latency rather than pipelined throughput.

NUMA and hardware control

Both systems are Dell PowerEdge R820 servers with four Intel Xeon E5-4657L v2 sockets and four NUMA nodes. Each uses an NVIDIA/Mellanox ConnectX-5 MCX515A-CCAT at 100 Gb/s full duplex over a direct DAC:

The harness reads /sys/class/net/enp68s0np0/device/numa_node, verifies that each selected CPU belongs to that node, and aborts on any mismatch. Server and client commands are wrapped with both numactl --cpunodebind=1 --membind=1 and an exact taskset. Goblin Core also receives --cpu 5 --numa 1. The benchmark therefore cannot silently migrate a process or its allocations to a remote socket.

Both adapters used firmware 16.35.8008 and the inbox mlx5_core driver on Linux 5.15. Adaptive RX/TX coalescing remained enabled at the host defaults; the CPU governor was schedutil.

Median latency

All values are microseconds. Lower is better.

Full-suite SET note. The full-suite kernel SET interval was anomalous: its median was 91.860 microseconds while the neighboring kernel operations were near 40-45 microseconds. A separate one-million-sample SET repeat, under the same hard NUMA binding, measured 41.692 microseconds at p50, 60.655 at p75, and 69.362 at p99. The full-suite row remains in the raw results rather than being silently replaced.

SET gain note. The SET gain uses the 41.692-microsecond repeat as the conservative kernel baseline. Dividing by the anomalous full-suite interval would report 13.17x.

Native XLIO tail

All values are microseconds from the xlio-xlio run.

The common distribution is tight through p99. The roughly 6 ms maxima seen in six rows are isolated host scheduling events on these non-isolated, schedutil systems, not representative wire latency. The p99.99 values show the high tail without allowing a single event to define the transport.

Buffers and ownership

Each --xlio listener owns a separate Ultra polling group. Mixed ring, XLIO, RDMA, and ExaSock targets retain literal command-line order as strict priority; a busy earlier target can starve a later one. Kernel sockets receive a sparse pass only when the polled targets are idle.

Reproduction and raw data

The orchestration script builds the native transport, verifies NIC and CPU NUMA membership on both hosts, runs the four matched cases, and records its complete environment:

bash benchmarks/xlio_latency.sh

Published artifacts:

• All latency samples summarized as percentiles
• Transport and buffer configuration
• Hosts, firmware, NUMA binding, link, and coalescing metadata
• Isolated kernel TCP SET repeat

The measured tree was commit 810df152395bbe2d7d296a25f4b4e25534675cca.