Indexed Compact Hash Threshold Sweep

Goblin Core's compact hash now stores one pooled blob with a byte fingerprint and a 16-bit entry offset per field. Lookups scan fingerprints with the build ISA's widest byte vector, then compare the complete field only for fingerprint matches. The full hash uses the Swiss field index and arena.

The blob begins with a byte-packed four-byte header (u16 entry bytes and u16 count), followed by the fingerprint and offset directories. Each entry stores encoded-field length and encoded-value length in one or two bytes: lengths through 127 use one byte; larger lengths use a high-bit byte plus the low byte, up to 32,767. A larger encoded field or value promotes to the full hash. Compact fields use the shared exact decimal/UUID encoder with LZ4 forbidden. --disable-encoding instead stores both fields and values verbatim.

The initial sweep supported increasing the old 32-field default through 512. A second AVX2 sweep through 2048 supports removing the default count ceiling: the 64 KiB compact-blob limit now decides promotion. Operators can still force an earlier count boundary with --hash-listpack-max-entries; zero forces every hash to the full form and blob restores the size-only default.

Method

• Host: a quiet dedicated 64-core Threadripper PRO 5995WX system.
• Every cardinality and engine starts a fresh server; servers run one at a time.
• Each row holds approximately 500,000 total fields. Keys and fields are 14 bytes each and values are 16 bytes.
• A Python RESP pipeline issues one multi-field HSET per hash. Point tests use one redis-benchmark client and pipeline depth 256.
• RSS is the patched, live /proc value exposed as INFO memory used_memory_rss, measured as a delta over the warmed empty server.
• The compact/full decision uses 1,000,000 point operations, median of five runs. The incumbent matrix uses 200,000 operations, median of three runs.
• The extended 512-2048 pass uses the compile-time AVX2 fingerprint scanner.
• Redis and Valkey use benchmarks/redis-parity.conf. Dragonfly uses one proactor for parity. Incumbents are black-box RESP servers; their source was not inspected.

The exact driver is benchmarks/hash_threshold_sweep.py.

Compact Versus Full

RSS is bytes per hash. HSET is a same-width update to the middle field.

Compact uses less RSS at all 13 points. The non-monotonic savings are allocator size classes: 64 fields is an unfavorable edge, while 80 fields consumes almost no more RSS. Same-width HSET is 1.2-14.9% faster in compact form.

Lookup Shape

Each cell is compact throughput and its difference from forced-full Goblin. load is end-to-end Python construction throughput, not a server-only rate.

Hits do not show a reason to promote before 512. Misses are the weaker shape: they trail full Goblin by as much as 9.3% because every fingerprint group must be checked. Construction also trails by 6.1% at 512. A miss-heavy workload or one dominated by different-width updates and deletes can choose an earlier threshold; those mutations rebuild the compact blob.

Extended AVX2 Sweep

The second sweep tests 512, 645, 812, 1024, 1290, 1625, 2048. With these 14-byte fields and 16-byte values, each field costs 37 compact bytes including its directory entry. Exactly 1771 fields produce a 65,535-byte blob; 1772 must promote. Thus 1625 remains compact while the threshold-selected 2048 row is intentionally full.

The 2048 differences are fresh-server noise between two runs of the same full representation. Before promotion, compact saves 16-20% RSS throughout. HSET and middle-position HGET remain close to or faster than full, but position and miss shape expose the cost of scanning a larger directory:

The default follows Goblin's memory-first policy and lets blob size decide. A miss-heavy workload, one dominated by late-position fields, or one that inserts mostly single-field commands can set a finite count ceiling. The count option is a workload control, not a structural requirement.

Incumbent Memory

RSS bytes per hash from the same six-engine pass:

Redis and Valkey are smaller below 128 fields. Their configured compact form ends at 128; Goblin becomes the smallest at 128 and widens the lead after they promote. At 512 fields Goblin compact uses 31% less RSS than Redis 8.8, 35% less than Valkey 9.1, and about half the RSS of Redis 7.2 and Dragonfly.

The extended matrix keeps the same direction. At 1625 fields the compact form uses 62,054 RSS bytes/hash versus 92,353 for Redis 8.8, 91,153 for Valkey, 124,094 for Redis 7.2, and 125,975 for Dragonfly.