Memory Latency Bounds Your Service

ClickHouse has an open pull request, #128536, that packs two-level hash table buckets into a single memory area and backs them with transparent huge pages. The hashing logic does not change. What changes is how many cache lines and TLB entries a single lookup has to touch. That is a memory-latency optimisation wearing the costume of a data-structure change, and it is the most common shape of performance work on backend services right now.

The argument in pivotlake.io’s post “It’s the Memory Latency, Stupid” is that the gap between a core and the data it needs, not the width of the pipe feeding it, sets the ceiling on throughput. Most production dashboards do not measure that gap. They measure CPU busy, request rate and mean latency, and then argue about which of those to blame when the p99 doubles.

Key Takeaways:

Bandwidth benchmarks sit flat while tail latency climbs, because throughput under dependent access is bounded by latency, not pipe width.

The working-set boundary matters more than absolute size: a step change at the last cache level shows up as a latency step change in production.

Transparent huge pages, NUMA binding and arena allocators all reduce memory stalls and all introduce new failure modes you have to monitor.

Measure the tail (p99, p999) plus cache and page-fault counters. Mean latency will tell you the system is fine while users disagree.

Every memory-behaviour change needs a pre-committed rollback trigger, because these changes often look neutral on average and hostile at the tail.

The Latency Ladder and the Working-Set Cliff

TechCompare’s latency visualiser lays the ladder out from L1 cache down to spinning disk, and the shape of it is the useful part: each tier costs roughly an order of magnitude more than the one above it. The absolute numbers drift with each CPU generation. The ratios do not, and the ratios are what your code actually experiences.

Two consequences follow for anyone operating a service. First, a small miss rate can dominate total time. If 2 percent of accesses fall through to main memory and each of those costs a couple of orders of magnitude more than a hit, that 2 percent can own the majority of wall-clock time inside the hot loop. Second, the location of the cliff is set by working-set size, not by request rate.

Why Bandwidth Benchmarks Lie

A streaming read benchmark measures how fast you can pull contiguous memory when the CPU knows what is coming. Hardware prefetchers and out-of-order execution overlap those accesses, so the measured figure reflects the memory bus and little else. Real workloads that suffer from memory stalls do not look like that. They look like pointer chasing: hash probes that follow a bucket pointer, tree walks, deserialisation loops, and linked structures spread across the heap.

Those accesses are serialised by dependency. The address of the next load is not known until the previous one returns, so no amount of out-of-order width hides the cost. Throughput under that access pattern is roughly outstanding requests divided by latency, which means a latency regression cannot be bought back with more bandwidth. It has to be bought back with more concurrency, more work per access, or fewer accesses.

The ClickHouse change is a clean example of the third lever. A two-level hash table normally sends a probe to an indirection layer and then to a payload. Collapsing both into one memory area removes a dependent hop, and transparent huge pages reduce the number of distinct TLB entries needed for that area. The algorithm is unchanged and the lookup is measurably cheaper because it touches less of the hierarchy.

Where Memory Stalls Surface in Production

Memory stalls rarely announce themselves. They arrive disguised as one of the following, and each disguise points at a different diagnostic path.

Tail latency that moves without a throughput change. Request rate flat, CPU flat, p99 up 40 percent. This is the classic signature of a working set crossing a cache boundary or of growing allocation churn.
Periodic latency spikes at a fixed cadence. On Linux, transparent huge page compaction and reclaim scans are the usual suspects when the period is tens of seconds and the spike hits every process on the node, not just yours.
Node-level asymmetry inside one cluster. Two nodes, same build, same traffic, different p99. That usually means NUMA placement, a noisy neighbour holding memory bandwidth, or mismatched instance generations.
Latency that scales with tenant size. One customer’s account is slow and others are fine. Working set, not code path.
Reclaim and OOM kills on containers whose resident memory looks small. Page cache accounted to the cgroup, plus allocator caches that never return memory to the kernel, produce kills that the application’s own RSS metric never predicted.

The container case deserves emphasis because it is a frequent on-call surprise. Memory limits are enforced on the cgroup, and the cgroup sees file-backed pages the process thinks of as kernel-managed. A service holding a large memory-mapped index can be killed for exceeding a limit while its application-level heap metric sits comfortably below it. If your only memory alert is a heap gauge, you will learn about the limit from an exit code at 03:00.

Signals Worth Putting on a Dashboard

The point of the table below is attribution. Each signal narrows the cause; none of them proves it alone, and the failure mode in the last column is why engineers chase the wrong thing for a week.

Signal	What it tells you	Where it misleads
Mean response time	Whether the bulk of traffic is healthy	Averages blur multi-modal distributions, so a stalled minority disappears
p99 and p999 latency, per endpoint	Whether a minority of requests pay a stall	Conflates memory stalls with scheduling, GC and network retransmits
Cache and TLB miss counters from CPU performance counters	Whether the hierarchy is being traversed more than expected	Counts events, not time; a higher miss count can still be a cheaper workload
Page-fault rate, split into minor and major	Whether the process is growing its mapping or waiting on IO	Minor faults from normal allocation are noise until the rate changes
Memory pressure stall time	Whether the kernel is stalling tasks on reclaim	Node-level, so it cannot tell you which container caused it
CPU steal and run-queue delay	Whether the hypervisor or scheduler is the bottleneck	Invisible to in-process tracing, which reports fast internal work

Instrument the tail first. A service with a stable p50 and a p99 that moves in 5-minute steps is telling you something changed in memory behaviour, and the step shape usually dates the change to a deploy, a data migration, or a traffic shift onto larger accounts.

Mitigations and What They Cost

There is no free lever here. Each option below shifts work somewhere else, and the somewhere else is where your next incident lives.

Mitigation	Where it helps	What it costs
Transparent huge pages, restricted to madvise mode	Fewer TLB entries for large contiguous regions such as hash tables and arenas	Background compaction can stall a node; memory bloat raises the chance of reclaim
Collapsing indirection in hot structures	Removes a dependent load per lookup, as in the ClickHouse bucket change	Requires a data migration and a rewrite of the access path; resize behaviour changes
Arena or slab allocation per request class	Locality, fewer page faults, cheaper teardown	Memory held past its useful life, so cgroup limits get hit earlier
NUMA binding and CPU pinning	Keeps accesses local on multi-socket hosts	Breaks under rescheduling and failover; a pinned pod that moves runs degraded
Batching work per memory access	Amortises a fixed stall across more units of output	Adds queueing delay and coarser tail behaviour

Two orderings matter in practice. Fix the access pattern before you tune kernel knobs, because a layout change usually beats a page-size change and is easier to roll back. Then size the instance family against memory per vCPU rather than vCPU count, since the ratio differs across families and generations and is the variable that decides whether your working set fits.

The honest trade-off on huge pages is that they are a global-ish setting with per-process consequences. Enabling them system-wide to fix one service exposes every other process on the node to compaction stalls. Restricting them to opt-in regions limits the blast radius and keeps the benefit where you measured it.

Rolling Out a Memory-Behaviour Change

Ship memory changes behind a flag and judge them on the tail. A canary that watches mean latency will pass a change that doubles p999, because the affected requests are too few to move the average.

Freeze the guardrails before the rollout: p99 and p999 per endpoint, page-fault rate, pressure stall time, and container OOM count.
Run the canary long enough to cover a full traffic cycle. Working sets grow with daily batch jobs, so a 20-minute canary sees the smallest working set of the day.
Compare like with like: same instance family, same zone, same traffic slice. Cross-family comparison will show the instance difference, not your change.
Write the rollback trigger as a number before you deploy. “p999 regresses more than 15 percent for two consecutive windows” is a trigger; “if it looks worse” is an argument.
Keep the old allocation path compiled in for one release cycle. Reverting a flag takes seconds; reverting a data migration does not.

Troubleshooting the Failures You Will Actually Hit

Containers killed with a low heap gauge. The cgroup counts page cache and unreleased allocator arenas. Reduce mapping size, drop file-backed caches before the peak, or raise the limit. Adding an exit-code metric that distinguishes OOM kills from application exits is the cheapest permanent fix, because it turns a mystery into an alert.

Tails got worse after enabling huge pages. Compaction is the likely cause, and the tell is a spike that hits unrelated processes on the same node at the same moment. Switch to opt-in mode, restrict the regions, and re-measure on a node that runs only the workload in question.

The benchmark is fast and production is slow. Check the working-set size of the benchmark against the production tenant distribution. A benchmark that fits in the last cache level measures a different machine than the one serving your largest account.

Latency steps up after a resize or a rehash. Growing a hash table reallocates and rehashes, and during that window every lookup touches a different memory region and pays a miss. Pre-size the table from observed cardinality, or grow it in place within one memory area so the access path stays contiguous.

One node in a group is consistently slower. Compare NUMA placement, instance generation and co-tenants before touching the application. A pinned workload that landed on the wrong socket reproduces exactly this symptom, and no amount of profiling inside the process will reveal it.

None of these are exotic. They are the ordinary consequences of running software whose cost is dominated by where the data lives. The teams that get ahead of them are the ones that treat memory location as a first-class production metric, on the same dashboard as request rate and error budget burn.

More in-depth coverage from this blog on closely related topics:

Rust LSP Low Memory: Swap Language Servers

Sources and References

Sources cited while researching and writing this article: