How to compile code quickly with JIT speed

A query that executes in 5 milliseconds cannot afford 200-millisecond compiler startup. That mismatch has kept many databases and short-lived runtimes on interpreters, even when native code would run much faster. The pgrust project changes that calculation: its author reports compiling each SQL query in around 5 microseconds, fast enough to make runtime code generation part of the common path rather than a costly optimization reserved for a few hot queries.

The mechanism is copy-and-patch compilation. Instead of asking a general-purpose compiler to select instructions, allocate registers, optimize an intermediate representation, and emit machine code, a copy-and-patch JIT selects prebuilt machine-code templates called stencils. It copies those bytes into a code buffer and fills holes with constants, branch targets, register offsets, and addresses known only at runtime.

Key Takeaways

• The reported 5-microsecond result belongs to pgrust’s specialized JIT and should not be treated as a universal time for arbitrary programs.
• Copy-and-patch moves expensive compiler work ahead of runtime by preparing machine-code stencils for supported operations.
• On the published regex benchmark, generated ARM64 code ran between 11.7x and 19.7x faster than the interpreter across five tested input lengths.
• Xu and Kjolstad’s copy-and-patch compiler generated SQL query code two orders of magnitude faster than LLVM -O0 and three orders of magnitude faster than higher LLVM optimization levels on their TPC-H evaluation.
• Microsecond compilation gives up some optimization freedom and requires architecture-specific code, executable-memory management, instruction-cache handling, and extensive correctness tests.

Why 5 Microseconds Matters in 2026

Just-in-time compilation converts code to machine instructions while a program is running. The runtime can compile a method, query, bytecode block, or another code fragment and cache the result for subsequent calls. This combines the flexibility of interpretation with direct execution on the processor, but compilation delays the first execution.

Turning Generated Bytes Into Executable Code

That delay is acceptable when generated code runs for minutes. It becomes a serious problem when a database query, regular expression, or short WebAssembly function completes in milliseconds. A compiler that spends more time optimizing a program than the optimized program saves in execution time is inefficient.

The pgrust JIT article reports a compile time of around 5 microseconds. That figure describes a specialized compiler that targets ARM64 in the documented example. It does not imply LLVM, a JVM compiler, or a Python JIT can translate any function in the same time. This result matters because it shows what becomes achievable after limiting the input language, target architecture, and optimization scope.

The concept has a long history. John McCarthy discussed runtime translation in work on LISP in 1960. Ken Thompson used runtime code generation for regular-expression matching on the IBM 7094 in 1968. Smalltalk later compiled code on demand and cached the result. Modern runtimes refined those ideas through profiling, tiered compilation, and recompilation.

Runtime compilation is most effective when its startup cost stays below the execution time it saves.

Start With the Common Case: A Runnable Interpreter

A fast JIT requires a clear execution model before generating machine code. The following complete Rust program implements the same small regular-expression model used in the pgrust article: literal strings, concatenation, and repetition. It represents an expression as an AST and interprets it through continuation-style matching.

enum Node {
 Literal(&'static str),
 Concatenation(Box<Node>, Box<Node>),
 Repetition(Box<Node>),
}

fn literal(text: &'static str) -> Node {
 Node::Literal(text)
}

fn concatenation(left: Node, right: Node) -> Node {
 Node::Concatenation(Box::new(left), Box::new(right))
}

fn repetition(body: Node) -> Node {
 Node::Repetition(Box::new(body))
}

fn match_node(
 node: &Node,
 input: &[u8],
 position: usize,
 next: &dyn Fn(usize) -> bool,
) -> bool {
 match node {
 Node::Literal(text) => {
 let remaining = input.get(position..).unwrap_or(&[]);
 let bytes = text.as_bytes();

 remaining.starts_with(bytes)
 && next(position + bytes.len())
 }

 Node::Concatenation(left, right) => {
 match_node(left, input, position, &|left_end| {
 match_node(right, input, left_end, next)
 })
 }

 Node::Repetition(body) => {
 match_node(body, input, position, &|body_end| {
 body_end > position
 && match_node(node, input, body_end, next)
 }) || next(position)
 }
 }
}

fn is_match(expression: &Node, input: &str) -> bool {
 let bytes = input.as_bytes();
 match_node(expression, bytes, 0, &|position| position == bytes.len())
}

fn main() {
 let expression = concatenation(
 literal("b"),
 repetition(literal("an")),
 );

 for candidate in ["b", "ban", "banan", "banana", "analytics"] {
 println!("{candidate}: {}", is_match(&expression, candidate));
 }
}

The interpreter is the common-case implementation since it is easy to inspect and portable across processors. It also exposes overhead that compiled code can eliminate. Each AST node requires a match, function calls or closures, position checks, and repeated traversal through generic node structures. Those operations dominate when the underlying work is byte comparison and pointer increment.

A JIT specializes the generic tree for one expression. For b(an)*, it can emit a direct comparison for b, two comparisons for a and n, a backward branch for repetition, and a shared failure block. The AST disappears from the execution path after compilation.

How Copy-and-Patch Removes Compiler Work

Traditional native-code compilation typically passes through an intermediate representation. The compiler performs instruction selection, register allocation, control-flow analysis, and optimization before it writes the final bytes. Those stages improve generated code, but they consume time and allocate temporary data structures.

Copy-and-patch performs much of that work before the application runs. An ahead-of-time process creates a library of binary implementation variants. Each variant contains machine code plus holes for values unavailable until runtime. Xu and Kjolstad call these variants stencils in their copy-and-patch paper.

A runtime compiler then performs a small sequence of operations:

• Read the next AST node or bytecode operation.
• Select the corresponding stencil.
• Copy its machine-code words into the destination buffer.
• Patch operands such as constants, register offsets, and function addresses.
• Resolve control-flow targets after output positions are known.
• Make the finished buffer executable and synchronize the instruction cache.

This design is fast because the runtime performs less decision-making. It does not search for an instruction sequence since the selected stencil already contains one. It does not run a general register allocator because each template already defines its register use. Runtime work becomes predictable copying and integer arithmetic.

The trade-off is specialization. Every supported operation needs a suitable stencil, and machine-code templates are tied to an instruction set. The Cognica implementation says it maintains approximately 110 stencils for each supported target architecture, x86-64 and ARM64, in its January 2026 implementation article. Adding a new opcode or processor target increases the code generator’s maintenance and testing burden.

A Runnable ARM64 Stencil Example

The pgrust example directly constructs ARM64 instructions as u32 words. The next program does not execute those words, so it runs safely as a normal Rust utility on any platform. It generates the documented prologue, match, and failure stencils and prints their encoded words. This is a useful first step before working with executable memory since developers can compare emitted values against a disassembler.

Note: The following code is an illustrative example and has not been verified against official documentation. Please refer to the official docs for production-ready code.

const PROLOGUE_WORDS: usize = 1;
const MATCH_WORDS: usize = 4;
const FAIL_WORDS: usize = 6;

fn stencil_prologue() -> [u32; PROLOGUE_WORDS] {
 [
 0xAA0103E2, // mov x2, x1
 ]
}

fn stencil_match(
 stencil_position: usize,
 failure_position: usize,
) -> [u32; MATCH_WORDS] {
 [
 0x39400009, // ldrb w9, [x0]
 0x35000009
 | cond_branch_offset(
 stencil_position + 1,
 failure_position,
 ), // cbnz w9, fail
 0xD2800020, // mov x0, #1
 0xD65F03C0, // ret
 ]
}

fn stencil_fail() -> [u32; FAIL_WORDS] {
 [
 0xEB02003F, // cmp x1, x2
 0x54000060, // b.eq +3
 0xA9FF0029, // ldp x9, x0, [x1, #-16]!
 0xD61F0120, // br x9
 0xD2800000, // mov x0, #0
 0xD65F03C0, // ret
 ]
}

fn cond_branch_offset(
 branch_position: usize,
 target_position: usize,
) -> u32 {
 let instruction_count =
 target_position as i64 - branch_position as i64;

 (((instruction_count as u64) & 0x7FFFF) as u32) << 5
}

These stencils illustrate a key property: the runtime does not assemble instructions from scratch. It copies the pre-encoded words and patches only the branch offset field. The encoding for cbnz and b.eq includes an immediate offset relative to the current instruction, which is why the position of each stencil in the final buffer matters.

Executable Memory and the JIT Runtime

Generating machine code is only half of the work. The runtime must place those bytes into executable memory, manage write protection, and synchronize the instruction cache. The following excerpt shows the macOS-specific approach used in the pgrust article.

Note: The following code is an illustrative example and has not been verified against official documentation. Please refer to the official docs for production-ready code.

type Matchfn = unsafe extern "C" fn(
 input: *const u8,
 backtrack_stack: *mut usize,
) -> u64;

struct Jit {
 buffer: *mut u32,
 byte_length: usize,
 backtrack_stack: Vec<usize>,
}

impl Jit {
 fn compile(machine_code: &[u32]) -> Jit {
 let byte_length =
 std::mem::size_of_val(machine_code);

 unsafe {
 let buffer = libc::mmap(
 std::ptr::null_mut(),
 byte_length,
 libc::PROT_READ
 | libc::PROT_WRITE
 | libc::PROT_EXEC,
 libc::MAP_PRIVATE
 | libc::MAP_ANON
 | libc::MAP_JIT,
 -1,
 0,
 ) as *mut u32;

 assert!(
 buffer as *mut libc::c_void
 != libc::MAP_FAILED,
 "mmap failed"
 );

 pthread_jit_write_protect_np(0);

 std::slice::from_raw_parts_mut(
 buffer,
 machine_code.len(),
 )
 .copy_from_slice(machine_code);

 pthread_jit_write_protect_np(1);

 sys_icache_invalidate(
 buffer as *mut libc::c_void,
 byte_length,
 );

 Jit {
 buffer,
 byte_length,
 backtrack_stack:
 vec![0; BACKTRACK_STACK_MAX * 2],
 }
 }
 }

 fn is_match(&mut self, input: &[u8]) -> bool {
 debug_assert_eq!(input.last(), Some(&0));

 unsafe {
 let matcher: Matchfn =
 std::mem::transmute(self.buffer);

 matcher(
 input.as_ptr(),
 self.backtrack_stack.as_mut_ptr(),
 ) != 0
 }
 }
}

impl Drop for Jit {
 fn drop(&mut self) {
 unsafe {
 libc::munmap(
 self.buffer as *mut libc::c_void,
 self.byte_length,
 );
 }
 }
}

This excerpt is intentionally not presented as a portable drop-in program. It uses APIs specific to macOS JIT memory handling and executes raw ARM64 code. The exact sequence matters: generated bytes must be written while the region permits writes, execution protection must be restored, and the instruction cache must be synchronized before calling the new function.

The null-terminated input contract also matters. The generated regex code relies on a zero byte to stop character matching without a separate length check for each character. Calling it with an ordinary byte slice that lacks a terminator allows generated code to read beyond valid input. A debug assertion is useful during development, but a production API should enforce the contract at its boundary.

Published Benchmark Results

The pgrust article compares an interpreted regex engine, a generated ARM64 matcher, and code written specifically for b(an)*. The table below reproduces the five published rows. Every value comes from that benchmark and belongs to its test implementation and environment.

The generated matcher stays close to the handwritten implementation. At 9 bytes, both take 3.8 ns. At 129 bytes, the JIT records 30 ns compared with 32 ns for the handwritten function. At 2,049 bytes, handwritten code is faster at 393 ns versus 470 ns, but the generated matcher still reduces the interpreter’s 8,301 ns result by 17.7x.

Compilation cost changes how these execution numbers should be used. A 5-microsecond compilation takes much longer than one 3.8-nanosecond match. Compilation pays off only after the generated function runs enough times, processes enough data, or replaces sufficiently expensive interpretation. Databases can recover the cost while scanning rows because one compiled expression may execute for every row in a query.

Instruction selection is removed from the hot compilation path, but generated bytes still depend on the target processor.

Calculate the Break-Even Point

The basic decision rule is straightforward. Let compilation cost be C, interpreted time per call be I, and generated-code time per call be J. Compilation pays for itself after more than C / (I - J) calls, provided generated code is faster.

The following complete Rust program uses values in Cognica’s published example: 1ms compile cost, 12μs interpreter time, and 4μs generated-code time. It reports a break-even threshold of 125 calls.

fn break_even_calls(
 compile_microseconds: f64,
 interpreted_microseconds: f64,
 jit_microseconds: f64,
) -> Option<f64> {
 let saving_per_call =
 interpreted_microseconds - jit_microseconds;

 if saving_per_call <= 0.0 {
 return None;
 }

 Some(compile_microseconds / saving_per_call)
}

fn main() {
 let calls = break_even_calls(1000.0, 12.0, 4.0);

 match calls {
 Some(calls) => {
 println!("Break-even calls: {calls:.0}");
 }
 None => {
 println!(
 "Generated code does not reduce per-call time"
 );
 }
 }
}

This calculation is more useful than applying one fixed “hot” threshold to every operation. A function that saves 8 microseconds per call recovers a 1-millisecond compile in 125 calls. Another function that saves only a fraction of a microsecond needs far more calls. Query engines can also estimate work from row counts, while virtual machines can use call and loop counters.

Copy-and-Patch, Baseline JITs, and LLVM

Xu and Kjolstad evaluated copy-and-patch in two settings. Their SQL query compiler generated code two orders of magnitude faster than LLVM -O0 and three orders of magnitude faster than higher optimization levels on TPC-H. Its generated code ran an order of magnitude faster than interpretation and 14% faster than LLVM -O0.

Their WebAssembly compiler generated code 4.9x to 6.5x faster than Liftoff, Google’s baseline WebAssembly compiler. On Coremark and PolyBenchC, copy-and-patch output ran 39% to 63% faster than Liftoff’s generated code. These are results from the paper’s implementations and benchmark setup, not universal multipliers for every WebAssembly program.

LLVM remains valuable when execution is long enough to repay heavier optimization. It can analyze an intermediate representation across operations, allocate registers with wider context, remove redundant work, and transform loops. Copy-and-patch takes the opposite position: prepare many good instruction templates ahead of time, keep runtime compilation small, and accept less freedom to optimize across stencil boundaries.

Single-pass baseline compilers occupy another point in this design space. They avoid expensive global optimization and translate input in one pass, but they still perform code-generation decisions at runtime. Copy-and-patch removes more of those decisions by storing binary variants. Interpretation has the shortest startup since it emits no native code, but each operation continues to pay dispatch and decoding costs.

Production runtimes can combine these methods. Cognica describes an interpreter, a baseline copy-and-patch tier, and a later optimizing tier. Cold code starts immediately. Frequently executed code receives native stencils. The hottest code can later justify IR construction and optimization. This structure keeps startup low without giving up the option of better peak performance.

Production Pitfalls and Failure Modes

The first failure mode is unsafe generated code. An incorrect immediate field, branch offset, calling convention, or stack adjustment can corrupt memory. Unit tests should verify stencil words, but byte-level tests alone are insufficient. Differential tests should run the interpreter and generated code on the same inputs and compare results. Disassembly checks help detect accidental changes in encoded instructions.

The second is incorrect memory protection. Writable and executable memory creates a valuable attack target. The pgrust macOS implementation uses Apple’s JIT write-protection function to switch the mapping between writing and execution. A production runtime should keep write windows narrow, avoid exposing writable executable pages to unrelated threads, and release mappings when compiled code is evicted.

The third is stale instruction-cache state. Writing bytes to a data-visible mapping does not guarantee that the processor’s instruction path immediately sees them. The documented implementation calls sys_icache_invalidate before execution. Skipping that step can execute old instructions on systems where instruction and data caches are not automatically coherent for this operation.

The fourth is unbounded code growth. Compiling every unique query or expression creates a code-cache problem even when each compilation is fast. The runtime needs cache keys, eviction, lifetime tracking, and a policy for generated code referenced by active threads. Code size also affects instruction-cache behavior, so generating native code for rare paths can reduce performance.

The fifth is zero-progress repetition. In the interpreter example, a repeated child that succeeds without advancing input would recurse forever. The runnable implementation checks that body_end exceeds the prior position before repeating. A generated matcher needs the same semantic rule, either during AST validation or in emitted control flow.

The sixth is architecture drift. ARM64 and x86-64 require different instruction encodings, calling conventions, relocation rules, and cache handling. A stencil library must be treated as platform code, with dedicated continuous-integration coverage for each target. Generic compiler backends are larger, but they absorb much of this maintenance. A small direct code generator assigns that responsibility to the runtime team.

Where Microsecond Compilation Fits

Databases are a natural fit because query planning exposes runtime facts such as selected columns, expression structure, types, and constants. Generated code can remove interpreter dispatch from filters and aggregates that run once per row. The pgrust implementation uses its reported compile speed to JIT every SQL query rather than applying JIT only to selected workloads.

Regular-expression matching is another direct fit. Thompson’s 1968 implementation already connected runtime generation with pattern matching. A compiled expression can turn an AST walk into a compact sequence of character loads, comparisons, branches, and pointer increments. The pgrust benchmark shows the generated matcher staying near code written specifically for the tested expression.

WebAssembly runtimes also benefit because modules arrive as bytecode and need fast startup. Xu and Kjolstad’s results against Liftoff show that stencil-based compilation can compete with an existing baseline compiler on both generation time and output speed. The cost is a larger set of prebuilt variants and more architecture-specific machinery.

Microsecond compilation is less effective when source programs are large, optimization opportunities span wide regions, or generated code will run long enough to repay a heavy compiler. It also fits poorly when the platform prevents runtime machine-code generation. In those cases, interpretation, ahead-of-time compilation, or a conventional optimizing JIT can be a better engineering choice.

The main change in 2026 is that compiler design now includes a proven path between interpretation and heavyweight optimization. Copy-and-patch makes native code cheap enough for workloads that previously finished before their compiler did. For query engines, parsers, bytecode runtimes, and specialized language VMs, that shifts runtime compilation from a rare optimization into an ordinary execution tier.

Related Reading

More in-depth coverage from this blog on closely related topics:

• How to Use hdiutil in macOS
• Walmart Pay: How It Works
• Cybersecurity Stocks After CVE Updates
• How to Run Linux MicroVM on Mac
• Lessons from the August 17 Outage