How to Train a 4B Model

A 4 billion-parameter language model (the “4B” in its model name, empero-ai/Qwen3.8-4B-Distill), post-trained with supervised fine-tuning and agentic reinforcement learning, reduced query execution time by up to 81% compared to PostgreSQL’s own planner on join-heavy analytical workloads. This result comes from a single engineer’s experiment, documented at rohanbansal.com/qorl, and it addresses a question that has influenced the database field for a decade: can a small, open-weights model learn to guide a mature optimizer more effectively than the optimizer guides itself?

Why Postgres Leaves Speed on the Table

Query optimization involves searching through a rapidly growing number of possible plans, and Postgres handles this using statistics rather than exact data.

Consider a three-table join over the IMDb dataset. Each join can use hash, merge, or nested-loop methods; each has two orientations (which input is outer, which is inner); each table can be read in four ways: sequential, index, index-only, or bitmap. A three-table query therefore has thousands of distinct ways to execute, and a twelve-table query from the Join Order Benchmark (JOB) reaches an extremely large number of possible plans, as the author of the qorl experiment explains.

Postgres cannot evaluate all of those. It applies dynamic programming, and for queries with twelve or more joins it switches to a genetic algorithm, according to the PostgreSQL GEQO documentation. More fundamentally, it cannot count rows during planning. It queries pg_statistic for histograms and most-common-value frequencies, then assumes uniform distribution when joining tables. That assumption causes most bad plans. For example, if 5% of companies are Japanese but those companies produced 50% of movies, the uniform assumption underestimates the intermediate join by a factor of ten, and one bad estimate affects the entire join tree.

Leis et al. documented this gap in 2015, then revisited it a decade later and found optimizers still leave significant performance on the table, as the qorl writeup references. The key idea behind the 4B model’s success is that verifying a plan is inexpensive even when finding one is difficult. A good plan runs quickly; a bad plan runs slowly. Execution time alone provides a clear reward signal.

Database servers in a data center running analytical workloads — Analytical workloads running the same suboptimal plan thousands of times are where learned optimization pays for itself.

The 4B Model and Its Agent Harness

The experiment began with a 4 billion-parameter model because it was small enough to train and run on a home rig with a pair of consumer GPUs. The base model is empero-ai/Qwen3.8-4B-Distill, a distillation from German lab Empero that used Qwen 3.8’s larger model as teacher to train a 4B student. The author notes the distilled model scores better on MMLU and slightly worse on GSM8K than its base, and chose it anyway.

The model does not replace Postgres. It guides it. Postgres always picks the lowest-cost plan, and you cannot change its cost model without patching the source. The third-party extension pg_hint_plan bypasses this by letting structured comments above a SQL statement force join types, scan types, and join order. A hint like + HashJoin(b) SeqScan(a) compels Postgres to use a hash join and a sequential scan. The model’s role is to generate those hints.

Around the model sits a lightweight agent harness, qo-agent, exposed through six tools: inspect_relation, get_column_stats, get_plan, evaluate_candidate, keep_default, and finish. The model emits structured PlanAction JSON objects, and each call to evaluate_candidate compiles that object into a hint, prepends it to the query, runs it once, and reports the measured latency as a ratio against Postgres’s default plan.

Training the Model to Write Hints

The training pipeline has two stages. Supervised fine-tuning (SFT) first teaches the model hint format, because any downstream gain depends on producing syntax that pg_hint_plan accepts. Then a custom variant of Group Relative Policy Optimization (GRPO) reinforces the model toward hints that reduce measured latency.

The custom reward design is important because the environment is noisy. The author built a measurement rig that minimizes Linux page cache contention across concurrent containers, then split reinforcement learning across two machines: vLLM and the trainer on a rented dual-H100 node, with four Postgres containers running locally. A separate off-policy distillation step ran across roughly five hundred GPT-6 Astra agent trajectories.

A sample trajectory shows how the agent reasons. It first calls get_plan("default") to see Postgres’s own plan and estimates. It then proposes a candidate join tree, receives feedback that the candidate is slower than the default plan, tries a nested-loop variant that returns faster than the default, and finally selects the faster candidate. Each rollout is capped at a small number of candidate submissions, forcing the model to learn efficient search rather than brute-force enumeration.

Here is the shape of the hint the model learns to produce, compiled from a PlanAction object:

SELECT cn.name, COUNT(*) AS titles
FROM title AS t, movie_companies AS mc, company_name AS cn
WHERE t.id = mc.movie_id
 AND mc.company_id = cn.id
 AND cn.country_code = '[jp]'
 AND t.prod_year BETWEEN 2000 AND 2009
GROUP BY cn.name
ORDER BY titles DESC
LIMIT 10;

The hint forces company_name to join movie_companies first with a hash join, before joining the result to title. The Japanese-company filter is more selective than the date-range filter, so this ordering reduces the intermediate result sooner. For production use, the harness validates each hint with EXPLAIN before execution, and any hint that fails to parse or produces an invalid plan is rejected rather than run.

Benchmarks Against Postgres

The headline numbers from the qorl experiment are specific and should be considered under their exact conditions. Across 113 join-heavy queries, the post-trained 4B model achieved a 44.7% reduction in latency, starting from a base model that could not produce a valid plan for 99 of those 113 queries at all. The 81% figure represents the top end of the distribution: the best single-query speedups reached roughly 5.5x on the hardest analytical cases, corresponding to an 81% reduction in execution time.

The result aligns with peer-reviewed work showing the same pattern. SEFRQO, a self-evolving RAG-based query optimizer accepted to SIGMOD 2026, reported up to 65.05% latency reduction on the CEB workload and 93.57% on Stack compared to PostgreSQL, using fine-tuned LLMs to generate hints through pg_hint_plan. LLM-QO generated execution plans directly in autoregressive fashion rather than through hints, and also outperformed both traditional and learned optimizers across three workloads.

System	Approach	Reported latency reduction vs. PostgreSQL	Source
qorl (4B model)	SFT + agentic RL, hint generation	44.7% mean across 113 queries; up to ~81% on hardest cases	rohanbansal.com/qorl
SEFRQO	Fine-tuned LLM + RAG, hint generation	65.05% on CEB; 93.57% on Stack	arXiv:2508.17556
LLM-QO	Two-stage fine-tuning, direct plan generation	Outperforms traditional and learned optimizers on three workloads	arXiv:2502.05562

Where the Speedup Holds and Where It Does Not

The gains vary, and conditions influence the results as much as the headline numbers. The author states that the goal is not to beat Postgres on one-off queries. Postgres’s planner runs in microseconds; any model inference adds overhead that outweighs the benefit for a query that runs once. The value is in heavy analytical workloads where the same query runs thousands of times under a suboptimal default plan. Training may require executing the query tens to hundreds of times upfront, but that cost spreads across every subsequent run.

The strongest speedups come from multi-join, high-cardinality queries where the uniform-distribution assumption fails most. Simpler queries, where Postgres’s cost model is accurate, show little or no improvement, and the model can even produce slower plans if it proposes a bad candidate. The harness’s keep_default tool exists so the agent can recognize when Postgres’s own plan is already the best available and decline to override it.

Inference latency also involves a trade-off. The first idea the author considered, building a better cardinality estimator, was abandoned because inference latency alone would outweigh any learned benefit against Postgres’s ultra-fast planner. The successful approach targets repeated queries rather than one-off cases, which is why the framing is offline optimization rather than a drop-in planner replacement.

Practical Integration

The simplest integration path is the one the experiment uses: keep Postgres as the execution engine, and add the model on top as a hint generator. The code is open source at github.com/polyphilz/qorl, a small, actively maintained Python project. The integration flow has three steps.

First, capture the query and its statistics. The harness gathers column types, nullability, index definitions, and estimated row counts through inspect_relation and get_column_stats. Second, run the model to propose candidate plans, validating each against EXPLAIN before execution. Third, measure the winning candidate against the default and cache it for reuse, so the expensive search happens once per query shape rather than once per execution.

This approach is an offline optimization layer for a specific workload, not a general-purpose planner upgrade. It requires a training period against your own schema and query mix, a measurement rig that controls for page-cache noise, and a fallback path for the cold-start period before the model has learned anything useful. For teams already running analytical workloads on Postgres, the payoff is clear: same hardware, running the same queries, with a measurable fraction of execution time removed.

More in-depth coverage from this blog on closely related topics:

Sources and References

Sources cited while researching and writing this article: