Improving Decision Models with Jeeves

On September 23, 2026, Nokia Applied Research released AnyJev, a training-free layer that converts an open language model into a typed decision system. In a Jeeves-style support workflow, a payment outage report can produce four probabilities: billing, technical, sales, and other. Application code immediately routes a confident result and sends an uncertain one to an engineer.

A specialized decision head did not consistently improve accuracy. In the independent Visual Jev project, answer-supervised post-training raised equal-weight macro accuracy from 70.6% to 76.1% across four visual benchmarks, but its dedicated decision-head configuration matched the 76.1% accuracy of the existing language-model head. Improving task training had a greater effect than using a more complex readout.

The improvements remained close to the training distribution. Results increased sharply on GQA and SNLI-VE, the task families included during adaptation, while held-out TextVQA and TallyQA showed little change. Targeted adaptation can improve structured judgments, but it does not create general reasoning ability.

The Limits of Fast Decisions

Standard language models solve fixed-choice tasks by generating text token by token. An application then parses the text, checks its schema, and converts it into an action. This wastes resources when the required output is a known label such as billing, technical, or sales.

The Jev Decision Framework

A Jev-like interface starts with input state, isolated questions, and a finite answer space. The model assigns each valid candidate a logit. Softmax converts those logits into probabilities:

p(cj | state, question) = exp(zj) / sum(exp(zl))

This operation places every allowed answer on one scale and measures its share of the total score. The system does not generate a JSON object token by token. Application code serializes the selected label and distribution after inference.

TypeSafe calls Jev a System One model because it targets quick judgments. The company’s published service price is $0.042 per million input tokens with no output-token charge, while reported response times range from 70 to 500 milliseconds. Those are vendor claims, not guarantees for every workload. The Register’s launch coverage notes that a structured probability can be confidently wrong.

A deliberate layer can break down an incident, retrieve evidence, check constraints, or determine whether an answer is safe to execute. A final Jev-style call converts the resulting state into machine-readable options. This keeps expensive reasoning away from routine cases without treating classification as multi-step logic.

How Training Improves Decision Quality

Visual Jev uses Qwen3-VL-4B-Instruct as its main backbone, freezes the vision tower, and applies LoRA to the language tower. Training uses 30,416 GQA Choice items and 9,000 SNLI-VE Claim items for 3,000 updates with a batch size of eight.

The original 4B backbone reached 70.6% equal-weight macro accuracy across GQA, SNLI-VE, TextVQA, and TallyQA. Answer-supervised post-training raised it to 76.1%.

Held-out results were nearly flat. Training improved represented task families without establishing general reasoning gains across unseen visual tasks.

The 8B experiment showed the same boundary. A higher aggregate did not mean universal progress.

The study also tested a specialized typed decision head. Its 4B decision cross-entropy configuration reached the same 76.1% macro accuracy as the language-model-head approach, with no consistent advantage. Adapting the backbone mattered more than adding a complex output module.

Calibration Without More Reasoning

Nokia Applied Research’s AnyJev converts an open model into a typed decision system by reading its next-token distribution, without additional model training. Its corrections address prior bias toward particular labels and position bias within an option list.

AnyJev’s L0 mode rotates K options through K cyclic positions, combines results in log space, and applies batch prior correction after observing inputs. L1 adds temperature scaling fitted from 100 to 500 labeled examples. Temperature scaling adjusts confidence without changing candidate rankings.

In a reported BANKING77 experiment with Qwen3-8B, raw next-token scoring reached 74.7% accuracy. Expected calibration error fell from 0.240 for raw logits to 0.184 with L0 and 0.095 with L1. At a 5% error target, cases eligible for automatic handling rose from 7.7% to 46.3% under L0 and 52.0% under L1, according to the published AnyJev benchmark summary.

These are measurement improvements, not deeper reasoning. Cyclic shifts reduce option-order sensitivity, prior correction offsets label preference, and temperature scaling aligns confidence with observed outcomes. The underlying knowledge remains unchanged.

Benchmark Comparison

Method	Evaluation	Before	After	What Changed	Source
Visual Jev 4B answer SFT	Four-task macro accuracy	70.6%	76.1%	LoRA adaptation on GQA and SNLI-VE answer data	Visual Jev paper
AnyJev L0	BANKING77 accuracy with Qwen3-8B	74.7%	80.3%	Cyclic option shifts and batch prior correction	AnyJev summary
AnyJev L1	BANKING77 expected calibration error	0.240	0.095	L0 corrections plus fitted temperature scaling	AnyJev summary
Visual Jev shared execution	Warm amortized time at 32 questions	Independent serial baseline	8.9x faster	Shared visual prefix and batched isolated suffixes	Visual Jev abstract

Supervised adaptation raises task accuracy, calibration makes confidence more useful for thresholds, and shared execution reduces repeated computation. A production pipeline can combine these, but the metrics remain separate.

A Practical Decision Pipeline

This AnyJev example routes support messages through Qwen3-8B using the public package interface. Execution policy remains outside the model. Production use also requires authentication controls, bounded caches, telemetry, and a labeled evaluation set from local tickets.

from anyjev import Decider, Question
from anyjev.backends.hf import HFBackend

decider = Decider(HFBackend("Qwen/Qwen3-8B"))

route_question = Question.choice(
 "Which team should handle this support request?",
 ["billing", "technical", "sales", "other"],
 name="route",
)

state = {
 "conversation": [
 {
 "role": "customer",
 "content": (
 "Our payment integration started rejecting valid cards "
 "after yesterday's deployment. Checkout is unavailable."
 ),
 }
 ]
}

result = decider.decide(state, [route_question])
distribution = result["route"].distribution

best_route = max(distribution, key=distribution.get)
best_probability = distribution[best_route]

if best_probability >= 0.85:
 print({"action": "auto_route", "team": best_route})
else:
 print({
 "action": "human_review",
 "candidate": best_route,
 "probability": best_probability,
 })

# Note: production use should validate calibration on local labels,
# add retry and timeout handling, protect customer data, and log
# model/version metadata without storing unnecessary message content.

The threshold policy carries more operational weight than the model call. Its cutoff should reflect a validation set and each error’s cost. Missing an urgent payment outage differs from sending a routine pricing question to the wrong queue.

Low-confidence cases can invoke slower reasoning to inspect deployment history or retrieve an incident record before another decision call with enriched state. Log the original probability, enriched-state probability, selected action, and reviewer outcome to track calibration drift.

Programming code used to integrate an AI decision model — Application code should own thresholds, fallbacks, and audit records rather than hiding them in a prompt.

Trade-offs and Failure Modes

Training gains are narrow. Visual Jev improved most on GQA and SNLI-VE, which appeared in its post-training data. A team adapting a model for claims triage should build a separate labeled claims-triage test rather than import accuracy from a visual benchmark.

Calibration requires extra computation. AnyJev’s cyclic shifts require K prefills for K options, although they can share a prefix and run as a batch. Its reported configuration took about 0.25 seconds per decision at batch 32 on one H100 with 20 options.

Shared execution uses more memory. Visual Jev’s 8.9x speedup is an amortized throughput result measured on one RTX 5090 after warm-up. Prefix sharing and batched suffixes increase peak memory. The timing excludes network transfer, serving queues, image decoding, and disk access, so it is not end-user latency.

Probabilities can mislead. A model can return a valid schema and a high score for the wrong label. Data drift, revised labels, and changing customer language can alter the relationship between confidence and accuracy, so calibration needs post-deployment measurement.

Simple classifiers remain credible alternatives. With stable labels and many reviewed examples, a conventional fine-tuned classifier may be easier to audit and cheaper to host. A Jev-like interface fits changing answer sets, several questions sharing one long state, or semantic decisions made before enough data exists for a dedicated classifier.

Deployment Guidance for 2026

Use a Jev-style layer for finite answer spaces tied to explicit downstream actions, including ticket routing, document triage, tool selection, rubric scoring, and semantic guardrails. Keep free-form writing, negotiation, diagnosis, and complex planning in a reasoning or human-review path.

Measure task accuracy: Evaluate local, human-labeled examples rather than agreement with another model.
Measure calibration: Group predictions by confidence and compare each group with observed correctness.
Test option order: Reverse and rotate candidate lists to detect position bias.
Add abstention: Send low-confidence or high-cost cases to a slower solver or reviewer.
Version the policy: Record the model, prompt, candidate definitions, threshold, and fallback for every automated action.
Recheck drift: Monitor accuracy and calibration separately after labels, products, or customer language change.

Visual Jev shows that supervised adaptation can raise quality on represented task families. AnyJev shows that correcting option and confidence bias can improve automation without retraining. Shared-prefix execution processes related decisions efficiently when they inspect the same context.

A production “Jeeves” system can combine these lessons: use reasoning to prepare evidence, a typed decision layer to expose probabilities, and explicit software policy to act, escalate, or abstain. This preserves the speed advantage without assigning Jev-like models reasoning abilities their benchmarks do not establish.

More in-depth coverage from this blog on closely related topics:

Sources and References

Sources cited while researching and writing this article: