TensorFold: Local LLMs on Mac and NVIDIA

Run tensorfold serve on a fresh Mac and it prints a listening URL on port 8080, with no API key required. TensorFold is an inference server written in Zig. It reads checkpoints from disk, applies the chat template, tokenizes requests and dispatches Metal or CUDA kernels directly, so the serving path contains no Python interpreter and no MLX install. The catch is the default. Anyone on your coffee-shop Wi-Fi can hit your model until you pass --api-key.

The server does support keys, through repeated --api-key flags, a comma-separated TENSORFOLD_API_KEY, or a --api-key-file whose permissions must be 0600. It simply does not require one until you configure it. That single default shapes how the rest of this engine should be deployed, because everything else about it is built for zero-friction installs.

The project’s GitHub repository states the design plainly, and its documented Linux install snippet pins the release archive at V=1.2.5, with the releases page carrying the per-architecture tarballs and checksums. The engine is Apache-2.0. The model weights are a different licensing question entirely.

What Ships in the Binary

There is no service mesh of Python packages to reconcile. The executable reads a checkpoint directory, applies the tokenizer and chat template, and runs kernels written for the specific model family. On macOS the Homebrew tap wraps the engine and its dependencies into one install command, which is why the directory listing for the project describes it as a single install step rather than a stack to assemble.

On Apple Silicon, the install is one Homebrew command:

brew install ashhart/tensorfold/tensorfold

The API Surface: OpenAI and Anthropic Shapes on One Port

The default bind is http://127.0.0.1:8080/v1, and one process answers both wire formats. Clients built against the OpenAI shape and clients built against Anthropic’s Messages shape can point at the same server at the same time, which is unusual enough to be the second-best reason to try it after the install story.

Route	What it accepts	Header a client sends
`POST /v1/chat/completions`	OpenAI chat, streamed or not, with tools and reasoning fields	`Authorization: Bearer KEY`
`POST /v1/completions`	Raw text completion without a chat template, or token IDs	`Authorization: Bearer KEY`
`POST /v1/responses`	OpenAI Responses API, run as an equivalent chat completion	`Authorization: Bearer KEY`
`POST /v1/messages`	Anthropic Messages: text, function tools, thinking, JSON or SSE	`x-api-key: KEY`

Streaming is a request field on the same route. Add "stream": true and the reply arrives as server-sent events, with the final event carrying usage. A first request looks like this:

curl -fsS http://127.0.0.1:8080/v1/chat/completions \
 -H 'Content-Type: application/json' \
 -d '{"model":"local-model","messages":[{"role":"user","content":"Summarise the retry policy in one line."}],"max_tokens":128,"stream":true}'
# Note: no auth header, which only works while the server has no keys configured.

For an SDK client, the migration is a base URL swap. Apps already speaking the OpenAI or Anthropic wire formats move to local inference by repointing the base URL, with no client-side code change and no SDK exchange, as the project write-up puts it. The header convention follows the shape: OpenAI-style clients send Authorization: Bearer KEY, and Messages clients send x-api-key: KEY.

The consequence is concrete. A coding agent or internal tool that was pointed at a hosted endpoint can be pointed at a laptop instead, and the only thing that has to exist on the laptop is the model directory and the binary.

API Auth Is Off by Default

Authentication exists, and it is optional. Three sources feed the key set and they add together: repeated --api-key KEY flags, a comma-separated TENSORFOLD_API_KEY environment variable, and --api-key-file PATH. The file takes one key per line, allows label: key entries, and treats lines starting with # as comments.

The file path is the one worth using, and the server enforces that intention. Permissions must be 0600; startup refuses a key file that other users can read. Internally the keys become SHA-256 digests, and every configured digest is compared on each authentication attempt, so the plaintext is not held in memory the way a naive string comparison would require. Rotating means replacing the file, and a reload does not require a restart.

The default is the part that needs saying out loud. A review of the server’s auth path traces the history: version 0.6.0 had no API auth at all, keys arrived in 0.6.5 and 1.0, and they stay off unless you set them. With no keys configured, the health route returns queue detail rather than a bare status, and binding to a non-loopback address prints a warning rather than refusing.

A safe setup takes one flag and one decision. Keep the bind on loopback for anything single-user, and when you need another machine to reach the model, add a key file first:

mkdir -p ~/.tensorfold
printf 'agent: %s\n' "$(openssl rand -hex 32)" > ~/.tensorfold/key
chmod 600 ~/.tensorfold/key
tensorfold serve mlx-community/Qwen3.5-2B-MLX-4bit --name local-model --context 8192 --no-thinking --api-key-file ~/.tensorfold/key

A labelled entry can be limited to inference routes, read routes, or both, while an unbracketed key keeps every route. A monitoring key gets [read]; an agent key gets [inference]. Request logs and the request counter identify keys by their label, such as cli-1 or file-2, never by the secret itself.

The consequence: on a shared build box or a LAN, a default install is open to anyone who can reach port 8080, and the fix is a flag you have to remember to pass.

Qwen3.5-2B and the Model Pull Path

The binary and the weights are separate downloads with separate licenses. The engine is Apache-2.0; checkpoints carry their own terms, and the repository says so in the quick start rather than leaving it to discovery. That distinction matters before you plan a deployment around a specific checkpoint.

The release-binary path is tensorfold pull with a pinned revision, followed by tensorfold serve. The pinned commit is what makes the test reproducible, since an unpinned pull can land on different weights next week:

tensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3
tensorfold serve mlx-community/Qwen3.5-2B-MLX-4bit --name local-model --context 8192 --no-thinking
# Note: serves text only. Check the memory floor for larger checkpoints before committing disk.

Qwen3.5-2B is the small checkpoint the project documents as checked on Apple Silicon and on DGX Spark hardware, which makes it the right first pull before you commit disk and memory to a 27B or 30B model. The build-from-source recipe for the same model takes a different route, using a download into a local directory and a zig build for a preview engine, so pick the path that matches what you installed. For the release binary, pull and serve is the shorter road.

Before serving anything bigger, ask the binary: tensorfold info REPO reports a checkpoint’s family, format, context and memory floor, and tensorfold models lists what is already cached. The consequence is that you size the machine from the checkpoint, not from a blog post.

Minimal Mode, Full Mode and the CLI

The Mac app and the CLI are separate products with separate version numbers, and the numbers drift. The project site advertises app 1.0.0 with engine 1.2.0, while the how-to guide is written against app beta 8 with engine 1.1.0, released 18 October 2026. Both are true of different artifacts. Read the version line after you install instead of assuming either.

Inside the app, the guide splits behaviour into two modes. Minimal is a local model server: Server, Models and an API connection panel, with a menu-bar option that keeps the process running when the window closes. Full adds chats, bots, projects, rooms and developer panes. Choosing a mode does not send a message or load a model; those stay deliberate actions, and closing the Minimal window leaves the server up until you quit the app.

Headless use is the CLI’s job. The same engine that the app bundles runs a server you can script, restart and pin independently of the app’s release cadence. That matters when an app update would otherwise move the engine under a running workload.

Two runtime behaviours are worth knowing before an agent fills the window. Every reply reports context_window and context_used, so a client can see how close a conversation is to the limit rather than inferring it from failures. Compaction is off by default: with --compact-at set, a conversation that would overflow is compacted instead of refused, and the same request produces the same compaction and the same reply. Adding --compact-memory lets the stored note carry over between requests. The consequence is that the default is stateless across requests, and memory only persists when you ask for it.

Where the Portability Claim Stops

The shipped backends are Metal on Apple Silicon and CUDA on NVIDIA, with Linux release archives for both linux-aarch64 and linux-x86_64. The tagline is broader than the documented install paths. The repository description reads “LLM Inference Engine for Metal, CUDA and Vulkan”, but the install guide and the release archives cover Metal and CUDA. Treat Vulkan as a name in a description until you find a build you can install and a guide that covers it. Planning a Vulkan deployment from the tagline alone is how a weekend disappears.

Performance claims deserve the same treatment. One review puts exact speculative decoding at 1.6 to 3.1 times faster than vLLM, but attributes those figures to the project rather than to an independent measurement. Treat that as a design argument, not a procurement benchmark. Measure on your hardware before you quote a multiplier.

A compiled kernel image is not a qualification for a specific card either. Apple Silicon and NVIDIA are the two families the install guide actually covers, and a machine outside them needs a build that exists before it needs a plan.

A defensible starting configuration is short: install from Homebrew or the checksummed archive, verify with capabilities --json, pull Qwen3.5-2B at its pinned revision, generate a key file with 0600 permissions, and only then widen the bind. That sequence keeps the default open port a local-only fact, which is what it was designed to be. For sizing the machine underneath it, our GPU guide covers the memory math before the install matters. Teams that need one governed endpoint across a fleet of these engines have a separate option in the project’s managed rollout line, which is a different deployment model from the binary described here.

Sources: tensorfold.dev, the how-to guide, ashhart/TensorFold on GitHub, the release pages, .

More in-depth coverage from this blog on closely related topics:

Sources and References

Sources cited while researching and writing this article: