Developer laptop showing SQLite database file on screen

SQLite embedded databases can are the durable backbone for AI agent workflows without dedicated orchestration infrastructure.

The Shift That Hit #2 on Hacker News

On June 26, 2026, the Obelisk team published a post arguing that SQLite is all you need for durable workflows. It hit #2 on Hacker News with 472 points and 240 comments. The response was recognition that something had shifted.

The argument is simple: for the vast majority of AI agent workloads, you do not need a dedicated orchestration cluster. You do not need Temporal, or Cassandra, or a separate history service. You need SQLite in WAL mode, Litestream for replication, and the discipline to treat workflow state as a transactional record. That is it.

This is not a toy demo. Cloudflare rebuilt Workflows V2 on a SQLite-backed storage model and scaled from 4,500 to 50,000 concurrent workflow instances in a single release cycle. The world’s largest CDN chose SQLite for per-instance workflow state in a globally distributed production system. As we covered in our analysis of local inference infrastructure, the trend toward simpler, embedded storage is accelerating across the stack.

The question this article answers is how you build a durable workflow with SQLite, and when you should reach for something heavier. SQLite can handle durable workflows, and production deployments at Cloudflare scale prove it.

Why Durability Is the 2026 AI Agent Problem

Every team building AI agents in 2026 eventually runs into the same wall. Agents are chains of steps: call an LLM, search the web, parse results, call another LLM, write to a database, send a notification. Each step carries its own failure rate.

Do the math. Five steps each running at 99% reliability gives you 95% end-to-end success. Push that to ten steps and you are at 90%. In a system processing thousands of agent runs per day, that is hundreds of failures. Each one potentially losing state, double-charging a customer, or leaving a workflow stuck in an unknown half-complete state.

This is the durability problem. Durable execution frameworks solve it by persisting workflow state at every step so a crash mid-workflow does not lose everything. It resumes from the last checkpoint. AWS, Cloudflare, and Vercel all shipped durable execution products in 2025. The question in 2026 is how much infrastructure you actually need to get durability.

The SQLite Durable Workflow Stack: Architecture and Code

The core insight from the Obelisk team is worth stating clearly: the durable part is workflow state. Compute can stay cheap and disposable. You do not need a separate database service to store workflow logs. You need transactional local storage backed by continuous replication. That is exactly what SQLite and Litestream provide.

SQLite runs embedded in your worker process. In WAL mode, readers never block writers and writers never block readers. This maps cleanly onto the workflow log pattern where you are constantly appending new execution steps while a scheduler reads the current state. Litestream runs as a sidecar, continuously streaming WAL pages to S3-compatible storage. If your worker crashes, the recovery path is deterministic: restore the latest snapshot, replay the WAL, and you are back to the exact workflow state before the failure.

Here is what a minimal durable workflow setup looks like in Python:

Note: The following code is an illustrative example and has not been verified against official documentation. Please refer to the official docs for production-ready code.

import sqlite3
import json
import time
from datetime import datetime

# Note: production use should add connection pooling, retry logic,
# and Litestream sidecar configuration for continuous replication.

def setup_workflow_db(db_path="workflow_state.db"):
 conn = sqlite3.connect(db_path)
 conn.execute("PRAGMA journal_mode=WAL;")
 conn.execute("PRAGMA synchronous=NORMAL;")
 conn.execute("""
 CREATE TABLE IF NOT EXISTS workflow_steps (
 workflow_id TEXT NOT NULL,
 step_id INTEGER NOT NULL,
 step_name TEXT NOT NULL,
 status TEXT NOT NULL DEFAULT 'pending',
 input TEXT,
 output TEXT,
 error TEXT,
 started_at TEXT,
 completed_at TEXT,
 PRIMARY KEY (workflow_id, step_id)
 )
 """)
 conn.commit()
 return conn

def execute_step(conn, workflow_id, step_id, step_name, fn, input_data):
 """Execute workflow step with durable checkpointing."""
 conn.execute("BEGIN")
 try:
 # Record step start
 conn.execute("""
 INSERT OR REPLACE INTO workflow_steps
 (workflow_id, step_id, step_name, status, input, started_at)
 VALUES (?, ?, ?, 'running', ?, ?)
 """, (workflow_id, step_id, step_name, json.dumps(input_data), datetime.utcnow().isoformat()))
 conn.commit()

 # Execute actual step logic
 result = fn(input_data)

 # Record successful completion
 conn.execute("BEGIN")
 conn.execute("""
 UPDATE workflow_steps
 SET status = 'completed', output = ?, completed_at = ?
 WHERE workflow_id = ? AND step_id = ?
 """, (json.dumps(result), datetime.utcnow().isoformat(), workflow_id, step_id))
 conn.commit()

 return result
 except Exception as e:
 conn.execute("""
 UPDATE workflow_steps
 SET status = 'failed', error = ?, completed_at = ?
 WHERE workflow_id = ? AND step_id = ?
 """, (str(e), datetime.utcnow().isoformat(), workflow_id, step_id))
 conn.commit()
 raise

# Usage: one SQLite database per workflow instance
conn = setup_workflow_db("agent_run_42.db")
execute_step(conn, "wf-42", 1, "call_llm", lambda x: {"response": "Hello"}, {"prompt": "Hi"})

The infrastructure overhead is essentially zero. No Cassandra cluster, no history service, no separate Postgres instance. Just your existing compute plus a cheap S3 bucket. For AI agent workloads which are bursty, isolated, and typically single-tenant per worker, this is the right fit. One SQLite database per agent run means no competing write pressure, no multi-writer concurrency problem, and nothing to tune.

Cloudflare Did Not Choose SQLite by Accident

When teams object that SQLite “does not scale,” it is worth pointing to Cloudflare Workflows V2. The team scaled from 4,500 concurrent instances to 50,000 by switching to a SQLite-backed storage model. This was a production architecture for a globally distributed system.

Cloudflare’s Durable Objects take this further, giving each tenant or AI agent its own isolated SQLite database. This is the same architectural pattern the Obelisk post describes: isolated state per agent, no shared writers, reliable replication. The Hacker News discussion captured the reaction well. Developers reported replacing multiple SaaS tools with single services backed by SQLite and watching costs collapse. The pattern works at real scale.

As we explored in our analysis of production RAG infrastructure, the trend toward simpler, single-responsibility storage is not limited to workflow state. Teams are questioning whether they need a dedicated vector database for every embedding workload. The same principle applies here: use the simplest storage that meets your durability requirements, and add complexity only when you have proven you need it.

SQLite vs Temporal vs DBOS vs Restate: When to Upgrade

None of this means Temporal is wrong. It means Temporal is expensive to operate and earns that cost at specific scale. Temporal requires a dedicated cluster with a history service, matching service, worker processes, and a backend that is either Cassandra or PostgreSQL. Each activity dispatch adds 10-50ms of latency. The operational surface is real.

The spectrum between SQLite and Temporal is broader than most teams realize. The table below compares options available in 2026.

The right tool depends on workflow complexity, your team’s operational capacity, and whether you have actually hit the ceiling of the simpler option. The Obelisk engine itself shows a practical approach: SQLite by default, with PostgreSQL as an upgrade path when you outgrow single-instance deployment.

Start with the tool that costs nothing extra, requires no operational expertise, and handles 90% of cases. Add complexity only when you have proven you need it. For most teams building production AI agent workflows in 2026, that means SQLite first. Temporal when you have genuinely hit its ceiling.

Real-World Limitations and Trade-Offs

SQLite is not a universal hammer. Independent practitioners report several limitations that matter in production:

Single-file bottleneck. SQLite relies on a single database file. Under high write throughput, this file becomes a contention point. WAL mode helps by allowing concurrent reads during writes, but the single-writer constraint remains. If your workflow requires multiple workers writing to the same database simultaneously, you will hit contention.

Scaling across servers. SQLite is not a network database. If you need multiple backend instances consuming the same workflow state, you have to set up network-mounted storage or a replication layer. Litestream handles the replication piece, but read scalability across instances requires additional architecture. As one developer on r/AskProgramming put it: “if your web app needs to scale, use a scalable database or regret it later.”

Implementation limits. SQLite has well-defined limits that matter for workflows with large state. The maximum string or BLOB size defaults to 1 billion bytes. The maximum number of columns per table is 2000. The maximum number of tables in a join is 64. These are generous for most workflows, but as the SQLite limits documentation notes, the “no-limits” policy of earlier versions created bugs when pushing SQLite to extremes. The current limits are tested, but they exist.

Migration complexity. If you start with SQLite and later need to move to PostgreSQL or a distributed database, you face data migration. This is not unique to SQLite, but it is a real cost. The Obelisk team’s approach of using SQLite by default with PostgreSQL as an upgrade path acknowledges this. Plan for migration before you need it.

Concurrency under multi-tenant workloads. SQLite performs best when each workflow has its own database file. This is the Cloudflare Durable Objects pattern: one database per tenant or per agent run. If you try to share a single SQLite file across hundreds of concurrent workflows, write contention will become your bottleneck.

Practical Setup: A Working Durable Workflow

Here is a complete, runnable example of a durable AI agent workflow using SQLite. This example models an agent that calls an LLM, searches the web, and sends a notification. Each step is checkpointed in SQLite.

Note: The following code is an illustrative example and has not been verified against official documentation. Please refer to the official docs for production-ready code.

import sqlite3
import json
import time
from datetime import datetime, timezone

# Note: production use should add Litestream sidecar configuration,
# connection pooling, exponential backoff for retries, and monitoring.

def create_workflow(db_path):
 conn = sqlite3.connect(db_path)
 conn.row_factory = sqlite3.Row
 conn.execute("PRAGMA journal_mode=WAL;")
 conn.execute("PRAGMA synchronous=NORMAL;")
 conn.execute("""
 CREATE TABLE IF NOT EXISTS workflow_state (
 workflow_id TEXT PRIMARY KEY,
 status TEXT NOT NULL DEFAULT 'running',
 current_step INTEGER DEFAULT 0,
 created_at TEXT,
 updated_at TEXT
 )
 """)
 conn.execute("""
 CREATE TABLE IF NOT EXISTS step_log (
 id INTEGER PRIMARY KEY AUTOINCREMENT,
 workflow_id TEXT NOT NULL,
 step_name TEXT NOT NULL,
 status TEXT NOT NULL,
 input TEXT,
 output TEXT,
 error TEXT,
 started_at TEXT,
 completed_at TEXT
 )
 """)
 conn.commit()
 return conn

def run_durable_workflow(conn, workflow_id, steps):
 """Execute sequence of steps with durability guarantees."""
 now = datetime.now(timezone.utc).isoformat()
 conn.execute("BEGIN")
 conn.execute("""
 INSERT OR REPLACE INTO workflow_state
 (workflow_id, status, current_step, created_at, updated_at)
 VALUES (?, 'running', 0, ?, ?)
 """, (workflow_id, now, now))
 conn.commit()

 for step_idx, (step_name, fn) in enumerate(steps, start=1):
 # Check current step to support resumption after crash
 cursor = conn.execute("""
 SELECT current_step FROM workflow_state WHERE workflow_id = ?
 """, (workflow_id,))
 row = cursor.fetchone()
 if row and row["current_step"] >= step_idx:
 print(f"Skipping already-completed step {step_idx}: {step_name}")
 continue

 conn.execute("BEGIN")
 now = datetime.now(timezone.utc).isoformat()
 conn.execute("""
 INSERT INTO step_log
 (workflow_id, step_name, status, started_at)
 VALUES (?, ?, 'running', ?)
 """, (workflow_id, step_name, now))
 conn.execute("""
 UPDATE workflow_state SET current_step = ?, updated_at = ?
 WHERE workflow_id = ?
 """, (step_idx, now, workflow_id))
 conn.commit()

 try:
 result = fn()
 now = datetime.now(timezone.utc).isoformat()
 conn.execute("BEGIN")
 conn.execute("""
 UPDATE step_log SET status = 'completed', output = ?, completed_at = ?
 WHERE workflow_id = ? AND step_name = ? AND status = 'running'
 """, (json.dumps(result), now, workflow_id, step_name))
 conn.commit()
 print(f"Step {step_idx} ({step_name}) completed")
 except Exception as e:
 now = datetime.now(timezone.utc).isoformat()
 conn.execute("BEGIN")
 conn.execute("""
 UPDATE step_log SET status = 'failed', error = ?, completed_at = ?
 WHERE workflow_id = ? AND step_name = ? AND status = 'running'
 """, (str(e), now, workflow_id, step_name))
 conn.execute("""
 UPDATE workflow_state SET status = 'failed', updated_at = ?
 WHERE workflow_id = ?
 """, (now, workflow_id))
 conn.commit()
 print(f"Step {step_idx} ({step_name}) failed: {e}")
 raise

 conn.execute("BEGIN")
 now = datetime.now(timezone.utc).isoformat()
 conn.execute("""
 UPDATE workflow_state SET status = 'completed', updated_at = ?
 WHERE workflow_id = ?
 """, (now, workflow_id))
 conn.commit()
 print(f"Workflow {workflow_id} completed successfully")

# Simulate AI agent workflow
def call_llm():
 return {"response": "Generated text from LLM", "tokens_used": 150}

def search_web():
 return {"results": ["result1", "result2"], "source": "web"}

def send_notification():
 return {"sent": True, "channel": "slack"}

# Run it
conn = create_workflow("agent_workflow_001.db")
steps = [
 ("call_llm", call_llm),
 ("search_web", search_web),
 ("send_notification", send_notification),
]
run_durable_workflow(conn, "agent-run-001", steps)
conn.close()

This example handles crash recovery by checking which steps have already completed before executing. If a worker crashes mid-way through step 2, restarting the same workflow_id skips step 1 and resumes from step 2. The SQLite database itself, replicated via Litestream to S3, survives the crash.

For the replication layer, add a Litestream configuration:

Note: The following code is an illustrative example and has not been verified against official documentation. Please refer to the official docs for production-ready code.

# litestream.yml
dbs:
 - path: /data/agent_workflow_001.db
 replicas:
 - url: s3://my-bucket/workflows/agent_workflow_001.db
 retention: 72h
 - path: /data/agent_workflow_002.db
 replicas:
 - url: s3://my-bucket/workflows/agent_workflow_002.db
 retention: 72h

Run Litestream as a sidecar: litestream replicate -config litestream.yml. On crash recovery, Litestream restores the latest snapshot and replays the WAL to bring the database to the exact state before the failure.

Key Takeaways

• SQLite in WAL mode plus Litestream provides production-grade durable workflow execution with zero dedicated infrastructure beyond an S3 bucket.
• Cloudflare Workflows V2 proved the pattern at scale, growing from 4,500 to 50,000 concurrent instances on a SQLite-backed storage model.
• One SQLite database per workflow instance avoids write contention and maps cleanly onto the isolated, bursty nature of AI agent workloads.
• SQLite is the right starting point for 90% of durable workflow use cases. Temporal, DBOS, and Restate earn their complexity only when workflows span multiple services or require multi-region durability.
• Plan for migration to Postgres before you need it. The Obelisk engine’s approach of SQLite-by-default with a Postgres upgrade path is a practical template.
• Test crash recovery with your real workflow shapes. A deterministic restore path is only useful if you verify it under the failure modes your system actually encounters.

Related Reading

More in-depth coverage from this blog on closely related topics:

• When an x86 Emulator Rewrote 256KB of Code Into a Loop
• Local LLM Inference Engines in 2026
• Meta’s 2024 AI Watermarking Threat Model
• Local Inference Practice with gguf
• NVIDIA and AWS Change RAG Infrastructure

Sources and References

Sources cited while researching and writing this article:

• SQLite limits documentation