A Disposable Postgres for Ephemeral Dev Environments in 300 ms

FN2 is built mostly by AI agents. On a busy day there are somewhere between 20 and 50 of them working at once: fixing bugs, building features, reviewing each other’s code, all pushing branches around the clock. A human team pushes a handful of branches a day and can coordinate over a shared staging environment. A fleet of agents can’t. Nobody is going to Slack an agent to ask if staging is free.

So every pushed branch gets its own full preview environment: its own Kubernetes namespace, its own pods, its own URL at p-<hash>.test.fn2.ai. These are ephemeral by design. One comes up when a branch is pushed, lives as long as the branch does, and gets destroyed without ceremony when the branch merges or is abandoned. Nothing in them is precious; any of them can be deleted and rebuilt from scratch in seconds, and the agents neither know nor care which one they’re on.

One preview, unpacked:

            p-8c4d21ea.test.fn2.ai
                      │
                      ▼
┌─ k8s namespace: fn2-p-8c4d21ea ────────────────────────┐
│                                                        │
│ stock-web        nginx edge for this branch            │
│ stock-server     API · PGHOST=storage1.internal :16112 │
│ livesearch       search, pointed at the same fork      │
│ migrate (init)   runs this branch's migrations         │
│                                                        │
│ created by: git push · destroyed with the branch       │
└─────────────────────┬──────────────────────────────────┘
                      │  netpol pinhole
                      │  tcp 16000-16299
                      ▼
     storage1 · fn2-pgfork@p-8c4d21ea · Postgres 17 :16112
     zfs clone of seed@v20260719160208
     created in ~300 ms · destroyed in 444 ms

I’ve had that part running for a while and it’s genuinely great. It also had a dirty secret: every one of those “isolated” environments talked to the same shared test database.

If you’ve ever run previews against a shared DB, you know how this movie goes. Two previews write to the same tables and see each other’s data. Tests get flaky in ways nobody can reproduce. And the dumbest part: a branch that added a migration couldn’t be previewed at all. I had a CI gate that flat-out refused to deploy a preview if the branch touched the schema, because running one branch’s migration against the shared database would break every other preview, plus the test environment itself. The feature whose entire point is “see your branch running” didn’t work for exactly the branches you most need to see running.

So over the weekend I built the obvious thing: copy-on-write. Every preview now forks its own disposable Postgres from a golden ZFS snapshot in about 300 milliseconds, runs its branch’s migrations inside it, and takes the whole database to the grave when the preview dies. Nine of them running at once cost 454 MB of disk, total.

Some notes on how it works, why it’s fast, and the corners I cut on purpose.

The clean shutdown is the whole trick

CoW database branching isn’t a new idea. Neon built a storage engine around it, postgres.ai’s Database Lab Engine does it with ZFS, and half the “database branching” startups are selling some flavor of the same thing. If you self-host, ZFS hands you the primitive for free: zfs clone of a snapshot is O(1). You pay in metadata, and new blocks only get written where the fork diverges.

But one easy-to-miss detail decides whether your clone is ready in 300 milliseconds or 30 seconds: whether Postgres was shut down cleanly before the snapshot was taken.

Snapshot a running Postgres and the clone is crash-consistent. Perfectly safe, but every clone boots into crash recovery and replays WAL before it takes a single connection. Do that per preview and your “instant” fork isn’t. So the seed pipeline builds the golden image like this:

pg_dump the test database  →  restore into a fresh local PGDATA
  →  ANALYZE  →  clean shutdown  →  zfs snapshot seed@v<timestamp>

The snapshot captures a Postgres that believes it exited gracefully. Clones of it start with zero WAL replay. That one property is most of the speed budget:

283 ms from curl to pg_isready. It holds under load too: eight forks back to back land in 2.5 seconds, and the whole fleet of nine running databases shares 454 MB because every unmodified block is literally the same block:

The architecture is aggressively boring

One ZFS dataset. One ~400-line Python daemon, stdlib only. systemd. It runs directly on my SSD-backed storage host. Not in Kubernetes, no operator, no CRDs.

The whole flow, from push to connected app:

flowchart LR
  push[branch push] --> ctl[preview controller]
  ctl -->|POST /v1/forks| forkd[pgforkd on the storage host]
  forkd -->|zfs clone + systemd unit| fork[("fork: Postgres 17, ready in ~300 ms")]
  ctl -->|helm upsert with PGHOST / PGPORT| ns[preview namespace]
  ns -->|NetworkPolicy pinhole| fork
  subgraph seed [nightly seed build]
    refresh[seed-refresh] -->|pg_dump| test[(shared test DB)]
    refresh -->|"restore, clean shutdown, zfs snapshot"| gold[("golden seed@vN")]
  end
  gold -.->|zfs clone| fork

The golden seed gets rebuilt nightly: pg_dump -Fc of the test database (plus pg_dumpall --globals-only for the role passwords), restored into a fresh initdb, ANALYZE, clean stop, snapshot. 27 seconds end to end for the ~300 MB database. Old snapshots are pruned with zfs destroy -d, which defers destruction until the last clone referencing them is gone, so a seed refresh can never yank the floor out from under a live preview.

pgforkd, the daemon, is a small HTTP API. POST /v1/forks {"id": "p-1a2b3c4d"} does zfs clone, starts a systemd template unit, waits for pg_isready, and hands back a host and port. DELETE tears it down in 444 ms. Both are idempotent, so the preview controller can retry blindly.

Each fork is a real systemd unit with its own cgroup: MemoryMax=3G, CPUQuota=400%, a 40 GB refquota. That means forks survive daemon restarts, because systemd owns the processes; restarting the daemon never touches a running fork. A reaper thread enforces a TTL backstop so a crashed cleanup can’t leak databases forever.

On the preview side, the controller asks for a fork during upsert, injects PGHOST/PGPORT into the pods, and runs the branch’s own migrations inside the fork via an init container. The schema-preflight gate is deleted. A branch that rewrites half the schema now previews like any other branch, which was the whole point.

Isolation being the whole point, I checked it the direct way: commit a write in a fork, then go look for it in the shared database:

Tradeoffs, before you ask

systemd on the storage host. The box that holds the ZFS pool runs the forks directly: one daemon, one template unit per fork, everything caged in a systemd slice with low CPU and IO weight so the box’s day job always wins contention. The alternative was a ZFS-aware CSI driver, PersistentVolumes, and a StatefulSet per preview: three new failure domains to save zero user-visible latency. Forks also run with fsync=off, which reads like a war crime until you remember they’re disposable. Worst case after a power loss, the next upsert recreates them from the seed in 300 ms; the seed build itself is fully durable.

A port pool instead of SNI routing. The elegant version routes p-abc123.db.internal through one endpoint via SNI. But the Postgres wire protocol starts cleartext and upgrades to TLS with a STARTTLS-style dance, so an SNI router has to actually speak the protocol (Traefik has a well-known hack for this), or you need PG17’s sslnegotiation=direct on every client. DLE, the reference implementation for exactly this kind of system, ships port pools. A 300-port firewall pinhole to one internal host costs nothing and needs no TLS rollout. If forks ever leave the private network, I’ll revisit.

Dump/restore instead of pg_basebackup. A physical basebackup would be faster to build and byte-identical. The logical dump is version-proof (the test cluster and the fork host run different Postgres 17 builds), it naturally produces the cleanly-shut-down PGDATA that makes clones instant, and it’s a convenient seam for scrubbing data if the test DB ever holds anything sensitive. The cost: seed builds are O(database size). At 27 seconds for ~300 MB, don’t care. At 100x, I’d switch to a DLE-style standby-and-promote pipeline.

Fail closed, no fallback. If the fork API is down, preview creation fails. It does not quietly fall back to the shared database. A silent fallback would reintroduce cross-preview write coupling exactly when nobody’s looking, and I’d rediscover it weeks later as an unexplainable flaky test. Broken and loud beats subtly wrong.

TTL is only a backstop. Forks die when their preview dies; that’s the real cleanup path. The 14-day TTL exists only to collect orphans from crashed destroys. I deliberately did not tie fork lifetime to the reconciler’s short-lived leases: a fork dying under a live preview is much worse than a leaked 50 MB clone.

Stuff I didn’t build: per-fork credentials (forks share the test DB’s auth, the same trust domain as the shared database they replaced), point-in-time recovery, branch-from-branch, cross-host HA. All real work, none of it earns its keep for disposable test databases.

Did it work?

A CI push now goes from webhook to fully-ready preview (namespace, pods, its own migrated database, edge health green) in about 17 seconds, of which the database fork is 0.3. The database went from “the reason some branches can’t preview at all” to a rounding error.

My favorite part: I didn’t announce anything. Minutes after the rollout, one of the agents pushed a perfectly ordinary branch. No CI changes, no opt-in flag. It got its own database, ran its migrations in it, and came up green:

The shared test database is still there, serving the persistent test environment. No preview will ever write to it again.

The stack: ZFS (lz4, 16 K recordsize), PostgreSQL 17, systemd template units, a dependency-free Python daemon, and a Kubernetes preview controller speaking to it over HTTP. Prior art worth your time: postgres.ai Database Lab Engine and Neon’s storage engine, which land on the same insight from opposite directions.