Hapax
When an AI agent starts a long job — charge this card, send this email, book this flight — and the machine running it dies halfway through, the work is normally either lost or silently repeated. Hapax is the storage layer that makes neither possible.
hapax — Greek, “once”, from hapax legomenon: a word that occurs exactly once.
The experiment
kill -9. Same instant.Both columns charge one customer $50, and both are hard-killed at the identical moment — right after the money moves. The only difference is where each one keeps its record of having done the work. Press run and watch the two ledgers diverge.
Remembers what it charged in process memory
Records the work in the same transaction that does it
The harder case
Killing a process after the money moved is the easy half — the hard half is dying while the charge is half-written. Postgres unwinds the charge and the “task finished” flag together, because they were written together, so the retry finds a clean slate rather than a mess.
Measured, not asserted
Every figure below came off a benchmark in this repository that you can re-run against your own Postgres. Methodology is in bench/RESULTS.md.
kill -9s. The naive worker, same test: 121.SIGKILLs against a real Postgres, green in CI.| 500 randomly-timed kills | Hapax | Naive |
|---|---|---|
| Charged exactly once | 500 | 379 |
| Charged twice | 0 | 121 |
| Charge lost entirely | 0 | 0 |
The two crashes you watched above were chosen — the interesting boundaries. These 500 were not: each kill lands at a uniformly random moment in the worker's measured lifetime, so they fall wherever they fall, including inside the commit. About a quarter of them are enough to make the naive worker bill someone twice.
| Simultaneous retries | Naive: duplicate charges | Hapax |
|---|---|---|
| 2 | 1.00 | 0 |
| 8 | 7.00 | 0 |
| 16 | 14.99 | 0 |
Duplicates scale as roughly N−1 for the naive version: with 16 concurrent retries it charges the customer about 16 times. Averages over 200 trials per row.
| Backend | Operation | p50 | p99 | Throughput |
|---|---|---|---|---|
| In-memory | create | 0.003 ms | 0.009 ms | 256,657/s |
| Postgres | create | 0.122 ms | 0.621 ms | 6,950/s |
| Postgres | update | 0.164 ms | 0.750 ms | 5,163/s |
Surviving a crash costs roughly 30–50× in latency — the price of writing to disk instead of memory. Reported as percentiles rather than averages, because the tail is where a durable commit actually hurts.
Why it works
The side effect and the “this task is finished” flag are written in a single database transaction. A crash before the commit rolls back both, so the retry finds nothing half-done. A crash after it finds the task already marked finished and refuses to run again.
Deduplication is a UNIQUE index plus INSERT … ON CONFLICT DO NOTHING.
Checking “does this already exist?” in application code has a gap between the check and
the insert that two simultaneous requests can both slip through. A database constraint
has no such gap.
A worker takes a job by writing a deadline onto it. Finish, and the job is closed for good. Die, and nothing renews that deadline — it simply passes, and the job goes back in the queue for someone else. Nothing has to detect the crash or report it, because the silence is the report. Recovering takes 20 ms, and needs no human and no retry.
Five task states, with an explicit allow-list of legal moves between them. The three terminal states have zero outgoing transitions — that immutability is exactly what recovery relies on when it decides not to re-charge someone.
For an outside effect that cannot join the transaction — a real Stripe call — true exactly-once is impossible for anyone. What you get is at-least-once plus an idempotency key at the boundary. Hapax stores that key; the README says so plainly rather than overselling.