---
name: durability
description: Keep a rat-stack app standing when a provider, pool, or dependency slows down or a worker dies. Use at any I/O boundary, connection pool, provider call, shared waiter, fan-out, flag, gate, or work that can outlive a request, or when asked "what happens when this is slow", "add a timeout", "make this durable", or "what survives a crash".
plain: "Put recoverable work in storage, give recovery one owner, hold nothing across a slow call, bound every wait, and prove recovery after a crash."
diagram: |-
  request
    │ query ── lease, release
    │ commit + marker ── answer
    │   └ waitUntil ── provider
    │       one deadline
    │ sweep ── same key
    │ fan-out ── bounded
    ▼
  gates
    open on delay
    closed on harm
---

# Durability

> For agents: start with the [agent guide](https://ratstack.sh/llms.txt). Every page is Markdown by default; add `Accept: text/html` for HTML.

Free workshop: [Build Agent Loops That Never Stop Shipping](/tokenmaxx#interested).

**Effect is not BEAM.** Durability lives in storage. [Effect](/lore/effect-basics) runs each [fiber](/lore/concurrent-work-needs-an-owner) as a concurrent computation in one JavaScript runtime, not as an isolated process. Effect 4.0.0 has no OTP restart supervisor. Its structured fiber ownership is useful, but it does not rebuild lost state.

> Remember from chapters 8 and 9 that processes lose their state on termination.

Saša Jurić, *Elixir in Action*, Other Use Cases for ETS ([B2]). Even OTP restart needs a separate state-recovery decision.

A slow provider must not become our outage. Each pattern below uses an Effect primitive at a real seam. Each one has a [fence](/lore/the-fence) that fails on a planted copy of the bug. The public page is [/systems/durability](/systems/durability).

## When to use it

Load this skill before you change any of these:

- a database connection, socket, or provider SDK that a [handler](/systems/capabilities) reuses;
- a provider call that starts from a request;
- work that must survive the request, the isolate, or a deploy;
- a timeout or a retry;
- a loop or `forEach` over many items;
- a `Deferred`, `Cache`, `Ref`, `Semaphore`, `Latch`, or `PubSub` that callers share;
- a tunable that might need a deploy to change;
- a gate, check, or refusal that can stop work.

For ordinary pure functions, use [uncomplect](/skills/uncomplect). For finite modes, use [add a lifecycle machine](/skills/add-a-lifecycle-machine).

## Patterns

1. **Lease a connection for the query**
   - Principle: `principles/lease-connections-per-query.md`
   - Example: `packages/database/src/connection-lease.ts`
   - Fence: `packages/database/test/connection-lease.test.ts`
2. **Answer at commit; dispatch off the request**
   - Principle: `principles/answer-at-commit.md`
   - Example: `packages/core/src/join-intake.ts`, `packages/core/src/intake-sweep.ts`
   - Fence: `packages/core/test/join-answer-at-commit.model.test.ts`, `packages/core/test/intake-sweep.model.test.ts`
3. **Set a provider timeout once**
   - Principle: `principles/one-timeout-at-the-boundary.md`
   - Example: `packages/subscriber-delivery/src/drovr-intake.ts`, `packages/core/src/join-delivery-machine.ts`
   - Fence: `packages/subscriber-delivery/test/intake-timeout.test.ts`
4. **Bound work in flight**
   - Principle: `principles/bound-in-flight-work-with-keys.md`
   - Example: `apps/site/src/interest/applications.ts`
   - Fence: `apps/site/test/intake-applications-concurrency.test.ts`
5. **Scope waiters to the request**
   - Principle: `principles/scope-waiters-to-the-request.md`
   - Example: `rat-stack-patterns/shared-waiters`
   - Fence: `packages/core/test/patterns-rule.test.ts`
6. **Climb a flag with a ledger**
   - Principle: `principles/climb-flags-with-a-ledger.md`
   - Example: `packages/flags`, `packages/core/src/flag-change.ts`
   - Fence: `packages/flags/test/writes.model.test.ts`, `packages/flags/test/lifecycle.model.test.ts`
7. **Fail open on liveness, closed on harm**
   - Principle: `principles/fail-open-on-liveness.md`
   - Example: `packages/deploy/src/plan.ts`, `packages/core/src/intake-erase.ts`
   - Fence: `packages/deploy/test/plan.test.ts`, `packages/core/test/intake-sweep.model.test.ts`

The references hold the detail:

- [Effect map](https://github.com/joelhooks/rat-stack/blob/main/skills/durability/references/effect-map.md): each primitive, what it promises, its limit, and the version checked.
- [Request budget](https://github.com/joelhooks/rat-stack/blob/main/skills/durability/references/request-budget.md): the `waitUntil` ceiling, worked through join delivery.
- [Bibliography](https://github.com/joelhooks/rat-stack/blob/main/skills/durability/references/bibliography.md): the books, the chapters read, and checked quotes.

## Read the actual boundary

Read the project's instructions, the pinned versions, the [service](/lore/services-capture-dependencies), the storage [adapter](/lore/hexagonal-architecture), the acknowledgement path, and the restoration tests. Check Effect APIs against the installed source. The mappings here are our reading of Effect, not the book authors' advice about Effect.

For each accepted operation, name:

- the durable record that remains after the process disappears;
- the authority that decides its next step;
- the operation identity that every repeated side effect uses;
- the failure domain and the recovery owner;
- the acknowledgement point, and the outcome the caller can truthfully infer.

If no durable record exists, call the work best-effort. A running fiber, an in-memory queue, or a machine snapshot held only in memory is not that record.

## Work through a change

1. Name the slow thing: the provider, pool, or store, and the worst latency it can show.
2. List what the change holds while it waits: a connection, the request, a permit, or a shared waiter.
3. Name the durable record and the owner that redrives it.
4. Pick the pattern from the list. Read its principle file.
5. Put the primitive at one seam. Read the timeout from a `Context.Reference` or Config, never a literal.
6. Write a property or model test that generates the slow case and the crash.
7. Plant the bug. The test must fail. Restore the code.
8. Run `pnpm turbo run check test build`.

## Principles, ranked

Rank by risk of lost work, repeated effects, and cascading failure. The numbers are this skill's priorities, not a ranking from the books. Every Effect answer below exists in the pinned 4.0.0; the Effect map shows the check.

1. **Persist accepted intent and recovery progress. Name the backend.**
   - Failure prevented: A healthy restarted worker silently loses accepted work.
   - Effect answer and remaining work: `effect/workflow`: `Workflow`, `Activity`, `WorkflowEngine`. Select a persistent engine; `WorkflowEngine.layerMemory` is for tests. rat-stack's join intake uses a contact marker and a sweep instead.
   - Book evidence: [B1], [B2], [B3]
2. **Make repeated external effects safe before enabling retries.**
   - Failure prevented: A crash after remote success repeats a write.
   - Effect answer and remaining work: `Activity.idempotencyKey` names work. The receiver or authoritative store must enforce deduplication; drovr does it by submission id.
   - Book evidence: [B5], [B6]
3. **Couple state and checkpoint where one transaction can own both. Acknowledge only the promised handoff.**
   - Failure prevented: A checkpoint advances past missing work, or replay reapplies committed changes.
   - Effect answer and remaining work: `SqlClient.withTransaction`. No atomic transaction spans arbitrary providers.
   - Book evidence: [B5], Chs. 7, 9
4. **Give every task a lifetime owner and every failed worker a recovery owner.**
   - Failure prevented: Escaped tasks, unnoticed death, or endless restart churn.
   - Effect answer and remaining work: `Effect.forkScoped`, `Scope`, `FiberSet`. Add restart policy in XState or a host supervisor.
   - Book evidence: [B1], [B2], [B3]
5. **Bound waiting and carry the caller's remaining deadline.**
   - Failure prevented: Abandoned calls keep consuming scarce capacity.
   - Effect answer and remaining work: `Effect.timeoutOption`. A timeout does not prove a remote write stopped.
   - Book evidence: [B4] §5.1; [B12] Ch. 22
6. **Put retries at one informed boundary, with jitter and a finite budget.**
   - Failure prevented: Nested loops and synchronized retries amplify an outage.
   - Effect answer and remaining work: `Effect.retry` with `Schedule.exponential`, `jittered`, `recurs`, `upTo`. Shared retry accounting is separate policy.
   - Book evidence: [B4] §5.1; [B9]; [B10] §10.6; [B12] Ch. 22
7. **Bound admitted work and isolate scarce capacity by workload.**
   - Failure prevented: A noisy caller fills every slot and backlog grows without limit.
   - Effect answer and remaining work: Bounded `Effect.forEach`, `Semaphore`, `Queue.bounded`; `Pool` for a real resource pool.
   - Book evidence: [B3]; [B4] §§5.3, 5.6
8. **Stop calling a known-unhealthy dependency and probe recovery explicitly.**
   - Failure prevented: Retrying doomed calls exhausts the capacity recovery needs.
   - Effect answer and remaining work: No circuit-breaker API in 4.0.0. Model closed, open, and half-open in XState.
   - Book evidence: [B4] §5.2; [B11]
9. **Recover from known state, with stable replay inputs and recoverable deadlines.**
   - Failure prevented: Restart changes old decisions or forgets scheduled work.
   - Effect answer and remaining work: `Activity`, `DurableClock`, `DurableDeferred` under a persistent engine.
   - Book evidence: [B1]; [B3]; [B5] Ch. 6
10. **Preserve old payloads' meaning across deployments.**
    - Failure prevented: Old accepted work becomes undecodable during recovery.
    - Effect answer and remaining work: `Schema.decodeUnknownEffect` decodes; an optional key keeps old records valid. Define version support and transition tests.
    - Book evidence: [B5] Ch. 3; [B14] §3.11
11. **Park exhausted or malformed work durably and give it a return path.**
    - Failure prevented: Poison work disappears, retries forever, or parks invisibly.
    - Effect answer and remaining work: `PersistedQueue`, `DurableQueue` keep failure records. Redrive, visibility, and repair stay application work.
    - Book evidence: [B5] Ch. 9; [B9]
12. **Keep expected refusals, transient faults, and defects distinct.**
    - Failure prevented: A crash becomes a permanent refusal, or a refusal is retried.
    - Effect answer and remaining work: `Schema.TaggedError`, `Effect.catchTag`.
    - Book evidence: [B7]; [B10] Ch. 10
13. **Recycle accumulation and bound results at their source.**
    - Failure prevented: Retained records or a huge query exhaust storage or memory.
    - Effect answer and remaining work: `Cache`, `Scope`, `Stream.take`, `PersistedQueue.layerCleanup`. Add retention and query limits.
    - Book evidence: [B4] §§4.4, 5.4
14. **Prove recovery with deterministic histories and bad dependencies.**
    - Failure prevented: A design passes happy-path tests but loses work at the commit and acknowledgement gap.
    - Effect answer and remaining work: `TestClock`, `Deferred`, `Arbitrary`, `it.effect.prop`. Write the model, inject faults, test real storage separately.
    - Book evidence: [B4] §5.7; [B11]; [B15]
15. **Observe useful outcomes, backlog, and saturation.**
    - Failure prevented: Fast failures, parked work, or growing queues look healthy.
    - Effect answer and remaining work: `Metric.counter`, `gauge`, `histogram`. Dashboards and alerts sit outside the constructors.
    - Book evidence: [B8]; [B12] Ch. 26

> Protect critical clients by giving them their own pool to call.

Michael T. Nygard, *Release It!*, §5.3 ([B4]). One global limit does not reserve capacity for critical work.

## Choose what owns recovery

This is house composition, not a claim that the authors recommend these products.

- **Process-local resource and child lifetime**
  - Use: Effect `Scope`, acquire and release, scoped fibers
  - Do not claim: Cleanup after hard process death, separate heaps, or automatic restart
- **Post-response Worker work**
  - Use: `ctx.waitUntil` for bounded best-effort work; persist required work before responding
  - Do not claim: More than the 30-second shared allowance, or recovery after cancellation
- **Legal modes, retry and cancel events, restart thresholds**
  - Use: XState 6 with `@xstate/effect`
  - Do not claim: That a machine alone is a durable store or an OS supervisor
- **Per-entity durable state**
  - Use: A Durable Object adapter behind an Effect service
  - Do not claim: That an in-memory field or local timer survives restart
- **Authoritative relational state, dedupe ledger, checkpoint, or outbox**
  - Use: A Postgres adapter with explicit transaction and uniqueness contracts
  - Do not claim: That a local transaction includes a remote provider's side effect
- **Durable multi-step orchestration with no recovery owner yet**
  - Use: Effect workflow engine with persistent storage
  - Do not claim: That `layerMemory`, a detached fiber, or an API named Durable proves production durability
- **Heap, CPU, or fatal-runtime isolation**
  - Use: A separate process, worker, or service with a host recovery policy
  - Do not claim: That a `Scope`, `FiberSet`, or machine gives BEAM isolation

Keep one recovery authority for each operation. When an existing Durable Object or database already owns progress, adapt it; do not add a second ledger by default.

`Effect.forkDetach` is the detached-fiber name in 4.0.0; there is no `forkDaemon`. Detaching changes lifetime inside a live runtime. It does not persist work.

## Check the uncomfortable boundaries

Use the guarantee the [contract](/systems/capabilities) promises, not a generic claim of exactly-once execution.

- **Accepted, not yet started**: Kill the worker. The accepted identity stays discoverable in the durable store.
- **Remote effect succeeded, local result missing**: Retry the same identity. Show receiver deduplication or reconciliation. If neither exists, record the outcome as uncertain.
- **State committed, acknowledgement missing**: Redeliver. Prove state does not apply twice.
- **Retry exhausted**: Show a durable failure record, a visible owner, and a tested redrive.
- **Timer pending at crash**: Recover the persisted deadline. Test wakeup and cancellation.
- **Old payload after deploy**: Decode retained fixtures across versions.
- **Dependency slow, absent, or malformed**: Check the deadline, admission, typed failure, breaker transition, and capacity release.

These are proposed checks derived from the cited failure modes. Passing an in-memory test does not prove the backend's crash contract.

## Lore

The [durability index](/lore/durability) gives the reading order. Start with these pages:

- [Effect is not BEAM](/lore/effect-is-not-beam): fibers share one runtime and lose state with it.
- [Durable work needs storage](/lore/durable-work-needs-storage): accepted intent must live outside memory.
- [Request lifetime is not durability](/lore/request-lifetime-is-not-durability): `waitUntil` work needs a durable marker and redrive.
- [Timeouts bound waiting](/lore/timeouts-bound-waiting): a timeout stops the wait. The remote action may still finish.
- [Retry needs idempotent effects](/lore/retry-needs-idempotency) and [retry load needs a budget](/lore/retry-load-needs-a-budget).
- [Backpressure reaches the producer](/lore/backpressure-reaches-the-producer) and [bulkheads isolate capacity](/lore/bulkheads-isolate-capacity).
- [Failures keep their meaning](/lore/failures-keep-their-meaning): retry, refuse, and escalate by failure kind.
- [Recovery needs model tests](/lore/recovery-needs-model-tests): generate crash and retry sequences.

The [durability bibliography](/lore/durability-bibliography) lists the books and quotes behind these pages.

## Related skills

- [uncomplect](/skills/uncomplect): a held resource and a provider call in one scope are a braid to separate.
- [add-a-lifecycle-machine](/skills/add-a-lifecycle-machine): retries, backoff modes, breakers, and a degraded provider are lifecycles.
- [investigate-a-failure](/skills/investigate-a-failure): before raising a timeout, join the failure to the calls in flight.
- [ship](/skills/ship): the deploy driver's typed refusals and bounded waits hold pattern 7.

## Finish

Report the persistence owner, recovery owner, operation identity, acknowledgement boundary, remaining uncertainty, and checks run. Name what the design deletes. Leave a missing guarantee explicit.

## Bibliography

The [full bibliography](https://github.com/joelhooks/rat-stack/blob/main/skills/durability/references/bibliography.md) lists the chapters read, edition notes, and every checked quote.

- **[B1]** Martin Logan, Eric Merritt, Richard Carlsson. *Erlang and OTP in Action*. The Erlang/OTP Platform: Let It Crash. "The point of letting an entire subsystem terminate and be restarted is that it brings you back to a state known to function properly."
- **[B2]** Saša Jurić. *Elixir in Action*, third edition. Chapters 8–9; Other Use Cases for ETS. "Remember from chapters 8 and 9 that processes lose their state on termination."
- **[B3]** James Edward Gray II, Bruce A. Tate. *Designing Elixir Systems with OTP*. Customize Your Lifecycle. "Elixir can’t manage what it doesn’t know about."
- **[B4]** Michael T. Nygard. *Release It!* (2007). §5.1 Timeouts. "The timeout is a simple mechanism allowing you to stop waiting for an answer once you think it will not come."
- **[B5]** Adam Bellemare. *Building Event-Driven Microservices*. Chapter 6, Deterministic Stream Processing. "The overarching goal of deterministic processing is that a microservice should produce the same output whether it is processing in real time or catching up to the present time."
- **[B6]** Joe Reis, Matt Housley. *Fundamentals of Data Engineering*. Chapter 5, Message Queues. "In an idempotent system, the outcome of processing a message once is identical to the outcome of processing it multiple times."
- **[B7]** Scott Wlaschin. *Domain Modeling Made Functional*. Working with Errors. "These different kinds of errors require different implementations."
- **[B8]** Rob Ewaschuk. *Monitoring Distributed Systems*. The Four Golden Signals. "The four golden signals of monitoring are latency, traffic, errors, and saturation."
- **[B9]** Peter Sbarski, Yan Cui, Ajay Nair. *Serverless Architectures on AWS*, second edition. Chapter 6, Everything Fails, All the Time. "Do you put the failed event into a DLQ and process it later?"
- **[B10]** John Ousterhout. *A Philosophy of Software Design*. §10.3 Define Errors Out of Existence. "Rather than throwing an error when unset is asked to delete an unknown variable, it should have simply returned without doing anything."
- **[B11]** Fred Hebert. *Property-Based Testing with PropEr, Erlang, and Elixir*. Laying Out Stateful Properties. "Both function calls must be deterministic."
- **[B12]** Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy, editors. *Site Reliability Engineering*. Chapter 22, Addressing Cascading Failures. No checked quote.
- **\[B13]** Eric Normand. *Grokking Simplicity*. Chapters 15–17, timelines. No checked quote.
- **[B14]** Scott W. Ambler, Pramod J. Sadalage. *Refactoring Databases*. §3.11. "many refactorings require you to run both the original and new versions of the schema in parallel during a transition period that is long enough to allow any applications accessing that portion of the schema time to be redeployed."
- **[B15]** Titus Winters, Tom Manshreck, Hyrum Wright, editors. *Software Engineering at Google*. Chapter 15, Deprecation. No checked quote.
- **\[B16]** Andreas Zeller. *Why Programs Fail*. Chapter 4, Reproducing the Problem Environment. No checked quote.

[B1]: https://github.com/joelhooks/rat-stack/blob/main/skills/durability/references/bibliography.md#b1-erlang-and-otp-in-action

[B2]: https://github.com/joelhooks/rat-stack/blob/main/skills/durability/references/bibliography.md#b2-elixir-in-action

[B3]: https://github.com/joelhooks/rat-stack/blob/main/skills/durability/references/bibliography.md#b3-designing-elixir-systems-with-otp

[B4]: https://github.com/joelhooks/rat-stack/blob/main/skills/durability/references/bibliography.md#b4-release-it

[B5]: https://github.com/joelhooks/rat-stack/blob/main/skills/durability/references/bibliography.md#b5-building-event-driven-microservices

[B6]: https://github.com/joelhooks/rat-stack/blob/main/skills/durability/references/bibliography.md#b6-fundamentals-of-data-engineering

[B7]: https://github.com/joelhooks/rat-stack/blob/main/skills/durability/references/bibliography.md#b7-domain-modeling-made-functional

[B8]: https://github.com/joelhooks/rat-stack/blob/main/skills/durability/references/bibliography.md#b8-monitoring-distributed-systems

[B9]: https://github.com/joelhooks/rat-stack/blob/main/skills/durability/references/bibliography.md#b9-serverless-architectures-on-aws

[B10]: https://github.com/joelhooks/rat-stack/blob/main/skills/durability/references/bibliography.md#b10-a-philosophy-of-software-design

[B11]: https://github.com/joelhooks/rat-stack/blob/main/skills/durability/references/bibliography.md#b11-property-based-testing

[B12]: https://github.com/joelhooks/rat-stack/blob/main/skills/durability/references/bibliography.md#b12-site-reliability-engineering

[B14]: https://github.com/joelhooks/rat-stack/blob/main/skills/durability/references/bibliography.md#b14-refactoring-databases

[B15]: https://github.com/joelhooks/rat-stack/blob/main/skills/durability/references/bibliography.md#b15-software-engineering-at-google

## Lore on this page

- [Durable work needs storage](/lore/durable-work-needs-storage)
- [Timeouts bound waiting](/lore/timeouts-bound-waiting)
- [The fence](/lore/the-fence)
- [Durability](/systems/durability)
- [Scopes release resources when work ends](/lore/scopes-own-resources)
- [Retry needs idempotent effects](/lore/retry-needs-idempotency)
- [Checkpoints follow the promised commit](/lore/checkpoints-follow-commits)
- [Bulkheads isolate scarce capacity](/lore/bulkheads-isolate-capacity)
- [Dead letters need a return path](/lore/dead-letters-need-a-return-path)
- [Golden signals expose overload](/lore/golden-signals-expose-overload)
- [Cleanup can be an idempotent postcondition](/lore/cleanup-can-be-an-idempotent-postcondition)
- [Property-based testing](/lore/property-based-testing)