# Durability bibliography

> For agents: start with the [agent guide](https://ratstack.sh/llms.txt). Every page is Markdown by default; add `Accept: text/html` for HTML.

Free workshop: [Build Agent Loops That Never Stop Shipping](/tokenmaxx#interested).

Book principles and our [Effect](/lore/effect-basics) mappings are separate. Each entry names the book and the chapters or sections we read. Unknown edition details stay unknown. Quote numbers match the links on the durability pages.

## Books

### B1 Erlang and OTP in Action

Logan, Martin; Merritt, Eric; Carlsson, Richard. *Erlang and OTP in Action*.

**Read:**

- The Erlang/OTP Platform: Let It Crash
- process isolation
- §4.2.2 supervisor restart strategy

**Edition:** Edition, publisher, and year are not confirmed.

[Public book reference](https://www.manning.com/books/erlang-and-otp-in-action).

#### Q01

> The point of letting an entire subsystem terminate and be restarted is that it brings you back to a state known to function properly.

### B2 Elixir in Action

Jurić, Saša. *Elixir in Action*, Third Edition. Manning, 2024.

**Read:**

- §5.4.3 Shared-Nothing Concurrency
- Chapters 8–9, fault tolerance and supervision
- Other Use Cases for ETS

[Public book reference](https://www.manning.com/books/elixir-in-action-third-edition).

#### Q02

> Remember from chapters 8 and 9 that processes lose their state on termination.

#### Q03

> Shared-nothing concurrency ensures complete isolation between processes: one process can’t compromise the internal state of another.

### B3 Designing Elixir Systems with OTP

Gray II, James Edward; Tate, Bruce A. *Designing Elixir Systems with OTP*. Pragmatic Bookshelf, 2019.

**Read:**

- Build a Functional Core
- Isolate Process Machinery in a Boundary
- Customize Your Lifecycle
- Big, Loud Worker-Bees
- Add Persistence as a Boundary [Service](/lore/services-capture-dependencies)
- Test the Boundary

**Edition:** Not stated.

[Public book reference](https://pragprog.com/titles/jgotp/designing-elixir-systems-with-otp/).

#### Q04

> Elixir can’t manage what it doesn’t know about.

#### Q14

> Our core doesn’t access external interfaces or use any process machinery your component might use.

### B4 Release It

Nygard, Michael T. *Release It!: Design and Deploy Production-Ready Software*. Pragmatic Bookshelf, 2007.

**Read:**

- §4.4 Unbounded Result Sets
- Chapter 5 Stability Patterns
- §5.1 Timeouts
- §5.2 [Circuit Breaker](/lore/circuit-breakers-have-state)
- §5.3 Bulkheads
- §5.4 Steady State
- §5.5 Fail Fast
- §5.6 Handshaking
- §5.7 Test Harness
- §5.8 Decoupling Middleware

**Edition:** Not stated. Do not substitute second-edition chapter numbering.

[Public book reference](https://archive.org/details/releaseitdesignd0000nyga).

#### Q05

> The timeout is a simple mechanism allowing you to stop waiting for an answer once you think it will not come.

#### Q06

> calls to the circuit breaker fail immediately, without any attempt to execute the real operation.

#### Q07

> Protect critical clients by giving them their own pool to call.

#### Q08

> The Steady State pattern says, for every mechanism that accumulates a resource, some other mechanism must recycle that resource.

#### Q09

> The Test Harness can produce slow responses, no responses, or garbage responses.

### B5 Building Event-Driven Microservices

Bellemare, Adam. *Building Event-Driven Microservices: Leveraging Organizational Data at Scale*. O’Reilly, 2020.

**Read:**

- Chapter 3 Communication and Data Contracts
- Chapter 6 Deterministic Stream Processing
- Chapter 7 Stateful Streaming
- Chapter 9 Microservices Using Function-as-a-Service
- Chapter 14 Supportive Tooling

**Edition:** Not stated. Quotes come from the 2020 text.

[Public book reference](https://www.oreilly.com/library/view/building-event-driven-microservices/9781492057888/).

#### Q11

> The overarching goal of deterministic processing is that a microservice should produce the same output whether it is processing in real time or catching up to the present time.

#### Q12

> Schema evolution ensures that the consumer can translate it to a familiar version.

#### Q13

> allowing a single stateful transaction to atomically update both the state and input offsets.

### B6 Fundamentals of Data Engineering

Reis, Joe; Housley, Matt. *Fundamentals of Data Engineering*. O’Reilly.

**Read:**

- Part II, Chapter 5: Message Queues
- DataOps considerations
- Incident Response

**Edition:** Edition and year are not confirmed.

[Public book reference](https://www.oreilly.com/library/view/fundamentals-of-data/9781098108298).

#### Q10

> In an idempotent system, the outcome of processing a message once is identical to the outcome of processing it multiple times.

### B7 Domain Modeling Made Functional

Wlaschin, Scott. *Domain Modeling Made Functional: Tackle Software Complexity with Domain-Driven Design and F#*. Pragmatic Bookshelf, 2018.

**Read:**

- Implementation: Working with Errors
- Using the Result Type to Make Errors Explicit
- Handling Exceptions
- Composing in Parallel with Applicatives

**Edition:** Printing P1.0. The numbered edition is not stated.

[Public book reference](https://pragprog.com/titles/swdddf/domain-modeling-made-functional/).

#### Q15

> These different kinds of errors require different implementations.

### B8 Monitoring Distributed Systems

Ewaschuk, Rob. *Monitoring Distributed Systems*. O’Reilly, 2016.

**Read:**

- The Four Golden Signals
- Black-Box Versus White-Box
- Symptoms Versus Causes

**Edition:** First edition. It is also a chapter of the Google SRE book.

[Public book reference](https://sre.google/sre-book/monitoring-distributed-systems/).

#### Q18

> The four golden signals of monitoring are latency, traffic, errors, and saturation.

### B9 Serverless Architectures on AWS

Sbarski, Peter; Cui, Yan; Nair, Ajay. *Serverless Architectures on AWS*, Second Edition.

**Read:**

- Chapter 6, The New Search Microservice: Everything Fails, All the Time
- Pay Attention to Retry Configurations

AWS-specific mechanisms are not universal runtime guarantees.

**Edition:** Publisher and year are not confirmed.

[Public book reference](https://www.manning.com/books/serverless-architectures-on-aws-second-edition).

#### Q20

> Do you put the failed event into a DLQ and process it later?

### B10 A Philosophy of Software Design

Ousterhout, John. *A Philosophy of Software Design*. 2018.

**Read:**

- §4.4 Deep Modules
- §10.2 Too Many Exceptions
- §10.3 Define Errors Out of Existence
- §10.6 Mask Exceptions
- §10.7 Exception Aggregation
- §10.8 Just Crash?
- §10.10 Taking It Too Far

**Edition:** Edition and publisher are not confirmed.

[Public book reference](https://web.stanford.edu/~ouster/cgi-bin/book.php).

#### Q16

> Rather than throwing an error when unset is asked to delete an unknown variable, it should have simply returned without doing anything.

### B11 Property-Based Testing

Hebert, Fred. *Property-Based Testing with PropEr, Erlang, and Elixir*. Pragmatic Bookshelf, 2019.

**Read:**

- Chapter 7 Shrinking
- Laying Out Stateful Properties
- Testing Parallel Executions
- Initial State for the State Machine
- Adjusting the Model

Use the actual breaker’s threshold semantics.

**Edition:** Not stated.

[Public book reference](https://pragprog.com/titles/fhproper/property-based-testing-with-proper-erlang-and-elixir/).

#### Q17

> Both function calls must be deterministic.

### B12 Site Reliability Engineering

Beyer, Betsy; Jones, Chris; Petoff, Jennifer; Murphy, Niall Richard, editors. *Site Reliability Engineering: How Google Runs Production Systems*. O’Reilly, 2016.

**Read:**

- Chapter 3 Embracing Risk
- Chapter 22 Addressing Cascading Failures (Mike Ulrich)
- Chapter 26 General Principles of SRE as Applied to Data Integrity
- Chapter 27 Reliable Product Launches at Scale

Section-only excerpts limit detailed attribution.

**Edition:** Not stated.

[Public book reference](https://sre.google/sre-book/table-of-contents/).

### B13 Grokking Simplicity

Normand, Eric. *Grokking Simplicity: Taming Complex Software with Functional Thinking*. Manning, 2021.

**Read:**

- Chapters 15–17: Isolating Timelines, Sharing Resources Between Timelines, Coordinating Timelines

**Edition:** Not stated.

[Public book reference](https://www.manning.com/books/grokking-simplicity).

### B14 Refactoring Databases

Ambler, Scott W.; Sadalage, Pramod J. *Refactoring Databases: Evolutionary Database Design*. Addison-Wesley, 2006.

**Read:**

- §3.11 What You Have Learned (transition period)
- Chapter 4 Deploying into Production

**Edition:** Not stated.

[Public book reference](https://www.amazon.com/Refactoring-Databases-Evolutionary-paperback-Addison-Wesley/dp/0321774515).

#### Q19

> many refactorings require you to run both the original and new versions of the schema in parallel during a transition period that is long enough to allow any applications accessing that portion of the schema time to be redeployed.

### B15 Software Engineering at Google

Winters, Titus; Manshreck, Tom; Wright, Hyrum, editors. *Software Engineering at Google: Lessons Learned from Programming Over Time*. O’Reilly, 2020.

**Read:**

- Chapter 15 Deprecation: Deprecation Tooling and Preventing Backsliding
- CI Is Alerting (hermetic and flaky tests)

**Edition:** Not stated.

[Public book reference](https://abseil.io/resources/swe-book).

### B16 Why Programs Fail

Zeller, Andreas. *Why Programs Fail: A Guide to Systematic Debugging*. Morgan Kaufmann, 2009.

**Read:**

- Chapter 4 Reproducing the Problem Environment
- Chapter 10 Reference Runs

Short section excerpts do not establish a complete debugging method.

**Edition:** Not stated.

## Coverage limits

Some books gave no usable passage for this topic:

- *Designing Data-Intensive Applications* and Newman’s *Building Microservices* gave only fragments for their [scope](/lore/scopes-own-resources).
- *Adopting Elixir* and *Programming Elixir* gave no usable findings.
- The Scala text gave no usable passage.

These pages make no book-backed claims about fencing, consensus, compensation, or replication. They add no hot-upgrade or Governor rule.

## Effect source

Effect contributors. *Effect*, `effect@4.0.0`, commit `67ba4e46a11ccda0b6761578bfd22c04ae00167d`. See [the durability skill](/skills/durability) and its [reference copy of this bibliography](https://github.com/joelhooks/rat-stack/blob/main/skills/durability/references/bibliography.md). All mappings and proposed checks are our own reading of the books.

Return to [the durability index](/lore/durability).

## Sources

1. [Logan, Martin; Merritt, Eric; Carlsson, Richard. Erlang and OTP in Action.](https://www.manning.com/books/erlang-and-otp-in-action)
   Manning. The Erlang/OTP Platform: Let It Crash; process isolation; §4.2.2 supervisor restart strategy. Accessed 2026-10-23.

2. [Jurić, Saša. Elixir in Action, Third Edition. Manning, 2024.](https://www.manning.com/books/elixir-in-action-third-edition)
   Manning. §5.4.3 Shared-Nothing Concurrency; Chapters 8–9, fault tolerance and supervision; Other Use Cases for ETS. Accessed 2026-10-23.

3. [Gray II, James Edward; Tate, Bruce A. Designing Elixir Systems with OTP. Pragmatic Bookshelf, 2019.](https://pragprog.com/titles/jgotp/designing-elixir-systems-with-otp/)
   Pragmatic Bookshelf. Build a Functional Core; Isolate Process Machinery in a Boundary; Customize Your Lifecycle; Big, Loud Worker-Bees; Add Persistence as a Boundary Service; Test the Boundary. Accessed 2026-10-23.

4. [Nygard, Michael T. Release It!: Design and Deploy Production-Ready Software. Pragmatic Bookshelf, 2007.](https://archive.org/details/releaseitdesignd0000nyga)
   Pragmatic Bookshelf. §4.4 Unbounded Result Sets; Chapter 5 Stability Patterns; §5.1 Timeouts; §5.2 Circuit Breaker; §5.3 Bulkheads; §5.4 Steady State; §5.5 Fail Fast; §5.6 Handshaking; §5.7 Test Harness; §5.8 Decoupling Middleware. Accessed 2026-10-23.

5. [Bellemare, Adam. Building Event-Driven Microservices: Leveraging Organizational Data at Scale. O’Reilly, 2020.](https://www.oreilly.com/library/view/building-event-driven-microservices/9781492057888/)
   O’Reilly. Chapter 3 Communication and Data Contracts; Chapter 6 Deterministic Stream Processing; Chapter 7 Stateful Streaming; Chapter 9 Microservices Using Function-as-a-Service; Chapter 14 Supportive Tooling. Accessed 2026-10-23.

6. [Reis, Joe; Housley, Matt. Fundamentals of Data Engineering. O’Reilly.](https://www.oreilly.com/library/view/fundamentals-of-data/9781098108298)
   O’Reilly. Part II, Chapter 5: Message Queues; DataOps considerations; Incident Response. Accessed 2026-10-23.

7. [Wlaschin, Scott. Domain Modeling Made Functional: Tackle Software Complexity with Domain-Driven Design and F#. Pragmatic Bookshelf, 2018.](https://pragprog.com/titles/swdddf/domain-modeling-made-functional/)
   Pragmatic Bookshelf. Implementation: Working with Errors; Using the Result Type to Make Errors Explicit; Handling Exceptions; Composing in Parallel with Applicatives. Accessed 2026-10-23.

8. [Ewaschuk, Rob. Monitoring Distributed Systems. O’Reilly, 2016.](https://sre.google/sre-book/monitoring-distributed-systems/)
   O’Reilly. The Four Golden Signals; Black-Box Versus White-Box; Symptoms Versus Causes. Accessed 2026-10-23.

9. [Sbarski, Peter; Cui, Yan; Nair, Ajay. Serverless Architectures on AWS, Second Edition.](https://www.manning.com/books/serverless-architectures-on-aws-second-edition)
   Manning. Chapter 6, The New Search Microservice: Everything Fails, All the Time; Pay Attention to Retry Configurations. AWS-specific mechanisms are not universal runtime guarantees. Accessed 2026-10-23.

10. [Ousterhout, John. A Philosophy of Software Design. 2018.](https://web.stanford.edu/~ouster/cgi-bin/book.php)
    John Ousterhout. §4.4 Deep Modules; §10.2 Too Many Exceptions; §10.3 Define Errors Out of Existence; §10.6 Mask Exceptions; §10.7 Exception Aggregation; §10.8 Just Crash?; §10.10 Taking It Too Far. Accessed 2026-10-23.

11. [Hebert, Fred. Property-Based Testing with PropEr, Erlang, and Elixir. Pragmatic Bookshelf, 2019.](https://pragprog.com/titles/fhproper/property-based-testing-with-proper-erlang-and-elixir/)
    Pragmatic Bookshelf. Chapter 7 Shrinking; Laying Out Stateful Properties; Testing Parallel Executions; Initial State for the State Machine; Adjusting the Model. Use the actual breaker’s threshold semantics. Accessed 2026-10-23.

12. [Beyer, Betsy; Jones, Chris; Petoff, Jennifer; Murphy, Niall Richard, editors. Site Reliability Engineering: How Google Runs Production Systems. O’Reilly, 2016.](https://sre.google/sre-book/table-of-contents/)
    O’Reilly. Chapter 3 Embracing Risk; Chapter 22 Addressing Cascading Failures (Mike Ulrich); Chapter 26 General Principles of SRE as Applied to Data Integrity; Chapter 27 Reliable Product Launches at Scale. Section-only excerpts limit detailed attribution. Accessed 2026-10-23.

13. [Normand, Eric. Grokking Simplicity: Taming Complex Software with Functional Thinking. Manning, 2021.](https://www.manning.com/books/grokking-simplicity)
    Manning. Chapters 15–17: Isolating Timelines, Sharing Resources Between Timelines, Coordinating Timelines. Accessed 2026-10-23.

14. [Ambler, Scott W.; Sadalage, Pramod J. Refactoring Databases: Evolutionary Database Design. Addison-Wesley, 2006.](https://www.amazon.com/Refactoring-Databases-Evolutionary-paperback-Addison-Wesley/dp/0321774515)
    Addison-Wesley. §3.11 What You Have Learned (transition period); Chapter 4 Deploying into Production. Accessed 2026-10-23.

15. [Winters, Titus; Manshreck, Tom; Wright, Hyrum, editors. Software Engineering at Google: Lessons Learned from Programming Over Time. O’Reilly, 2020.](https://abseil.io/resources/swe-book)
    O’Reilly. Chapter 15 Deprecation: Deprecation Tooling and Preventing Backsliding; CI Is Alerting (hermetic and flaky tests). Accessed 2026-10-23.

16. [Effect contributors. Effect 4.0.0. workflow/DurableQueue.ts](https://github.com/Effect-TS/effect/blob/67ba4e46a11ccda0b6761578bfd22c04ae00167d/packages/effect/src/workflow/DurableQueue.ts)
    Effect-TS. Pinned source for our Effect mapping, not a claim from the book. Accessed 2026-10-23.

## Lore on this page

- [Durable work needs storage](/lore/durable-work-needs-storage)
- [Effect is not BEAM](/lore/effect-is-not-beam)
- [Supervision chooses what restarts](/lore/supervision-is-a-recovery-policy)
- [Pure cores reduce timeline coupling](/lore/pure-cores-reduce-timelines)
- [Steady state recycles resources](/lore/steady-state-recycles-resources)
- [Admission control protects recovery](/lore/admission-controls-new-work)
- [Timeouts bound waiting](/lore/timeouts-bound-waiting)
- [Circuit breakers have state](/lore/circuit-breakers-have-state)
- [Old events must still decode](/lore/old-events-must-still-decode)
- [Golden signals expose overload](/lore/golden-signals-expose-overload)
- [Cleanup can be an idempotent postcondition](/lore/cleanup-can-be-an-idempotent-postcondition)
- [Property-based testing](/lore/property-based-testing)

## Linked from

- Admission control protects recovery → Refuse or defer work that cannot be served usefully under current load. → [Read page](https://ratstack.sh/lore/admission-controls-new-work)

- Agent guide → An Effect stack so pure (aspirational) Kit Langton will blush. → [Read page](https://ratstack.sh/llms.txt)

- Backpressure reaches the producer → Bounded consumption helps only when upstream respects waiting or refusal. → [Read page](https://ratstack.sh/lore/backpressure-reaches-the-producer)

- Bulkheads isolate scarce capacity → Reserve independent capacity for workloads that must not starve each other. → [Read page](https://ratstack.sh/lore/bulkheads-isolate-capacity)

- Change log → What changed in the files served here, newest first. → [Read page](https://ratstack.sh/log)

- Checkpoints follow the promised commit → Persist state and progress together when one store can transact both. → [Read page](https://ratstack.sh/lore/checkpoints-follow-commits)

- Circuit breakers have state → A breaker stops doomed calls and controls probes before restoring traffic. → [Read page](https://ratstack.sh/lore/circuit-breakers-have-state)

- Cleanup can be an idempotent postcondition → Already-absent state can be success when absence is the operation’s intended result. → [Read page](https://ratstack.sh/lore/cleanup-can-be-an-idempotent-postcondition)

- Dead letters need a return path → Exhausted work needs durable evidence, an owner, and deliberate redrive. → [Read page](https://ratstack.sh/lore/dead-letters-need-a-return-path)

- durability → Keep a rat-stack app standing when a provider, pool, or dependency slows down or a worker dies. Use at any I/O boundary, connection pool, provider call, shared waiter, fan-out, flag, gate, or work that can outlive a request, or when asked "what happens when this is slow", "add a timeout", "make this durable", or "what survives a crash". → [Read page](https://ratstack.sh/skills/durability)

- Durability starts with a recovery record → A reading order for storage, replay, worker ownership, overload, and recovery tests. → [Read page](https://ratstack.sh/lore/durability)

- Durable timers need recovery → Persist scheduled intent when losing a worker must not lose its deadlines. → [Read page](https://ratstack.sh/lore/durable-timers-need-recovery)

- Durable work needs storage → A restarted worker cannot recover intent that existed only in memory. → [Read page](https://ratstack.sh/lore/durable-work-needs-storage)

- Effect is not BEAM → Fibers share a runtime; scope ownership does not supply isolated heaps or OTP restart strategies. → [Read page](https://ratstack.sh/lore/effect-is-not-beam)

- Every task needs an owner → Give child work a lifetime, a failure observer, and a recovery decision. → [Read page](https://ratstack.sh/lore/every-task-needs-an-owner)

- Failures keep their meaning → Expected refusals, infrastructure faults, and defects need different policies. → [Read page](https://ratstack.sh/lore/failures-keep-their-meaning)

- Full agent guide → An Effect stack so pure (aspirational) Kit Langton will blush. → [Read page](https://ratstack.sh/llms-full.txt)

- Golden signals expose overload → Measure latency, traffic, errors, and saturation, including failed work. → [Read page](https://ratstack.sh/lore/golden-signals-expose-overload)

- Old events must still decode → Retained data needs compatibility rules before recovery depends on it. → [Read page](https://ratstack.sh/lore/old-events-must-still-decode)

- Pure cores reduce timeline coupling → Move deterministic rules out of time-dependent actions and shared mutable state. → [Read page](https://ratstack.sh/lore/pure-cores-reduce-timelines)

- Queries need result bounds → Limit result cardinality before the driver materializes an oversized response. → [Read page](https://ratstack.sh/lore/queries-need-result-bounds)

- Rat Stack lore | rat-stack → Short, source-grounded notes on the ideas and decisions behind rat-stack. → [Read page](https://ratstack.sh/lore)

- Recovery needs model tests → Test sequences of failures and recoveries against explicit invariants. → [Read page](https://ratstack.sh/lore/recovery-needs-model-tests)

- Replay needs stable inputs → Historical processing must not silently depend on today’s clock or external decisions. → [Read page](https://ratstack.sh/lore/replay-needs-stable-inputs)

- Request lifetime is not durability → Worker waitUntil has a shared post-response allowance; required work needs a durable marker and redrive. → [Read page](https://ratstack.sh/lore/request-lifetime-is-not-durability)

- Restart must reconstruct known state → Execution recovery and state recovery need separate contracts. → [Read page](https://ratstack.sh/lore/restart-from-known-state)

- Retry load needs a budget → One retry owner must control attempts, delay, and aggregate load. → [Read page](https://ratstack.sh/lore/retry-load-needs-a-budget)

- Retry needs idempotent effects → Repeating an operation must preserve its intended observable result. → [Read page](https://ratstack.sh/lore/retry-needs-idempotency)

- Shared schemas need a transition period → Keep old and new representations usable while consumers migrate. → [Read page](https://ratstack.sh/lore/schema-changes-need-a-transition)

- Source change history (log.md) → The generated change log as Markdown. → [Read page](https://ratstack.sh/log.md)

- Steady state recycles resources → Every accumulation needs a deliberate recycling policy. → [Read page](https://ratstack.sh/lore/steady-state-recycles-resources)

- Supervision chooses what restarts → Restart policy follows dependency shape and bounds repeated recovery. → [Read page](https://ratstack.sh/lore/supervision-is-a-recovery-policy)

- Timeouts bound waiting → A local timeout stops waiting; it does not prove the remote action never happened. → [Read page](https://ratstack.sh/lore/timeouts-bound-waiting)