# "Systems that run forever self-heal and scale" by Joe Armstrong (2013)

> For agents: start with the [agent guide](https://ratstack.sh/llms.txt). Every page is Markdown by default; add `Accept: text/html` for HTML.

[Strange Loop Conference](https://www.youtube.com/@StrangeLoopConf) · Published 2021-03-26 · 1:10:22 · B tier · Systems thinking

[Watch on YouTube](https://www.youtube.com/watch?v=cNICGEwmXLU) · [All RatTube videos](/tube)

## Summary

Joe Armstrong discusses systems that must keep running in a highly concurrent, connected world. He examines architectures for [recovery](/lore/restart-from-known-state) rather than assuming the whole application can simply stop and restart.

The [Erlang](/lore/effect-is-not-beam) examples use processes, mailboxes, links, and [supervision trees](/lore/supervision-is-a-recovery-policy). Processes receive matching messages, and links report failures. Armstrong also stresses identifying why a failure happened, rather than only observing that something crashed.

## Section notes

- [1:42](https://www.youtube.com/watch?v=cNICGEwmXLU\&t=102s): The talk frames the problem around highly concurrent and connected systems.
- [12:16](https://www.youtube.com/watch?v=cNICGEwmXLU\&t=736s): Continuously available services cannot rely on stopping the whole system for maintenance.
- [25:19](https://www.youtube.com/watch?v=cNICGEwmXLU\&t=1519s): Consensus protocols illustrate the difficulty of explaining distributed algorithms.
- [38:10](https://www.youtube.com/watch?v=cNICGEwmXLU\&t=2290s): Supervision trees observe processes across machine boundaries, with fault identification alongside detection.
- [50:51](https://www.youtube.com/watch?v=cNICGEwmXLU\&t=3051s): Erlang mailboxes select matching messages and links deliver process-failure notifications.

## Key concepts

- [supervision trees](/lore/supervision-is-a-recovery-policy)
- [Erlang](/lore/effect-is-not-beam)
- [recovery](/lore/restart-from-known-state)

## Our take

Joe Armstrong on systems that run for a very long time: isolated processes with private memory that talk only by message passing, expecting crashes and fixing them as they run, and changing the code while it executes.

## Related videos

- ["Simple Made Easy" - Rich Hickey (2011)](/tube/simple-made-easy-rich-hickey-2011)
- [John Hughes - Testing the Hard Stuff and Staying Sane](/tube/testing-hard-stuff-john-hughes)
- [John Hughes - How to specify it! A guide to writing properties of pure functions | Code Mesh LDN 19](/tube/how-to-specify-pure-function-properties)
- [Leslie Lamport: Thinking Above the Code](/tube/leslie-lamport-thinking-above-the-code)

Summary based on the transcript. Written 2026-11-15.
