# Queues, Pipelines, Workflows and K2 do different jobs

> For agents: start with the [agent guide](https://ratstack.sh/llms.txt). Every page is Markdown by default; add `Accept: text/html` for HTML.

New on /news: [tla-ts 0.0.9 adds TLC's -continue](/news).

Each of these products moves work or data out of the request that started it.
They differ in the unit they track.
Choose by that unit, then check the delivery promise.

```text
request ──┬─▶ Queue ──────▶ consumer: one job, ack or retry
          ├─▶ Pipeline ───▶ SQL ──▶ R2 table or files
          ├─▶ Workflow ───▶ step ─▶ step ─▶ sleep ─▶ step
          └─▶ K2 stream ──▶ subscription A (reads every record)
                       └──▶ subscription B (reads every record)
```

## Queues track one job at a time

A queue holds messages for a consumer Worker. Cloudflare lists three uses: offload work from a request, send data from Worker to Worker, and buffer or batch writes. [Storage options](https://developers.cloudflare.com/workers/platform/storage-options/)

- **Delivery:** at least once by default. Give each message a unique id and use it as an [idempotency](/lore/http-responses-need-validation) key. [Delivery guarantees](https://developers.cloudflare.com/queues/reference/delivery-guarantees/)
- **Per message:** a consumer can `ack()` or `retry()` each message in a batch. [Batching and retries](https://developers.cloudflare.com/queues/configuration/batching-retries/)
- **After retries:** a message moves to a dead letter queue after `max_retries`, three by default. Without a dead letter queue, Cloudflare deletes it. [Dead letter queues](https://developers.cloudflare.com/queues/configuration/dead-letter-queues/)

Cloudflare's rule: "Use Queues when each message is a unit of work that needs to be completed, retried, and tracked individually." [Storage options](https://developers.cloudflare.com/workers/platform/storage-options/)

In [Alchemy](/skills/learn-alchemy), `Cloudflare.Queues.Queue` declares the queue and `Cloudflare.Queues.Consumer` attaches a Worker, with `deadLetterQueue` and `settings.maxRetries`. A Worker sends through `Cloudflare.Queues.WriteQueue`. [Queue.ts](https://github.com/alchemy-run/alchemy/blob/v2.0.0-beta.85/packages/cloudflare/src/Queues/Queue.ts) [Consumer.ts](https://github.com/alchemy-run/alchemy/blob/v2.0.0-beta.85/packages/cloudflare/src/Queues/Consumer.ts)

See [Dead letters need a return path](/lore/dead-letters-need-a-return-path) for what the dead letter queue still needs: an owner and a redrive.

## Pipelines load rows into R2

Basin Pipelines is Cloudflare's streaming ingestion product, and it has three parts. Alchemy names it `Cloudflare.Pipelines`. [Basin Pipelines](https://developers.cloudflare.com/basin-pipelines/)

1. A **stream** receives events over HTTP or a Worker [binding](/lore/bindings).
2. A **pipeline** runs SQL over the stream.
3. A **sink** writes Iceberg tables to Basin Catalog, or Parquet and JSON files to R2.

Sinks deliver exactly once. [Sinks](https://developers.cloudflare.com/basin-pipelines/sinks/) Cloudflare lists clickstream analytics, [telemetry](/lore/trace-meaningful-operations) and logs as uses. [Storage options](https://developers.cloudflare.com/workers/platform/storage-options/)

The output is a table or a set of files in R2 that you query later.

### How rat-stack uses it

Pipelines is the only one of the four that rat-stack declares. `@rat-stack/basin` builds the stream for both [request analytics](/systems/analytics) and join intake. The stream accepts no HTTP input, so only the Worker binding writes to it.

packages/basin/src/index.ts at a3f5a14861ce89913e82bea0f7ba02c83e8d4725; lines 14-25; highlighted 20-22. Title: A private stream beside its bucket.

```typescript
export const basinStream = (name: string) =>
  Effect.gen(function* declareBasinStream() {
    const bucket = yield* Cloudflare.R2.Bucket(`${name}Bucket`, {}).pipe(
      retainInProd
    );

    const stream = yield* Cloudflare.Pipelines.Stream(`${name}Stream`, {
      http: { enabled: false },
    }).pipe(retainInProd);

    return { bucket, stream } as const;
  });
```

`icebergSink` then copies every row unchanged into one table:

packages/basin/src/index.ts at a3f5a14861ce89913e82bea0f7ba02c83e8d4725; lines 71-73; highlighted none. Title: SELECT \* keeps the raw row.

```typescript
    return yield* Cloudflare.Pipelines.Pipeline(`${name}Pipeline`, {
      sql: Output.interpolate`INSERT INTO ${sink.name} SELECT * FROM ${streamName}`,
    }).pipe(retainInProd);
```

[Capture first](/lore/capture-first) explains why the copy stays raw.

## Workflows run steps that resume

A Workflow chains steps, retries failed steps, and persists state for minutes, hours or weeks. It can sleep, and it can wait for an external event such as an approval. [Workflows](https://developers.cloudflare.com/workflows/)

Each step retries on its own, so make each step idempotent. [Rules of Workflows](https://developers.cloudflare.com/workflows/build/rules-of-workflows/) A step that charges a card must first check whether the charge already happened.

In Alchemy, a Workflow body is an [Effect](/lore/effect-basics). `Cloudflare.Workflows.task` runs one named durable step. Its result is persisted and replayed on retries. `sleep`, `sleepUntil` and `waitForEvent` pause the run. The same source describes Workflows as at-least-once. [Workflow.ts](https://github.com/alchemy-run/alchemy/blob/v2.0.0-beta.85/packages/cloudflare/src/Workflows/Workflow.ts)

Choose a Workflow when the unit is a process with ordered steps and waits. Each instance has a unique ID and runs many steps. [Rules of Workflows](https://developers.cloudflare.com/workflows/build/rules-of-workflows/)

## K2 keeps a log that many readers replay

K2 is a durable log. Producers append records. Consumers read through subscriptions, and each subscription receives every record. [K2 concepts](https://developers.cloudflare.com/k2/concepts/)

- **Reading does not delete:** records stay until retention ends. Retention goes up to 30 days, and longer by request. [K2](https://developers.cloudflare.com/k2/)
- **Replay:** a new subscription can start at `earliest` and read records produced before it existed. [Delivery guarantees](https://developers.cloudflare.com/k2/reference/delivery-guarantees/)
- **Leases:** a consumer holds a batch for five minutes, then acks, nacks, or loses it to redelivery. [K2 concepts](https://developers.cloudflare.com/k2/concepts/)
- **Limits of the promise:** delivery is at least once per subscription. K2 acks whole batches, does not deduplicate, and does not order records across workers. [Delivery guarantees](https://developers.cloudflare.com/k2/reference/delivery-guarantees/)

Cloudflare says to use K2 when what matters is "that every record is processed rather than the state of any one record." [Storage options](https://developers.cloudflare.com/workers/platform/storage-options/)

K2 is in public beta on the Workers Paid plan. [K2](https://developers.cloudflare.com/k2/) Alchemy declares it as `Cloudflare.K2.Stream` and `Cloudflare.K2.Subscription`. A stream can take an Effect [Schema](/lore/schemas-define-the-boundary), so consumers receive decoded values. [K2/Stream.ts](https://github.com/alchemy-run/alchemy/blob/v2.0.0-beta.85/packages/cloudflare/src/K2/Stream.ts)

## Choose by the unit you track

- **One job that must finish:** Queues. Track and retry each message.
- **Rows you query later:** Pipelines. Load them into R2 as a table.
- **A process with steps and waits:** Workflows. Resume from the last finished step.
- **Events for several independent readers:** K2. Replay them from the log.

Three of the four deliver at least once. Make every consumer and step safe to repeat. [Retry needs idempotent effects](/lore/retry-needs-idempotency) shows how.

## Where rat-stack does without them

Join intake must deliver each contact after the response. It uses no queue. It saves the contact with a pending-delivery marker, then adds its id to a sweep index in a separate call. A [cron every five minutes](https://github.com/joelhooks/rat-stack/blob/a3f5a14861ce89913e82bea0f7ba02c83e8d4725/apps/site/src/interest/intake-sweep.ts#L13) redrives each pending id in that index.

That recovery has a documented gap. If the index write fails, the contact keeps its marker, but the sweep never finds it. [Join intake spec](https://github.com/joelhooks/rat-stack/blob/a3f5a14861ce89913e82bea0f7ba02c83e8d4725/packages/core/spec/README.md?plain=1#L105-L111) The backstop works by id: `rat-stack intake redeliver` re-drives a delivery, and `rat-stack intake erase` re-drives an erase.

[Durability](/systems/durability) shows the model tests that check it.

The marker plus its index entry is the recovery record. Cloudflare lists this job, offloading work from a request, as a Queues use. [Storage options](https://developers.cloudflare.com/workers/platform/storage-options/) [Request lifetime is not durability](/lore/request-lifetime-is-not-durability) explains why a record is needed at all.

Next, read [Durable work needs storage](/lore/durable-work-needs-storage), then [the durability index](/lore/durability).

## Sources

1. [Choose a data or storage product](https://developers.cloudflare.com/workers/platform/storage-options/)
   Cloudflare. Compares Queues, K2 and Basin Pipelines by use case. Says when to use Queues and when to use K2. Accessed 2026-11-08.

2. [Cloudflare Queues](https://developers.cloudflare.com/queues/)
   Cloudflare. Guaranteed delivery, offloading work from a request, Worker-to-Worker messages, batching. Accessed 2026-11-08.

3. [Queues delivery guarantees](https://developers.cloudflare.com/queues/reference/delivery-guarantees/)
   Cloudflare. At-least-once delivery by default; use a unique id as an idempotency key. Accessed 2026-11-08.

4. [Queues batching, retries and delays](https://developers.cloudflare.com/queues/configuration/batching-retries/)
   Cloudflare. Per-message ack() and retry(); batch-level ackAll() and retryAll(). Accessed 2026-11-08.

5. [Queues dead letter queues](https://developers.cloudflare.com/queues/configuration/dead-letter-queues/)
   Cloudflare. Messages move to a DLQ after max\_retries, three by default. Without a DLQ they are deleted. Accessed 2026-11-08.

6. [Basin Pipelines](https://developers.cloudflare.com/basin-pipelines/)
   Cloudflare. Formerly Cloudflare Pipelines. Ingests events, transforms them with SQL, delivers Iceberg tables or Parquet and JSON files to R2. Accessed 2026-11-08.

7. [Basin Pipelines sinks](https://developers.cloudflare.com/basin-pipelines/sinks/)
   Cloudflare. Sinks provide exactly-once delivery to Basin Catalog or R2. Accessed 2026-11-08.

8. [Cloudflare Workflows](https://developers.cloudflare.com/workflows/)
   Cloudflare. Durable multi-step execution, retries, sleeping, and waiting for external events. Accessed 2026-11-08.

9. [Rules of Workflows](https://developers.cloudflare.com/workflows/build/rules-of-workflows/)
   Cloudflare. Each step is individually retryable, so steps should be idempotent. Accessed 2026-11-08.

10. [K2 concepts](https://developers.cloudflare.com/k2/concepts/)
    Cloudflare. Streams, records, subscriptions with earliest or latest start, and five-minute leases. Accessed 2026-11-08.

11. [K2 delivery guarantees](https://developers.cloudflare.com/k2/reference/delivery-guarantees/)
    Cloudflare. At least once per subscription, batch-level acks, no deduplication, no ordering across workers. Accessed 2026-11-08.

12. [K2](https://developers.cloudflare.com/k2/)
    Cloudflare. Public beta on the Workers Paid plan; retention up to 30 days, longer by request. Accessed 2026-11-08.

13. [Alchemy Cloudflare provider at v2.0.0-beta.85](https://github.com/alchemy-run/alchemy/tree/v2.0.0-beta.85/packages/cloudflare/src)
    GitHub alchemy-run/alchemy. Declares Cloudflare.Queues, Cloudflare.Pipelines, Cloudflare.Workflows and Cloudflare.K2 resources and bindings. Accessed 2026-11-08.

14. [Shared Basin declarations](https://github.com/joelhooks/rat-stack/blob/a3f5a14861ce89913e82bea0f7ba02c83e8d4725/packages/basin/src/index.ts)
    GitHub joelhooks/rat-stack. basinStream and icebergSink declare the stream, sink and SELECT \* pipeline used by events and intake. Accessed 2026-11-08.

15. [Join intake sweep](https://github.com/joelhooks/rat-stack/blob/a3f5a14861ce89913e82bea0f7ba02c83e8d4725/apps/site/src/interest/intake-sweep.ts)
    GitHub joelhooks/rat-stack. A five-minute cron redrives the pending ids it reads from the sweep index. Accessed 2026-11-08.

16. [Join contact store](https://github.com/joelhooks/rat-stack/blob/a3f5a14861ce89913e82bea0f7ba02c83e8d4725/apps/site/src/interest/join-contact-store.ts)
    GitHub joelhooks/rat-stack. joinSave writes the contact, then markPending writes the sweep index in a separate call. Accessed 2026-11-08.

17. [Join intake spec: JoinIntakeIndexAfterRecord](https://github.com/joelhooks/rat-stack/blob/a3f5a14861ce89913e82bea0f7ba02c83e8d4725/packages/core/spec/README.md?plain=1#L105-L111)
    GitHub joelhooks/rat-stack. A failed index write leaves a marked contact the sweep never finds; redeliver and erase re-drive it by id. Accessed 2026-11-08.

## Lore on this page

- [Workers answer requests; Durable Objects own named state](/lore/workers-answer-requests-durable-objects-own-named-state)
- [Dead letters need a return path](/lore/dead-letters-need-a-return-path)
- [Bindings](/lore/bindings)

## Linked from

- Agent guide → An Effect stack so pure (aspirational) Kit Langton will blush. → [Read page](https://ratstack.sh/llms.txt)

- Bindings → Sam Goodwin describes bindings as the place where runtime permissions and configuration meet infrastructure. → [Read page](https://ratstack.sh/lore/bindings)

- Change log → What changed in the files served here, newest first. → [Read page](https://ratstack.sh/log)

- Durability starts with a recovery record → A reading order for storage, replay, worker ownership, overload, and recovery tests. → [Read page](https://ratstack.sh/lore/durability)

- Full agent guide → An Effect stack so pure (aspirational) Kit Langton will blush. → [Read page](https://ratstack.sh/llms-full.txt)

- Rat Stack lore | rat-stack → Short, source-grounded notes on the ideas and decisions behind rat-stack. → [Read page](https://ratstack.sh/lore)

- Source change history (log.md) → The generated change log as Markdown. → [Read page](https://ratstack.sh/log.md)

- Workers answer requests; Durable Objects own named state → A Worker must not rely on module state between requests, so state that must persist lives in a Durable Object addressed by name. → [Read page](https://ratstack.sh/lore/workers-answer-requests-durable-objects-own-named-state)