# Private catalog

> For agents: start with the [agent guide](https://ratstack.sh/llms.txt). Every page is Markdown by default; add `Accept: text/html` for HTML.

New on /news: [Foldkit 0.171.0 requires result Message schemas](/news).

## What it does

An Item aggregate owns the observations and decisions for one canonical HTTP or HTTPS URL.
It commits events and state together in one [Durable Object](/lore/workers-answer-requests-durable-objects-own-named-state).
Operators send `RecordSighting`, `Keep`, and `Drop` through one authenticated Api.
Recording a decision never publishes content.

`CatalogIndex` copies committed events into D1 through the `CatalogStore` [port](/lore/hexagonal-architecture).
The `Rising` and `RepeatAuthors` Queries read that derived index.
The public site receives no catalog data.

## The standard

- The Item event log is authoritative. D1 and `ItemHistory` are rebuildable copies.
- The first sighting from each source on each UTC day wins. A retry with another command id adds no observation.
- Item kinds are repo, post, video, and news. An observed item's kind cannot change.
- Keep and Drop require a sighting. An opposite decision can follow; repeating the current decision records nothing.
- Unknown metrics stay absent. A missing measurement never becomes zero.
- Author identity combines the source and lowercased handle. `RecordSighting` lowercases the handle before committing it.
- Every command, history read, and Query requires the operator token. Missing or empty configuration denies access.
- Domain's Api defines the [contract](/systems/capabilities). Its native projections expose the same schemas and authorization.

`CatalogIndex` keys each row by `stream:seq`.
A repeated identical delivery leaves the same row in place.
Different event content at the same position fails loudly.

Each D1 call commits its batch atomically.
The Domain checkpoint advances after every row is written or durably quarantined.
A retry between these commits adds no duplicate rows.

Permanent invalid input goes to `catalog_quarantine` with its source position and reason.
The checkpoint advances only after that row is durable.
Transient store failures leave the checkpoint unchanged and Domain redrives the pending log.
Inspect quarantined rows before treating index results as complete.

### Trend queries

Both Queries use observation time in an explicit UTC window: `from` is inclusive and `until` is exclusive.
The window must span more than zero and at most 30 days.
Pages default to 50 entries, with a maximum of 200 and an opaque continuation cursor.
A cursor belongs to its Query and filters.

`Rising` compares the first and last known `stars` or `views` measurement within each item and source.
It requires two measurements and a positive gain.
It returns the strongest source once per item, ordered by gain and then canonical URL.

Each row includes `from`, `to`, absolute `gain`, `relativeGain`, measurement times, source, and the latest indexed decision.
Relative gain divides gain by `from`; a zero baseline returns null.
The decision follows the latest indexed Keep/Drop event, even outside the observation window.
Keep and Drop do not filter these results.

`RepeatAuthors` counts N distinct canonical item URLs per source and lowercased author handle.
Repeated sightings of the same item do not increase that count.
The caller supplies `minimumItems` and up to 500 source-qualified `excludedAuthors`; handle matching is case-insensitive.
Results sort by item count and then the encoded source/handle key.

D1 uses indexed time-range reads.
The page limit bounds returned rows; aggregation still scans observations inside the selected window.
These initial bounds follow a local SQLite baseline.
Deployed D1 latency remains unproven.

Watch D1 rows read and query duration as the observation window grows denser.
A rollup comes later.

### Known limits

For `RepeatAuthors`, automatic exclusion of admitted people waits for the Person slice.
The caller supplies exclusions until that source exists.
There is no Person aggregate, `Feature` command, or JSONL importer yet.

The D1 index can lag committed events.
Pass command receipts as `atLeast` to wait for their `CatalogIndex` checkpoints.
Receipts cover only their streams.
Results can change between pages as new observations arrive.

Decision sequence ordering assumes one stream per canonical item.
A stream reset or multiple streams for one item require an epoch or reconciliation.
Equal observation timestamps use numeric sequence order within that stream.

One corrupt result row fails the whole Query with `CatalogUnavailable`.
Repair or rebuild the affected index data before retrying.

History state holds the latest observation and decision; `ItemHistory.list` pages through earlier events.
Operators pass an explicit `take` on each history list call.
Domain beta.87 supplies no default or maximum for `View.list`; omission reads the item's full history.
The Query page limits above do not change that native history behavior.

## How to check

Run `pnpm --filter @rat-stack/catalog test` for stories, declarations, and generated command histories.
`Domain.verify(Catalog)` must return no problems.

Run `pnpm --filter @rat-stack/catalog-server test` for both store implementations, replay, windows, authorization, and native HTTP smoke behavior.
SQLite exercises the D1 [adapter](/lore/hexagonal-architecture)'s SQL and migrations locally; it does not prove deployed D1 behavior.
The suite also boots the actual bundled [Worker](/lore/workers-answer-requests-durable-objects-own-named-state) with local D1 through `@alchemy.run/test`.
Run `pnpm --filter @rat-stack/site startup:check` to measure its entry and static startup graph.

Review `pnpm infra:plan` before deployment.
The owner supplies `CATALOG_OPERATOR_TOKEN` through deploy configuration; the [Worker](/lore/workers-answer-requests-durable-objects-own-named-state) binds a supplied value as a redacted secret.
No secret value ships in source.
Provisioning or rotating this configuration requires a deploy.

Run `pnpm --filter @rat-stack/catalog-server smoke <url>` with the token supplied securely through the environment.
The smoke uses typed `ItemHistory.get` calls: no token must return Unauthorized; the configured token must succeed or report a missing item.
It prints fixed results and sanitized failures, never the token.
HTTPS is required except for HTTP on exact `localhost` or `127.0.0.1`.

A raw GET is neither an authentication check nor a health probe: an empty native [RPC](/skills/learn-rat-stack) request returns an empty NDJSON response.

Run `pnpm env:check`, then `TURBO_RUN_SUMMARY=true pnpm turbo run check test build --summarize` for the full [fence](/lore/the-fence).

## Sources

1. [Domain on Cloudflare](https://alchemy.run/domain/cloudflare)
   Alchemy. Durable Object hosting and atomic aggregate commits. Accessed 2026-11-15.

2. [Domain queries](https://alchemy.run/domain/query)
   Alchemy. Projection-backed reads, pagination, authorization, and receipt consistency. Accessed 2026-11-15.

3. [D1 limits](https://developers.cloudflare.com/d1/platform/limits/)
   Cloudflare. Invocation query calls, batch accounting, and row-size limits. Accessed 2026-11-15.

4. [Catalog model](https://github.com/joelhooks/rat-stack/blob/main/packages/catalog/src/Catalog.model.ts)
   GitHub joelhooks/rat-stack. Item history, index projection, and query declarations. Accessed 2026-11-15.

## Lore on this page

- [Checkpoints follow the promised commit](/lore/checkpoints-follow-commits)

## Linked from

- add-a-store → Add persistence through a service named for an application job, with validation, vendor Layers, migrations, bounded reads, and shared backend tests. → [Read page](https://ratstack.sh/skills/add-a-store)

- Agent guide → An Effect stack so pure (aspirational) Kit Langton will blush. → [Read page](https://ratstack.sh/llms.txt)

- Agent rules for rat-stack (AGENTS.md) → What you may change, which commands to run, and which changes need approval. → [Read page](https://ratstack.sh/AGENTS.md)

- Alchemy Domain models a business as commands, aggregates, and events → Alchemy Domain is a domain-driven design framework for Effect whose one model runs in memory for tests and on Cloudflare Durable Objects in production. → [Read page](https://ratstack.sh/lore/alchemy-domain)

- Analytics → One event per HTTP request, stored unchanged for queries to select and interpret later. → [Read page](https://ratstack.sh/systems/analytics)

- Build and run rat-stack (README.md) → What is in the repo, how the example works, and how to run it. → [Read page](https://ratstack.sh/README.md)

- Change log → What changed in the files served here, newest first. → [Read page](https://ratstack.sh/log)

- Durability starts with a recovery record → A reading order for storage, replay, worker ownership, overload, and recovery tests. → [Read page](https://ratstack.sh/lore/durability)

- Full agent guide → An Effect stack so pure (aspirational) Kit Langton will blush. → [Read page](https://ratstack.sh/llms-full.txt)

- keep-or-cut → Learn which pieces depend on each other, then keep only the ones your project needs. → [Read page](https://ratstack.sh/skills/keep-or-cut)

- Rat Stack systems | rat-stack → The systems rat-stack ships, the standard each keeps, and how to check it. → [Read page](https://ratstack.sh/systems)

- Source change history (log.md) → The generated change log as Markdown. → [Read page](https://ratstack.sh/log.md)
