# Tokenize the whole file before slicing

> For agents: start with the [agent guide](https://ratstack.sh/llms.txt). Every page is Markdown by default; add `Accept: text/html` for HTML.

New on /news: [Foldkit 0.171.0 requires result Message schemas](/news).

Tokenization splits source text into parts with syntax roles, such as strings and comments.
A highlighter uses those roles to style code.

The [code-snippets design brief](/lore/code-snippets) requires whole-file tokenization.
The highlighter receives the full source before the pipeline selects lines.
Shiki exposes [`codeToTokens`](https://shiki.style/guide/install) for token output.

An excerpt can start inside a multiline comment or string.
Tokenizing only the excerpt removes the preceding source from the highlighter's input.

The proposed pipeline:

1. Tokenizes every source line.
2. Selects the requested ranges.
3. Keeps the original line numbers.
4. Inserts a collapsed gap between separated ranges.

A collapsed gap labels omitted lines separately from source lines.

The planned property compares excerpt tokens with the same lines from whole-file tokenization.
It includes multiline comments and template strings.
See [property-based testing](/lore/property-based-testing).

Both renderers consume [one snippet model](/lore/one-snippet-model-two-renderers).
See [the glossary](/glossary) for whole-file tokenization and collapsed gap.

## Sources

1. [Shiki installation and usage](https://shiki.style/guide/install)
   Primary source. codeToTokens API Accessed 2026-10-02.

2. [Code-snippets design brief](https://github.com/joelhooks/rat-stack/blob/main/.brain/resources/lore/code-snippets.svx)
   Primary source. Whole-file tokenization requirement Accessed 2026-10-02.

## Linked from

- Agent guide → An Effect stack so pure (aspirational) Kit Langton will blush. → [Read page](https://ratstack.sh/llms.txt)

- Change log → What changed in the files served here, newest first. → [Read page](https://ratstack.sh/log)

- Code snippets quote a pinned source → Shipped design brief: pinned Git objects, typed diagnostics, and one build-only snippet model for both views. → [Read page](https://ratstack.sh/lore/code-snippets)

- Full agent guide → An Effect stack so pure (aspirational) Kit Langton will blush. → [Read page](https://ratstack.sh/llms-full.txt)

- One snippet model feeds both renderers → HTML and agent Markdown derive their code, repository, commit, path, and line numbers from the same snippet. → [Read page](https://ratstack.sh/lore/one-snippet-model-two-renderers)

- Rat Stack lore | rat-stack → Short, source-grounded notes on the ideas and decisions behind rat-stack. → [Read page](https://ratstack.sh/lore)

- Source change history (log.md) → The generated change log as Markdown. → [Read page](https://ratstack.sh/log.md)
