# Tokenize the whole file before slicing

> For agents: start with the [agent guide](https://ratstack.sh/llms.txt). Every page is Markdown by default; add `Accept: text/html` for HTML.

Free workshop: [how to burn a trillion tokens and get good results](/tokenmaxx#interested).

The [code-snippets design brief](/lore/code-snippets) requires whole-file tokenization.
The highlighter receives the full source before the pipeline selects lines.
Shiki exposes [`codeToTokens`](https://shiki.style/guide/install) for token output.

An excerpt can start inside a multiline comment or string.
Tokenizing only the excerpt removes the preceding source from the highlighter's input.

The proposed pipeline:

1. Tokenizes every source line.
2. Selects the requested ranges.
3. Keeps the original line numbers.
4. Inserts a collapsed gap between separated ranges.

A collapsed gap reports omitted lines.
It is not a source line.

The planned property compares excerpt tokens with the same lines from whole-file tokenization.
It includes multiline comments and template strings.
See [property-based testing](/lore/property-based-testing).

Both renderers consume [one snippet model](/lore/one-snippet-model-two-renderers).
See [the glossary](/glossary) for whole-file tokenization and collapsed gap.

## Sources

1. [Shiki installation and usage](https://shiki.style/guide/install)
   Primary source. codeToTokens API Accessed 2026-10-02.

2. [Code-snippets design brief](https://github.com/joelhooks/rat-stack/blob/main/.brain/resources/lore/code-snippets.svx)
   Primary source. Whole-file tokenization requirement Accessed 2026-10-02.

## Linked from

- Agent guide → An Effect stack so pure (aspirational) Kit Langton will blush. → [Read page](https://ratstack.sh/llms.txt)

- Change log → What changed in the files served here, newest first. → [Read page](https://ratstack.sh/log)

- Code snippets quote a pinned source → Design brief for a build-only code pipeline: pinned Git objects, typed diagnostics, and one snippet model for both views. → [Read page](https://ratstack.sh/lore/code-snippets)

- Full agent guide → An Effect stack so pure (aspirational) Kit Langton will blush. → [Read page](https://ratstack.sh/llms-full.txt)

- log.md → The generated change log as Markdown. → [Read page](https://ratstack.sh/log.md)

- One snippet model feeds both renderers → HTML and agent Markdown derive their code and provenance from the same snippet. → [Read page](https://ratstack.sh/lore/one-snippet-model-two-renderers)

- Rat Stack lore | rat-stack → Short, source-grounded notes on the ideas and decisions behind rat-stack. → [Read page](https://ratstack.sh/lore)