---
author: architect
responsible: architect
phase: 39
phase_type: implementation
status: Ready for execution
hard_bar: true
security_relevant: true
design_fork: false
wave: 1
depends_on: [38]
files_modified:
  - web/src/lib/tiptap/preserved.ts
  - web/src/lib/tiptap/extensions.ts
  - web/src/lib/tiptap/codec.corpus.ts
  - web/src/lib/tiptap/codec.test.ts
requirements: [RE-02]
autonomous: true
---

# Phase 39: Verbatim Preservation Tier - Plan

**Goal:** Add a Tier-2 verbatim-preservation mechanism (`PreservedBlock`/`PreservedInline` atom nodes +
math tokenizers) so tables, `$…$`/`$$…$$` math, and raw HTML round-trip **byte-for-byte (modulo outer
trim)** through the Phase-38 codec — with raw HTML kept **inert** (MAJ-04, no DOM path).

**Requirements:** RE-02 · **SECURITY: MAJ-04**

> **Hard bar.** Byte-for-byte means byte-for-byte. If a construct can't preserve exact bytes after
> honest effort (inline HTML is the research-flagged fragile one), fix it via the custom-tokenizer
> route OR document the exact limitation honestly — never loosen the byte assertion to "close enough".
> Raw HTML preservation MUST stay inert literal text (never `generateJSON`/DOM).

## must_haves

```yaml
truths:
  - id: RE-02-1
    statement: "PreservedBlock (block atom, code:true, raw attr) + PreservedInline (inline atom, code:true, raw attr) capture token.raw via parseMarkdown and re-emit it verbatim via renderMarkdown; registered for marked's table + block html tokens (markdownTokenName), and math via custom markdownTokenizer"
    verification: "grep -nE 'PreservedBlock|PreservedInline|markdownTokenName|renderMarkdown|parseMarkdown|markdownTokenizer' web/src/lib/tiptap/preserved.ts"
  - id: RE-02-2
    statement: "Tier-2 corpus: table, $$ block math, $ inline math, block HTML (and inline HTML or a documented limitation) round-trip byte-for-byte (modulo outer trim) + idempotent; mixed Tier-1+Tier-2 documents round-trip"
    verification: "cd web && npx vitest run --environment node src/lib/tiptap/codec.test.ts"
  - id: RE-02-3
    statement: "SECURITY (MAJ-04): raw HTML round-trips to its EXACT source bytes as inert literal text — no generateJSON/DOM; an <img onerror>/<script> fixture yields no element node and no executable attribute in the parsed JSON, under the node env"
    verification: "cd web && npx vitest run --environment node src/lib/tiptap/codec.test.ts; grep -n 'generateJSON' web/src/lib/tiptap/preserved.ts || echo 'no DOM path (correct)'"
  - id: RE-02-4
    statement: "Tier-1 does not regress: the Phase-38 corpus (tree-equality/idempotence) + parity tests still pass unchanged; the new extensions are additive"
    verification: "cd web && npx vitest run --environment node src/lib/tiptap/"
```

## Tasks

### Task 1: PreservedBlock/PreservedInline + math tokenizers (Wave 1)

<read_first>
- agent://PreservedTierResearch (the REAL 3.27.1 sketch: Node.create with markdownTokenName/parseMarkdown/renderMarkdown/markdownTokenizer/code; the caveats)
- web/src/lib/tiptap/extensions.ts (buildExtensions — where the new nodes register)
- web/src/lib/tiptap/codec.ts (parse/serialize/roundTrip — unchanged; new extensions flow through)
</read_first>

<action>
**1. `web/src/lib/tiptap/preserved.ts`** — define the preserved nodes (real 3.27.1 API per research):
- `PreservedBlock` = `Node.create({ name:"preservedBlock", group:"block", atom:true, code:true,
  addAttributes:()=>({ raw:{default:""} }), parseHTML/renderHTML (editor DOM — inert), markdownTokenName:"table",
  parseMarkdown:(token)=>({ type:"preservedBlock", attrs:{ raw: token.raw } }), renderMarkdown:(node)=>node.attrs.raw })`.
- A sibling registration for block **`html`** (a second thin extension or companion) — same node
  render, distinct `markdownTokenName:"html"` (1:1 rule). Capture the **merged** raw.
- `BlockMath` = preserved block via a custom `markdownTokenizer` `{ name:"blockMath", level:"block",
  start:(src)=>src.indexOf("$$"), tokenize:(src)=>{ const m=/^\$\$([\s\S]+?)\$\$/.exec(src); return m?{type:"blockMath",raw:m[0]}:undefined } }`
  + `markdownTokenName:"blockMath"` + verbatim render.
- `PreservedInline` = `Node.create({ name:"preservedInline", group:"inline", inline:true, atom:true,
  code:true, raw attr, renderMarkdown:(n)=>n.attrs.raw })` for inline math (and inline HTML — see Task 3).
- Inline math via a custom inline `markdownTokenizer` `{ name:"inlineMath", level:"inline",
  start:(src)=>src.indexOf("$"), tokenize:(src)=>{ const m=/^\$(?!\$)([^$\n]+?)\$/.exec(src); return m?{type:"inlineMath",raw:m[0]}:undefined } }`.
  Guard: do NOT match `$$` (block) or a lone `$`; `.raw` = the exact `$…$` slice.

**2. Register in `buildExtensions()`** (extensions.ts): append the preserved nodes so the codec parses
+ serializes them. The `parse`/`serialize`/`roundTrip` API is unchanged.
</action>

<acceptance_criteria>
- `grep -nE 'PreservedBlock|PreservedInline|markdownTokenName|markdownTokenizer' web/src/lib/tiptap/preserved.ts` matches
- `grep -n 'preserved' web/src/lib/tiptap/extensions.ts` shows registration
- `cd web && npx tsc --noEmit` exits 0
</acceptance_criteria>

### Task 2: Table + block-math + block-HTML byte-parity corpus (Wave 1)

<read_first>
- web/src/lib/tiptap/codec.corpus.ts (TIER1_CORPUS pattern — add TIER2_CORPUS + MIXED_CORPUS)
- web/src/lib/tiptap/codec.test.ts (the node-env gate — add a Tier-2 byte-equality describe)
</read_first>

<action>
**1. `TIER2_CORPUS`** in codec.corpus.ts — real fixtures: a GFM table (header + rows), a `$$…$$` block
math block, a fenced-adjacent table, a block of raw HTML (e.g. `<div class="x">\n  <p>hi</p>\n</div>`),
and an inline-math-in-paragraph fixture.

**2. `MIXED_CORPUS`** — documents combining Tier-1 + Tier-2 (paragraph + table + inline math + fenced
code + a heading) to prove the preserved nodes compose with the bounded slice.

**3. Tier-2 gate in codec.test.ts** (node env): for each Tier-2 + mixed fixture,
`expect(roundTrip(src).trim()).toBe(src.trim())` (byte-for-byte modulo outer trim) AND idempotence
`roundTrip(roundTrip(src)) === roundTrip(src)`. (Tier-2 asserts BYTE equality — stronger than Tier-1's
tree-equality — because verbatim preservation is exact by contract.)
</action>

<acceptance_criteria>
- `grep -n 'TIER2_CORPUS\|MIXED_CORPUS' web/src/lib/tiptap/codec.corpus.ts` matches
- Tier-2 fixtures cover table, block math, block HTML, inline math, and mixed Tier-1+Tier-2
- `cd web && npx vitest run --environment node src/lib/tiptap/codec.test.ts` — Tier-2 byte-equality passes, OR a documented per-construct limitation is stated (honest, not a loosened assertion)
</acceptance_criteria>

### Task 3: HTML byte-parity + inertness (Wave 1, MAJ-04 — the security core)

<read_first>
- agent://PreservedTierResearch Q6/Q7/Q8 (default HTML path lossy; explicit token.raw handler; inline-HTML pre-emptive branch → custom inline tokenizer name≠'html'; capture merged raw)
- web/src/lib/tiptap/codec.test.ts (the existing SECURITY_CORPUS + HTML-inertness section from Phase 38)
</read_first>

<action>
**1. Block HTML:** register the block-`html` handler (Task 1) storing `token.raw` (merged) → verbatim.
Replaces the Phase-38 lossy default for block HTML.

**2. Inline HTML:** route via a custom **inline** `markdownTokenizer` whose `name` is NOT `'html'`
(e.g. `"inlineHtml"`) so it bypasses marked's pre-emptive inline-html merge branch and lands in the
generic handler → `PreservedInline` storing `.raw`. If, after honest effort, inline-HTML byte-parity
is fragile (per the research caveat), **document the exact limitation** in the SUMMARY (block HTML +
tables + math are byte-exact; inline HTML is best-effort/stated) — do NOT loosen the byte assertion.

**3. Sharpen the MAJ-04 security assertions** (extend the Phase-38 HTML-inertness section, node env):
- The raw-HTML fixtures (`<img src=x onerror=...>`, `<script>...`, `<div onclick=...>`, inline
  `<em>...`) now round-trip to their **EXACT source bytes** (byte-equality, not just escaped) — AND
  stay inert: the parsed JSON has NO element node (no `image`/`div`/`script`) and NO executable attr
  (`onerror`/`onclick`/...). The preserved node holds the HTML as a `raw` STRING; `renderMarkdown`
  re-emits it as literal text. `generateJSON`/DOM is never called.
- The no-window/no-document env guard stays.
</action>

<acceptance_criteria>
- `grep -n 'generateJSON' web/src/lib/tiptap/preserved.ts` returns NO match (no DOM path)
- The MAJ-04 section asserts BOTH exact-byte preservation AND no element node / no executable attr for raw HTML
- `cd web && npx vitest run --environment node src/lib/tiptap/codec.test.ts` passes (or inline-HTML limitation documented)
</acceptance_criteria>

### Task 4: No Tier-1 regression + verdict (Wave 1)

<action>
**1.** Run the FULL tiptap suite under node: `cd web && npx vitest run --environment node src/lib/tiptap/`
— the Phase-38 Tier-1 corpus + parity tests MUST still pass (the new extensions are additive; a `$`
in prose that isn't math, a stray `<`, must not break Tier-1 fixtures). If a Tier-1 fixture regresses,
the math/HTML tokenizers are over-matching — fix the guards.

**2.** `tsc --noEmit` + `eslint` clean; `npm run build` stays green (must not break the build).

**3.** SUMMARY states, per Tier-2 construct: byte-exact ✓ or documented limitation; the MAJ-04
inertness result; and that Tier-1 did not regress.
</action>

<acceptance_criteria>
- `cd web && npx vitest run --environment node src/lib/tiptap/` — all pass (Tier-1 + Tier-2 + parity + security)
- `cd web && npx tsc --noEmit` + `npx eslint src/lib/tiptap/` exit 0; `npm run build` exit 0
- SUMMARY enumerates per-construct byte-parity results honestly
</acceptance_criteria>

## Verification

- **Goal-backward (qa):** RE-02 delivers byte-for-byte preservation of unmodeled constructs, inert.
  1. RE-02-1: the preserved nodes use the REAL 3.27.1 hooks (markdownTokenName/parseMarkdown/
     renderMarkdown/markdownTokenizer); table + block-html intercepted; math tokenized.
  2. RE-02-2: Tier-2 fixtures genuinely assert BYTE-equality (modulo outer trim) + idempotence; mixed
     Tier-1+Tier-2 composes. Any per-construct limitation is documented, not hidden.
  3. RE-02-3 (SECURITY, MAJ-04): raw HTML is exact-byte preserved AND inert — no element node / no
     executable attr in parsed JSON; no `generateJSON`; node env. This is the milestone security-review surface.
  4. RE-02-4: Tier-1 (Phase-38 corpus + parity) does NOT regress.
- **Hard-bar honesty check:** if the SUMMARY claims byte-for-byte, qa confirms no byte assertion was
  loosened to "close enough" and no fixture skipped. A documented inline-HTML limitation is acceptable;
  a silently-widened assertion is a gap routed UP.
- **v1.6 lesson:** this extends shared `web/src/lib/tiptap/*` — run the FULL tiptap suite, not just the
  new Tier-2 tests; keep `npm run build` green.
