markdown_parse pipeline — newsletter to knowledge graph

markdown_parse pipeline — newsletter to knowledge graph A data-flow diagram generated by Archify. 01 / Sources 02 / Convert / Capture 03 / Parse & Curate 04 / Maintain 05 / Serve Reports/ PDFs · 01 / Sources · input Reports/ PDFs input Existing newsletter .md · 01 / Sources · input Existing newsletter .md input pdf_conv_md.py · 02 / Convert / Capture · local-first pdf_conv_md.py local-first capture_newsletter_images.py · 02 / Convert / Capture · md inputs only capture_newsletter_images.py md inputs only parse_newsletter.py · 03 / Parse & Curate · one entry point parse_newsletter.py one entry point findata/ vault · 03 / Parse & Curate · writer-owned findata/ vault writer-owned Stage 5 curation · 03 / Parse & Curate · only manual step Stage 5 curation only manual step make maint-full · 04 / Maintain · 14 steps make maint-full 14 steps SQLite research.db · 05 / Serve · source of truth SQLite research.db source of truth DuckDB graph cache · 05 / Serve DuckDB graph cache FTS + embeddings · 05 / Serve · rebuildable FTS + embeddings rebuildable Reader frontend · 05 / Serve Reader frontend convert (engine chain) edition note capture signed-URL figures in-place rewrite parse edition new entities + relations edition blocks + splice converge + re-derive backup · sync · derive graph-rebuild FTS + embed rebuild entities · events · quotes graph views note search Legend primary data async batch data store data flow

Evidence origin/main e37eb7db

  • • pdf_conv_md.py:454 · capture_newsletter_images.py:116
  • • parse_newsletter.py:774 · maint.py:203 (PRE_FULL)
  • • app.py:23 · Makefile maint-full

Engine chain

  • • pdf_conv_md.py: pymupdf4llm primary, OCR fallbacks, Paddle last
  • • writes <slug>.md + images/ + OKF frontmatter + verify manifest

Curation safety

  • • Stage 5 is the only manual step (worklist-driven)
  • • auto chatter blocks are sentinel-wrapped, never clobber hand ones
  • • sources[] spliced via backfill_okf_provenance / PRE_FULL

Idempotence & sync

  • • relations dedupe on UNIQUE(source, target, edge_type)
  • • derive rows DELETE-then-INSERT on source_ref LIKE 'derive:%'
  • • capture resumable; okf converger no-op once converged