FinData storage topology

FinData storage topology An architecture diagram generated by Archify. Ingest helpers · pdf_conv_md · parse_newsletter · Architecture component Ingest helpers pdf_conv_md · parse_newsletter Maintenance chains · make maint / maint-full · Architecture component · 14 steps Maintenance chains make maint / maint-full 14 steps findata/ vault · OKF markdown notes · Repo-local storage · writer-owned findata/ vault OKF markdown notes writer-owned SQLite research.db · source of truth · Repo-local storage · entities · edges · events SQLite research.db source of truth entities · edges · events DuckDB graph cache · graph.duckdb · Repo-local storage · views + analytics DuckDB graph cache graph.duckdb views + analytics Reader frontend · Flask + static TS · Architecture component Reader frontend Flask + static TS Derived caches · corpus.db · embeds · doc-search · Repo-local storage Derived caches corpus.db · embeds · doc-search db-backup twins · zstd sidecars · Repo-local storage db-backup twins zstd sidecars Snapshot parquets · duckdb/ + sqlite/ · Repo-local storage Snapshot parquets duckdb/ + sqlite/ Graph algorithms · Python + Mojo bridge · Architecture component Graph algorithms Python + Mojo bridge Writer · Architecture component · person · writer-owner Writer person · writer-owner GitHub origin · Architecture component · external system GitHub origin external system entities · events sync_tags rebuild materializes serves graph reads VACUUM · integrity reads corpus walk edits OKF notes hosts origin/main Repo-local storage Legend Frontend Backend Database External

Source of truth

  • • SQLite research.db owns entities, edges, events, quotes
  • • Ingest helpers write OKF notes into the vault; sync_tags lands entities/edges in SQLite
  • • findata/ vault is the writer-owned note corpus
  • • Writer owns vault edits; GitHub origin hosts the repo — sources anchors pin origin/main

Derived

  • • DuckDB cache re-materialized by graph-rebuild
  • • Parquet snapshots export both stores
  • • corpus.db, embeds and doc-search rebuildable; algorithms write analytics back to SQLite

Operator path

  • • maint-full runs 14 steps: PRE_FULL convergers, backup, rebuilds
  • • db_maint takes the recovery backup + zstd twins before mutation
  • • Note mutation carve-out: machine-owned provenance keys only