# Workflow — the OKF Studio pipeline in detail

A detailed, example-driven walkthrough of the six phases. Read this when you
need more than the summary in `SKILL.md`.

Guiding principle: **scripts do the mechanical work; you do the thinking.** The
hard, valuable part is Phase 3 (taxonomy design) and Phase 4 (synthesis) — that
is exactly the work LLMs are good at and humans abandon.

---

## Phase 1 — Connect (scope before you pull)

Decide and record, ideally in a short note to the user:

- **Bundle name & path** — name it after the source or topic and write it under
  `bundles/<name>/` (e.g. `./bundles/dezprox`, `./bundles/acme-data`). Use one
  bundle per source/dataset so different pages and data stay separate and can be
  regenerated independently. Avoid generic names like `my-kb`. The skill never
  overwrites a bundle silently; if the directory exists, confirm intent.
- **Purpose & consumer** — "an agent answering analytics questions", "onboarding
  new engineers", "a support bot". The purpose decides the taxonomy and the
  level of detail.
- **Sources & exclusions** — list every source and what to skip (secrets,
  archived docs, generated files). Get explicit approval before large pulls.

Output of this phase: a one-paragraph charter + a source list.

---

## Phase 2 — Ingest (normalize, with provenance)

Run `scripts/ingest.py` to stage everything into a scratch `_ingest/`
directory. Each capture keeps `source`, `source_kind`, `title`, and
`ingested_at` in its frontmatter so you can cite provenance later.

```bash
python scripts/ingest.py --out ./_ingest \
    --source ./engineering-docs \
    --source ./architecture.pdf \
    --source https://wiki.internal/runbooks/oncall
```

- `_ingest/manifest.json` lists every capture with `char_count`.
- For sources the script can't reach (Notion, Confluence, Drive, Slack,
  databases), see `references/connectors.md` — export via the host agent first,
  then point `ingest.py` at the exported files.
- **`_ingest/` is scratch input. Never ship it inside the final bundle.**

Read the captures (or have the host summarize large ones) to build a mental
model of the material before designing anything.

---

## Phase 3 — Understand & design the taxonomy

This is the highest-leverage step. **Design the structure before writing files.**

1. **Inventory concepts.** Scan captures and list the distinct units of
   knowledge. One idea = one concept = one future file. Typical concept kinds:
   datasets, tables, metrics, APIs/endpoints, services, runbooks/playbooks,
   policies, glossary terms, architecture decisions, how-to guides, references.

2. **Choose a hierarchy by consumer mental model, not by source.** Group into
   subdirectories the way a reader will navigate:
   ```
   acme-data-kb/
   ├── datasets/        # what data exists
   ├── tables/          # detailed schemas
   ├── metrics/         # business definitions
   ├── pipelines/       # how data moves
   ├── runbooks/        # what to do when X breaks
   └── references/      # mirrored external docs / citations
   ```
   Keep it 2–4 levels deep. Avoid one giant flat directory and avoid
   source-shaped folders like `notion/`, `pdf-exports/`.

3. **Assign a `type` to each concept.** Short, descriptive, free-form, reused
   consistently (a small controlled vocabulary makes filtering and coloring in
   the visualizer useful). Examples: `Dataset`, `Table`, `Metric`, `Runbook`,
   `API Endpoint`, `Concept`, `Decision`, `Reference`.

4. **Map the key relationships** (cross-links) up front: which metric depends on
   which table, which runbook references which pipeline. These become markdown
   links and the edges of the knowledge graph.

Output of this phase: a proposed file tree + a type list + a relationship map.
Confirm with the user if the bundle is large.

---

## Phase 4 — Synthesize concept documents

Author one file per concept using `assets/concept.template.md`. Rules:

- **Frontmatter:** always a non-empty `type`. Add `title`, `description` (a
  single clear sentence), `resource` (canonical URI if the concept maps to a
  real asset), `tags`, and an ISO-8601 `timestamp`.
- **Body:** lead with structure. Use tables for schemas, lists for steps, fenced
  code for examples. Use conventional headings `# Schema`, `# Examples`,
  `# Citations` where they fit.
- **Cross-link** related concepts inline using bundle-absolute paths:
  `Derived from [orders](/tables/orders.md) and [sessions](/tables/sessions.md).`
- **Ground every claim.** If a fact comes from a capture, keep it faithful; if
  from the web, add a `# Citations` entry. **Do not invent** schemas, numbers,
  or APIs. If something is unknown, say so or leave a `TODO` rather than guess.
- **Atomicity:** if a file is trying to describe two things, split it.

See `references/authoring-guide.md` for worked examples and anti-patterns.

---

## Phase 5 — Assemble

Generate navigation, the connection file, and a viewer:

```bash
python scripts/build_index.py ./bundles/acme-data --write    # index.md everywhere
python scripts/visualize.py --bundle ./bundles/acme-data     # → viz.html
```

- `build_index.py` writes a current `index.md` in every directory, pulling
  titles/descriptions from frontmatter. The root index gets `okf_version: "0.1"`.
- Write a root **`AGENTS.md`** (and a thin **`CLAUDE.md`** pointer) from
  `assets/agents.template.md`, replacing `{{BUNDLE_NAME}}`. This makes the bundle
  self-describing so any agent knows to read `index.md` first and navigate
  (OKF-native, not RAG). Agent-convention files (`AGENTS.md`, `CLAUDE.md`,
  `README.md`, …) may sit at the bundle root and are ignored by the validators.
- Optionally hand-author a root `log.md` (`assets/log.template.md`) summarizing
  what was created, newest first.
- `visualize.py` emits a single self-contained `viz.html` (a reference OKF
  consumer) for human review of the graph.

---

## Phase 6 — Validate & iterate (the loop)

```bash
python scripts/validate_okf.py ./bundles/acme-data      # conformance (errors vs warnings)
python scripts/link_check.py ./bundles/acme-data        # broken links + orphans
```

- **Errors** (unparseable frontmatter, empty `type`) MUST be fixed.
- **Warnings** (empty `description`, broken internal links, orphan concepts) —
  triage and fix the meaningful ones. Orphans usually mean a missing
  cross-link; broken links usually mean a wrong path or an unwritten concept.
- After fixes, **re-run `build_index.py --write` and `visualize.py`** so
  navigation and the viewer reflect the final state.

Repeat Phases 4–6 until `validate_okf.py` is clean and the quality bar in
`SKILL.md` is met. Then hand the bundle to the user — it is plain markdown they
can commit to git, host anywhere, or load directly into any LLM as context.

---

## Incremental updates (re-running on an existing bundle)
- Ingest only the new/changed sources.
- Add or edit the affected concept files; update their `timestamp`.
- Append a dated entry to `log.md`.
- Re-run `build_index.py --write`, then validate, link-check, and visualize.
OKF is designed for this: bundles grow and get refactored over time, and the
permissive consumption model means partial states stay usable.
