# OKF Studio

<p align="center">
  <img src="banner.png" alt="OKF Studio - turn any pile of knowledge into a portable LLM knowledge base" width="100%">
</p>

<p align="center">
  <em>Turn any pile of knowledge into a portable LLM knowledge base.</em><br>
  <em>An origin story, and a tool that came out of it.</em>
</p>

<p align="center">
  <a href="LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/License-MIT-yellow.svg"></a>
  <a href="https://github.com/ahn1305/OKF_Studio_EGT/stargazers"><img alt="Stars" src="https://img.shields.io/github/stars/ahn1305/OKF_Studio_EGT?style=social"></a>
  <a href="https://github.com/ahn1305/OKF_Studio_EGT/network/members"><img alt="Forks" src="https://img.shields.io/github/forks/ahn1305/OKF_Studio_EGT?style=social"></a>
  <a href="https://github.com/ahn1305/OKF_Studio_EGT/issues"><img alt="Issues" src="https://img.shields.io/github/issues/ahn1305/OKF_Studio_EGT"></a>
  <a href="CONTRIBUTING.md"><img alt="PRs welcome" src="https://img.shields.io/badge/PRs-welcome-brightgreen.svg"></a>
  <img alt="Python 3.9+" src="https://img.shields.io/badge/python-3.9%2B-blue">
  <img alt="Produces OKF v0.1" src="https://img.shields.io/badge/produces-OKF%20v0.1-8A2BE2">
</p>

---

### It started with a small, nagging annoyance.

Every time I asked an LLM a question about my own material (my notes, our
internal docs, a codebase I half-remembered), I had to hand it the same pile of
files *again*. Upload, wait, ask. Upload, wait, ask. The model would dutifully
read everything from scratch, answer once, and then forget it all. Tomorrow I'd
do it again. Nothing accumulated. Nothing got *smarter*. I was paying, over and
over, to rediscover things I'd already discovered.

That's RAG for most people: a brilliant librarian with no memory, re-reading the
whole shelf for every single question.

### Then two ideas collided.

First I read **Andrej Karpathy's "LLM Wiki"** note. His point was almost
obvious in hindsight: don't make the model re-derive knowledge at query time;
have it *build a wiki once* and keep it. A folder of plain markdown files,
cross-linked, that the LLM writes and maintains while you curate. The synthesis
happens up front; afterwards you just *navigate* it. The bookkeeping that makes
humans abandon their wikis (updating links, keeping summaries fresh) is exactly
the chore LLMs never get bored of.

Then Google shipped the **Open Knowledge Format (OKF)**, the same pattern,
finally written down as a tiny spec. A bundle is *just* a directory of markdown
with a sliver of YAML frontmatter. No SDK. No database. No lock-in. If you can
`cat` a file, you can read it; if you can `git clone` a repo, you can ship it.
Anyone can produce it, anything can consume it.

I remember thinking: *this is the format my scattered knowledge has been waiting
for.* But there was a gap. OKF tells you what a good knowledge bundle should
**look like**, but it doesn't **build one** from the mess you actually have:
Notion pages, Confluence spaces, a folder of PDFs, a few links, a repo.

### So I built the thing that fills the gap.

**OKF Studio** is a portable [Agent Skill](https://agentskills.io) you drop on
top of any LLM agent (Claude Code, VS Code Copilot, Cursor, Gemini CLI, Codex…).
You point it at your sources; it reads and understands them, designs a sensible
structure, and writes out a **conformant
[OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md)
bundle**: a durable, version-controllable, model-agnostic knowledge base you can
hand to *any* LLM as context.

The trick that makes it work is a division of labour I'm a little proud of:

> **Deterministic Python scripts do the boring, mechanical work**: ingesting
> sources, validating conformance, generating indexes, checking links, drawing
> the graph. **The LLM does the part it's uniquely good at**: understanding the
> material, designing the taxonomy, and writing the pages. Machines do bookkeeping;
> the model does the thinking.

And because the output is *just files*, the knowledge base outlives any one
model, vendor, or chat window. It's yours. It compounds. Every new source makes
it richer instead of starting the librarian over from page one.

That's the whole idea. The rest of this README is how to use it.

---

## Why it holds up

- **It replaces query-time RAG, not augments it.** The synthesis is done once, at
  build time. A consuming agent *navigates* curated pages instead of re-embedding
  and re-searching raw documents on every question.
- **Portable & lock-in free.** Output is plain markdown + YAML: `cat` it,
  `git clone` it, host it anywhere, load it into any model. No proprietary store.
- **Producer/consumer independence.** OKF Studio is a *producer*; anything that
  reads markdown is a *consumer* (an LLM, a search index, the bundled HTML graph
  viewer). The format is the contract.

  <a href="https://www.producthunt.com/products/github-429?embed=true&amp;utm_source=badge-featured&amp;utm_medium=badge&amp;utm_campaign=badge-okf-studio" target="_blank" rel="noopener noreferrer"><img alt="OKF Studio - Transform Enterprise Knowledge into Portable OKF Assets Fast | Product Hunt" width="250" height="54" src="https://api.producthunt.com/widgets/embed-image/v1/featured.svg?post_id=1186827&amp;theme=light&amp;t=1783093445429"></a>

## What's in the box

```
okf-studio/                     # the portable skill: this is what you install
├── SKILL.md                    # orchestrator: when/how, the 6-phase pipeline
├── references/                 # loaded on demand (progressive disclosure)
│   ├── okf-spec.md             # condensed authoritative OKF v0.1 rules
│   ├── workflow.md             # detailed phase-by-phase walkthrough
│   ├── connectors.md           # ingesting files, PDFs, URLs, Notion, Confluence…
│   └── authoring-guide.md      # how to write high-quality concept docs
├── scripts/                    # deterministic helpers, pure python3 stdlib
│   ├── ingest.py               # sources → staged captures + manifest
│   ├── validate_okf.py         # OKF v0.1 conformance checker
│   ├── build_index.py          # generate/refresh index.md files
│   ├── link_check.py           # cross-link graph + broken-link report
│   ├── visualize.py            # self-contained interactive HTML viewer
│   └── okf_common.py           # shared helpers
└── assets/                     # concept / index / log / agents templates
```

The split is deliberate: **scripts do the mechanical work** (ingest, validate,
index, link-check, visualize); **the LLM does the understanding and synthesis**
(taxonomy design and authoring), the part LLMs are uniquely good at.

## The pipeline

```
1 Connect   → confirm sources, scope, purpose
2 Ingest    → normalize sources into staged captures
3 Understand→ design the concept taxonomy + hierarchy
4 Synthesize→ author OKF concept docs (frontmatter + structured body + links)
5 Assemble  → generate index.md / log.md, then visualize
6 Validate  → conformance + link check; fix; iterate until clean
```

## Install the skill

OKF Studio follows the open [Agent Skills](https://agentskills.io/specification)
format, so installation is "put the `okf-studio/` folder where your agent looks
for skills." Common locations:

| Agent | Skills location |
|-------|-----------------|
| Claude Code | `~/.claude/skills/okf-studio/` (or `.claude/skills/` in a project) |
| VS Code / Copilot | your configured skills directory (`okf-studio/`) |
| Cursor | `.cursor/skills/okf-studio/` |
| Gemini CLI / Codex / others | the skills directory documented by the client |

Copy the `okf-studio/` directory there. The agent discovers it via `SKILL.md`
and loads the rest progressively. (Check your client's skills docs for the exact
path; the folder name must remain `okf-studio` to match the skill `name`.)

## Quick start (what the agent runs under the hood)

```bash
cd okf-studio

# 2. Ingest sources into a scratch staging dir
python scripts/ingest.py --out ./_ingest \
    --source ./docs --source ./report.pdf --source https://example.com/guide

# 3-4. (the LLM reads ./_ingest and authors concept docs into bundles/<name>/,
#       named after the source/topic, one bundle per source, never a generic name)

# 5. Assemble navigation + connection file + viewer
python scripts/build_index.py ./bundles/<name> --write
#    write a root AGENTS.md from assets/agents.template.md (self-describing bundle)
python scripts/visualize.py --bundle ./bundles/<name>   # → bundles/<name>/viz.html

# 6. Validate and link-check; iterate until clean
python scripts/validate_okf.py ./bundles/<name>
python scripts/link_check.py ./bundles/<name>
```

Then load `./bundles/<name>` into any LLM as context; the bundle's `AGENTS.md`
tells the agent to start at `index.md` and navigate (OKF-native, not RAG).

> **Output layout:** bundles live under `bundles/<name>/`, named after their
> source or topic, so different pages/datasets stay separate and regenerable.
> A bundle may include `AGENTS.md` / `CLAUDE.md` / `README.md` at its root to be
> self-describing; these are ignored by the validators (not OKF concepts).

## Requirements

- **Python 3.9+** for the scripts (standard library only).
- **Optional:** `pip install pypdf` for PDF text extraction. Network/auth
  connectors (Notion, Confluence, Drive, Slack) work best through your host
  agent's connectors/MCP: export to files, then ingest. See
  `okf-studio/references/connectors.md`.

## Security

- Ingested content is treated as **untrusted data**, never as instructions
  (prompt-injection aware). Suspicious instructions in sources are surfaced, not
  obeyed.
- Secrets/tokens found in sources are redacted, never written into the bundle.
- Scripts are read-only over your sources and never connect to production
  systems with write access.

## Contributing

Contributions of every size are welcome: bug reports, docs, new connectors, new
scripts, or whole new ideas.

- Read the [Contributing guide](CONTRIBUTING.md) for setup and the local checks.
- Be excellent to each other: [Code of Conduct](CODE_OF_CONDUCT.md).
- Found a security issue? See [SECURITY.md](SECURITY.md).
- Track changes in the [Changelog](CHANGELOG.md).

Good first contributions: add a connector recipe to
`okf-studio/references/connectors.md`, sharpen the `assets/` templates, or
contribute a small public sample bundle under `examples/`.

## One last thing

The first time you watch an agent take a folder of forgotten PDFs and a couple of
stale wiki links and turn them into a clean, cross-linked little library, one you
can open in a browser, commit to git, and hand to any model, and something clicks.
The knowledge stops being a pile you keep re-explaining and becomes a thing that
*remembers itself*.

That's the feeling this whole project is chasing. If it saves you from uploading
the same files one more time, it did its job.

## References & Further Reading

**Open Knowledge Format (OKF)**
- [Introducing the Open Knowledge Format](https://cloud.google.com/blog/products/data-analytics/how-the-open-knowledge-format-can-improve-data-sharing/): Google Cloud announcement blog
- [OKF v0.1 Specification (SPEC.md)](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md): the authoritative spec
- [knowledge-catalog repo](https://github.com/GoogleCloudPlatform/knowledge-catalog/tree/main/okf): reference producer agent, HTML visualizer, sample bundles

**The LLM-wiki pattern**
- [Andrej Karpathy, LLM Wiki gist](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f): the originating pattern (RAG alternative, raw to wiki to schema)

**Agent Skills**
- [Agent Skills Overview](https://agentskills.io/home)
- [Agent Skills Specification](https://agentskills.io/specification): `SKILL.md` format & rules
- [Anthropic Agent Skills](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview): origin of the open standard
- [Microsoft Agent Framework, Agent Skills](https://learn.microsoft.com/en-us/agent-framework/agents/skills): progressive disclosure, scripts, security
- [VS Code Agent Skills](https://code.visualstudio.com/docs/copilot/customization/agent-skills)
- [Claude Code Skills](https://code.claude.com/docs/en/skills)

**Inside this skill** (loaded on demand)
- [okf-studio/references/okf-spec.md](okf-studio/references/okf-spec.md): condensed authoritative OKF v0.1 rules
- [okf-studio/references/workflow.md](okf-studio/references/workflow.md): detailed phase-by-phase walkthrough
- [okf-studio/references/connectors.md](okf-studio/references/connectors.md): per-source ingestion guide
- [okf-studio/references/authoring-guide.md](okf-studio/references/authoring-guide.md): writing high-quality concept docs

## Star History

If OKF Studio is useful to you, a star helps others find it and tells us to keep
going.

<a href="https://star-history.com/#ahn1305/OKF_Studio_EGT&Date">
  <img alt="Star History Chart" src="https://api.star-history.com/svg?repos=ahn1305/OKF_Studio_EGT&type=Date" width="70%">
</a>

## Contributors

Thanks to everyone who has contributed.

<a href="https://github.com/ahn1305/OKF_Studio_EGT/graphs/contributors">
  <img alt="Contributors" src="https://contrib.rocks/image?repo=ahn1305/OKF_Studio_EGT">
</a>

<sub>Star count, history, and contributor avatars populate automatically once the
repo is public on GitHub.</sub>

## License

[MIT](LICENSE). Use it, fork it, ship it.

Open Knowledge Format is an open specification by Google Cloud; the Agent Skills
format is an open standard originally from Anthropic.
