# Conventions — the contract between services

Every service depends on these shapes; a mismatch is an integration bug rather
than a local one. `okf_core` is the shared library that encodes most of them —
import from it instead of re-implementing.

## S3 bundle bucket layout (source of truth)

```
okf/<data_domain>/
├── _domain/overview.md              # type: Domain (declared-domain concept doc)
└── <dataset>/
    ├── index.md                     # auto-generated (regenerate_indexes)
    ├── datasets/<dataset>.md        # type: Glue Database
    ├── tables/<table>.md            # type: Glue Table (one per table)
    ├── references/<type>/<slug>.md   # type: Reference — canonical fact-typed
    │                                 #   folders: joins/ metrics/ enums/
    │                                 #   named_sets/ glossary/ known_issues/
    │                                 #   recipes/ (mandatory query transforms,
    │                                 #   authored only when a dataset needs one)
    │                                 #   computations/ (type: Attested
    │                                 #   Computation — see "Attested
    │                                 #   Computations" below)
    │                                 #   (one doc per item; see okf-authoring skill)
    ├── external/<d>/<ds>/…           # type: Cross-Dataset Reference — one subtree
    │                                 #   per counterpart dataset, written ONLY by a
    │                                 #   cross-mode harvest (see "Cross-dataset
    │                                 #   references" below); overview.md + the same
    │                                 #   fact-typed folders (joins/ metrics/ …)
    ├── .context/                     # user-uploaded source docs (persisted)
    ├── .metadata/                    # read-only Glue metadata snapshot (per run)
    │                                 #   (+ .metadata/external/<d>/<ds>/ on a cross
    │                                 #   run: the target's snapshot + published docs)
    └── .harvest/state.json           # commit marker (status: complete | in_progress)
                                      #   + graph.json (precomputed /graph artifact)
```

The `_domain/overview.md` doc is a **derived** materialisation of the declared
domain's description + context. Written THROUGH the harvest mount (uid 1000) on
`PUT /domain-defs/{domain}` so the `<domain>/` directory is established with
correct ownership before any dataset-level write. `_domain` is a reserved
pseudo-dataset that `parse_bundle_key` parses normally (3 segments:
`domain/_domain/overview.md`) and reindex embeds with `type=Domain`. Hidden from
the dataset listing by `is_domain_dataset()`. Vector key:
`<domain>/_domain/overview`.

- **Concept id** is the path under `okf/<domain>/<dataset>/` minus `.md`, e.g.
  `tables/races`. Use the `okf_core.paths` helpers.
- **S3 object key** is `okf/<domain>/<dataset>/<concept_id>.md`.
- **Vector key** is `<domain>/<dataset>/<concept_id>` — the S3 key without the
  `okf/` prefix and `.md` suffix. Use `okf_core.embedding.vector_key`.
- `.context/`, `.metadata/`, and `.harvest/` are dot-prefixed and are not
  concepts. The reindex worker ignores any key with a dot-prefixed segment below
  the dataset root, and ignores `index.md` and `log.md`.
- `.metadata/` is a **read-only Glue metadata snapshot** the harvest writes ONCE
  at the start of each run (`harvest/metadata_export.py`): `index.md` (manifest),
  `database.md`, `columns.tsv` (one line per `table\tcolumn\ttype\tcomment` — the
  cross-table grep target for join/near-synonym discovery), and `tables/<t>.md`
  per table. The agent reads it with the built-in `read_file`/`glob`/`grep` (it
  replaced the old `list_concepts`/`read_concept_raw` tools); the OKF write-guard
  refuses any write into it. Live verification stays on the `sample_rows`/
  `run_sql` tools. Like `.context/`, it is a harvest INPUT and is never published,
  indexed, or embedded; `clean_authored_output` preserves it (dot-prefixed), and
  `export_metadata` rewrites it fresh each run so a dropped table leaves no stale
  sheet.
- A bundle is consumable only once `.harvest/state.json` exists with
  `status == "complete"`.
- `.harvest/graph.json` is the **precomputed link-graph artifact** the Control
  API's `GET /bundle/{d}/{ds}/graph` serves: `{completed_at, nodes, edges}`,
  built by `okf_core.graph_json.build_graph_json`. `finalize_bundle` writes it
  just BEFORE the commit marker, stamped with the SAME `completed_at` the
  marker carries — the endpoint serves it iff the two match (`status ==
  "complete"` and equal timestamps), so every mutating mode (full, scoped/
  incremental, annotation, cross) refreshes it by construction, and repromote
  rewrites it too (`restore_snapshot` only touches non-dot `.md` files, so the
  stale artifact would otherwise linger). Any mismatch — mid-run marker,
  legacy bundle, failed precompute — falls back to computing the same JSON
  live from the docs (parallel S3 fetch; one shared builder keeps the two
  paths identical). Derived data: writes are best-effort and never fail a
  harvest or repromote.
- `.harvest/review/` holds the review workflow's state: `clusters.json` (the
  persisted clustering with stable ids — a `run_review(cluster_ids=[...])`
  retry re-runs those clusters on THIS clustering, never a recomputed one;
  when the run recorded context digests it also carries a `context` section
  pairing them into `x1..xN` groups under the same retry contract) and
  one `report-<id>.md` per `run_review` call (unique name, nothing
  overwritten) with every reviewer/fixer transcript. Like the rest of
  `.harvest/`, it is state — never published, indexed, or embedded. A FULL
  harvest wipes `.harvest/review/` at start (alongside the authored-output
  wipe): a clustering from a previous run describes docs the run is about to
  rebuild, so nothing may retry against it.
- `.harvest/context/` holds `digest-NN.md` — the verbatim output (dispatch
  brief + returned digest) of every `context-extractor` dispatch, recorded
  automatically as each completes (`harvest/context_digests.py`, fed from the
  QuickJS shim and the static `task` path). `run_review`'s context-fidelity
  phase audits the bundle against these after all cluster fixes; no digests
  means the phase is skipped. Run-scoped on both ends: `finalize_bundle`
  deletes the dir right AFTER the commit marker (the audit is over; a run
  that fails earlier keeps them for debugging), and a full harvest wipes it
  at start anyway — the guard for a predecessor that crashed before its own
  cleanup.

**Derived artifacts live OFF the mount prefix.** Sibling top-level prefixes
sit next to `okf/` in the same bucket and are deliberately NOT under it, so
nothing an LLM role's file tools can reach ever sees them:
`benchmark/<domain>/<dataset>/` (the gold-carrying questions CSV + report
artifacts — see "Benchmark Studio invocation" below),
`policy/<domain>/<dataset>/` (the policy-check artifacts — see "Policy
checks (LLM-judge engine)"), and the export/import staging
`exports/<domain>/<dataset>/` + `imports/<domain>/<dataset>/` (fixed-key zip
+ validation record — see "Bundle export & import"). All are derived or
transient: the bundle stays the source of truth, and deleting a dataset
purges every one of these prefixes along with it (versions included; a
lifecycle rule also ages out the staging prefixes' noncurrent versions).

### Cross-dataset references (`external/`)

`external/<counterpart_domain>/<counterpart_dataset>/…` holds docs representing
knowledge that SPANS this dataset and one counterpart (verified cross-dataset
joins, cross-dataset metrics, the pair overview) — Roadmap §5's OSS flat-trust
mode. Rules:

- **The pair docs have exactly ONE home: the bundle of the dataset whose cross
  harvest authored them.** Nothing is ever written into the counterpart's
  bundle. This is the load-bearing decision: a mirrored copy would make the
  pair a distributed fact across two independently versioned, independently
  restorable bundles, so a full harvest OR a **repromote** of the counterpart
  (restoring a version from before/after a sync) would silently desynchronize
  it, with no transaction able to span the two. One home means a dataset's
  version history is self-contained and pair state cannot drift.
- Written ONLY by a `mode="cross"` harvest; the write guard confines a cross run
  to exactly its pair subtree, and every other mode never touches `external/`.
- The counterpart's discoverability is a DERIVED signal, not a copy — see
  "Cross-dataset reference signal" below.
- Every doc carries `type: Cross-Dataset Reference` and a `cross_dataset:
  {source: {data_domain, dataset}, target: {…}}` frontmatter block (`source` =
  the initiating side, i.e. where the docs live). Prose is symmetric (read by
  consumers of BOTH datasets) and tables are named as qualified SQL
  identifiers.
- **Links go to BOTH sides — a link is an address, and addresses may go
  stale.** Home-side docs are linked file-relative as usual (from a `joins/`
  doc: `../../../../tables/<t>.md`) — these resolve in the per-bundle link
  graph and stitch the pair subtree into it (backlinks from a table surface its
  cross-dataset joins). Counterpart docs are linked with the bundle-ESCAPING
  relative form (`../../../../../../<td>/<tds>/tables/<t>.md`): all bundles
  share one `okf/` tree, so the address resolves when the tree is browsed as
  files, and the UI's Browse view follows it into the other dataset. The link
  resolver (`okf_core/links.py`) deliberately DROPS bundle-escaping links from
  the graph (OKF tolerates dangling cross-bundle links), and a re-harvest of
  the counterpart may dangle the address — accepted; the qualified SQL
  identifier in the prose is the durable reference.
- The docs are ordinary published concepts: listed, served, embedded, and
  searchable exactly like the rest of the bundle (concept id
  `external/<d>/<ds>/joins/<slug>` etc.). Nothing downstream special-cases them
  beyond the annotation scope filter below.
- **A full harvest deletes `external/` along with everything else**
  (`clean_authored_output`'s delete-every-non-dot-entry rule — there is
  deliberately no keep-list). Re-run the cross harvest to restore the pair docs;
  vectors and the XREF signal are pruned/rebuilt through the normal reindex
  event path.
- A cross re-run of the same pair replaces that pair's subtree wholesale; other
  pairs' subtrees are untouched.

### Attested Computations (`references/computations/`)

Frozen, parameterized, read-only SQL authored at harvest and executed by
filling typed `@parameter` holes — never by editing the statement (design:
docs/ATTESTED_COMPUTATIONS.md; pure rules: `okf_core/computations.py`; shared
S3/engine runner: `okf_aws/computation_run.py`).

- **Doc contract** (write-guard-enforced): `type: Attested Computation`,
  directly under `references/computations/` (flat; the filename is the slug);
  `runtime: athena | redshift`; `parameters` list (`name`/`type`/`required`,
  `example` required, optional `default`/`enum`/`min`/`max`/`column`; an
  optional parameter must carry a `default`); ONE SELECT/WITH statement in a
  ```sql fence under `# Computation`, every `@hole` declared and every
  declared parameter used.
- **Content hash**: sha256 over the fence text (trailing-whitespace-stripped
  lines), the canonical JSON of `parameters`, and the `runtime` string
  (`computation_sha256`). Human verification signs this hash; any edit changes
  it and the stamp reads `stale`.
- **Verification triple** (`verified` / `verified_by` / `verified_sha256`):
  null until a human acts. Agents can never set them — the guard refuses any
  write that sets a non-null value it isn't PRESERVING verbatim from the
  existing doc. **A verified computation is FROZEN in the in-place modes**
  (incremental / annotation / cross): the runner resolves the verified set at
  run start (`harvest/verification.frozen_computation_paths` — folded stamps
  ∪ overlay, hash-checked), the guard refuses agent write/edit/delete on
  those docs, and lint downgrades their findings to warnings ("a human must
  unverify"). Human Unverify is the only unlock. A **full harvest is exempt**
  — `clean_authored_output` is unconditional (no keep-list) and the agent
  re-authors from source, so verification returns to the human's queue unless
  the fence was reproduced verbatim AND its overlay click had not yet folded
  in. Verify/Unverify flips land in the **off-mount overlay**
  `verification/<domain>/<dataset>.json` (`{version, entries: {slug: {slug,
  sha256, verified, verified_by}}}`; unverify writes a `revoked` tombstone) —
  never on the doc: the mount is the bundle tree's sole writer. The runtime
  folds the overlay into doc frontmatter inside `finalize_bundle` (every
  mode, after authoring, before the commit marker — `harvest/verification.py`);
  serving merges doc + overlay with the overlay winning, and a hash mismatch
  on either side surfaces as `stale`.
- **Execution surfaces** (one runner, three fronts): consumption MCP tools
  `list_computations` / `describe_computation` / `run_computation` (execution
  gated by `OKF_COMPUTATIONS_ENABLED` ⟵ `var.enable_attested_computations`,
  default **false**); chat's `run_computation` (ALWAYS bound; executes
  only under `var.enable_chat_sql`'s grants, policy checker when armed); Control API
  `GET/POST /bundle/{d}/{ds}/computations[/{slug}][/run|/verify|/unverify]`
  (verify identity comes from the JWT, never the body). Declared
  `enum`/`min`/`max` are CONTRACT (refused); profiled domains
  (`.metadata/profile/domains.json`, written by the profile pass) are ADVISORY
  (warn-and-run). Every run returns the receipt: `executed_sql` (verbatim),
  `computation_sha256`, `verification`, `engine_query_id`, `warnings`.
- **Lint**: the `computations` step validates doc shape, `column` bindings
  against the snapshot, and declared enums against the profile evidence; the
  EXPLAIN gate re-enters each valid computation with its `example` values
  substituted (raw `@hole` fences classify as templated and are never sent).

## Bundle versions & repromote

The bundle bucket is versioned, and `finalize_bundle` writes `.harvest/state.json`
LAST — so version history needs no manifest: a **bundle version** is one
`status: "complete"` object version of that marker, identified by the marker's
own S3 `VersionId` and labeled by its `completed_at`. The file set of a version
is reconstructed on read (`okf_aws.s3_versions`): for every non-dot `.md` under
the dataset prefix, the newest object version with `LastModified <=` the
marker's (absent if that entry is a delete marker). `in_progress` marker writes
delimit nothing and are filtered out — an interrupted (cancelled/crashed)
harvest therefore never becomes a version; its half-written live state is
inspectable via the diff `to=live` sentinel and rolled back by repromoting the
last good version.

Endpoints (Control API): `GET /bundle/{d}/{ds}/versions`,
`GET /bundle/{d}/{ds}/diff?from=&to=` (both optional — defaults answer "what
changed in the last harvest"; `to=live` compares against the working files),
`POST /bundle/{d}/{ds}/repromote {version_id}`, and `GET .../repromote` (the
convergence poll). The built-in chat agent additionally gets a
`get_bundle_diff` tool (same module, agent-bounded output) — deliberately NOT
registered on the consumption MCP server, so external agents see only the
published bundle, never its history.

**Repromote and the S3 Files mount.** The repromote's two ``state.json``
writes (``in_progress`` + the fresh ``complete`` marker) are plain ``PutObject``
calls from the Control API — they carry none of the POSIX file-mode metadata
the harvest runtime's S3 Files mount stores on mount-written objects, so the
mount presents the marker READ-ONLY afterward. ``harvest.fsutil.write_text``
heals this (EACCES → unlink + rewrite; the parent dir is mount-created and
writable), so the next harvest of a repromoted dataset proceeds normally.
Restored docs are unaffected: ``CopyObject`` preserves the source object's
metadata.

**Repromote is append-only**: every file of the target version is
`CopyObject`-ed from its source `VersionId` onto the same key (S3 mints NEW
current versions — old ids are never resurrected), live docs absent from the
target get a delete marker, and a FRESH `complete` marker is written carrying
`repromoted_from` (the restored marker VersionId) + `repromoted_by` (caller
identity). The untouched reindex pipeline converges the vector index from the
resulting object events. Repromote deliberately does NOT touch the freshness
table's Glue-version rows: it is a content rollback, not a pin — the next
genuine catalog change (or manual harvest) legitimately overwrites it.

**Retention**: `var.bundle_version_retention_days` (durable stack, default 90)
lifecycle-expires noncurrent bundle versions — this IS the repromote window —
while always keeping the 3 newest noncurrent versions per key. Expired versions
simply drop out of the reconstructed list (nothing dangles). SAFETY COUPLING:
lifecycle expiry emits `Object Deleted` events with deletion-type
`"Permanently Deleted"` for keys whose live doc is untouched; the reindex
worker MUST keep filtering those (only `"Delete Marker Created"` reaches
`DeleteVectors`) or daily expiry would delete live docs' vectors.

## Bundle export & import

A bundle is a portable artifact: **export** zips the PUBLISHED tree (the same
rule the file endpoints serve — non-dot `.md` only, so `.context`/`.metadata`/
`.harvest` and the zero-byte dir markers stay out, and computation
VERIFICATION — off-mount by design — cannot travel), rooted at `<dataset>/`
with a top-level `manifest.json` sibling (`{format, data_domain, dataset,
exported_at, files}` — the only machine-checkable provenance: docs carry no
dataset identity, which is exactly what makes them portable). `POST
/bundle/{d}/{ds}/export` stages the archive at the FIXED off-mount key
`exports/<d>/<ds>/okf-bundle.zip` (overwritten per export; the versioned
bucket keeps history) and returns a presigned GET with an attachment
disposition.

**Import** is repromote with the snapshot swapped for an uploaded zip, three
endpoints under `POST /bundle/{d}/{ds}/import[...]`:

1. **presign** (`/import`) — a presigned POST pinned to the off-mount staging
   key `imports/<d>/<ds>/upload.zip`, size-capped by the POST policy itself.
   404 unless the dataset is REGISTERED (the registry row is the lease
   anchor); a never-harvested registered dataset is a fine target — that is
   the point (move a bundle between deployments without re-running the
   authoring).
2. **validate** (`/import/validate`) — dry-runs the staged archive and names
   every problem at once (the content checks run even when structural
   blockers already exist — no second refusal round), in two deliberate
   tiers: **blockers** (zip traversal/absolute paths, dot-prefixed and
   agent-scratch segments per `is_reserved_rel_segments`, non-markdown,
   non-UTF-8, unreadable/corrupt/encrypted members, duplicate entry names,
   file-vs-directory path collisions, wrong root folder or manifest
   `dataset`, size/count caps) which apply refuses outright, and
   **findings** — the SAME offline `okf_core.lint` steps a harvest gate runs
   (coverage self-skips: no `.metadata` travels) plus live `EXPLAIN` of the
   archive's runnable ```sql against THIS deployment's catalog (budget-
   capped, skips and engine TIMEOUTS counted honestly as unchecked — a slow
   engine is never reported as a defect in the doc's SQL; `external/`
   fences are skipped like the harvest gate skips them — a counterpart's
   tables are outside this dataset's database) — which are the human's
   call. Caps bind on ACTUAL decompressed bytes via bounded streaming reads
   (64MB archive / 4MB per doc / 128MB total / `MAX_RESTORE_FILES` docs) —
   the zip's declared sizes are never trusted. The report carries
   `archive_sha256`, and validate persists a small record
   (`imports/<d>/<ds>/validate.json`: digest + finding counts) that apply
   requires.
3. **apply** (`/import/apply {acknowledged, archive_sha256}`) — refuses
   blockers (400), refuses an archive with NO validation record matching the
   staged digest (409 — validate first; the record is what lets the
   acknowledged gate cover the live-EXPLAIN findings apply cannot re-derive,
   and a client-supplied `archive_sha256` additionally pins what the human
   reviewed), refuses unacknowledged findings (409), then: harvest lease
   with `mode="import"` → `in_progress` marker → write docs + delete live
   published `.md` the archive lacks (dot-dirs survive, and non-markdown
   strays are LEFT ALONE — version snapshots are `.md`-only, so deleting one
   would be irreversible) in parallel (serial writes × hundreds of docs
   would blow the 30s Lambda) → fill MISSING `index.md` files
   deterministically (authored indexes travel and are preserved — the index
   regen never deletes) → `graph.json` → `ensure_dir_markers` over the
   written keys + the marker key (a marker-less directory mounts READ-ONLY —
   without this the next harvest dies EACCES) → freshness seeding → fresh
   `complete` marker → `IMPORT` row → release → policy-rebuild signal →
   staged zip + record deleted (consumed). Reindex needs nothing: the object
   events re-converge the vector index.

The synthesized marker carries `tables` (derived from the archive's top-level
`tables/` docs) and `table_versions` seeded from the LIVE Glue catalog for
exactly the tables the archive documents — apply also writes those tables'
`TABLE#…/VERSION` freshness rows (the item shape
`incremental.store.put_stored_version` owns), which is what stops the nightly
reconcile from seeing every imported table as drifted and re-authoring the
bundle the import just delivered. Catalog tables the archive does NOT
document stay unseeded, so the reconcile authors them; seeding is
best-effort (a Glue failure degrades to the re-author-everything behavior,
never a failed import). Also on the marker: `imported_by` (JWT identity),
`imported_files`, `imported_sha256`, and `imported_export` (the manifest's
`exported_at`, only when present). Imported computation docs land UNVERIFIED
until a human re-verifies.

## Policy checks (LLM-judge engine)

A per-dataset **policy document** (`policies.yaml`) — one individually
trackable, checkable policy per entry — authored from the wiki's reference
docs by a harvest-runtime agent, and enforced MID-TURN by the chat runtime's
**map-reduce fleets of LLM judges**, in two tracks selected per run from the
composer's Policy feature. Judge flags are advisory: they ride back into the
model's own context as hedged `<system-reminder>`s (the model decides the
correction) and surface in the UI as shield timeline steps; nothing blocks
or gates.

Underneath the judges sits a **deterministic tier** ("Deterministic policy
rules" below): computational policies may carry declarative `rules` that are
evaluated on the query's own syntax tree BEFORE the engine runs. It is
advisory like everything else — a proven violation rides back as an
unhedged proof-backed reminder (never a block) — and a policy it decides
never reaches the judges. Everything it cannot prove falls through to the
fleet unchanged.

Naming note: the `ar_` prefixes below (module `okf_aws.ar_policy`, the
registry attributes, `mode="ar_rules"`) are legacy naming from the retired v1
engine (Bedrock Guardrails Automated Reasoning), kept to avoid a data
migration. v2+ involves **no Bedrock control-plane resources and no region
limitation** — the judges run wherever the chat model runs.

**The document** (pure format: `okf_core.policy_doc`): a YAML mapping with a
top-level `policies` list; every entry has exactly `{id, type, condition,
action, source}` plus, on computational entries only, an optional `rules`
list (see "Deterministic policy rules"). `id` is `P`-prefixed, unique, and STABLE across edits (the
UI tracks policies over time; the author agent keeps surviving ids and never
reuses retired ones). `type` is exactly one of `computational` (the
violation is visible in a SQL query itself — additivity, grain/collapse,
fan-out joins, sentinel decoding; judged by the query-time check) or
`behavioural` (a process rule — ask-before-committing, refuse, require
scope; judged against the agent's steps). A hybrid (computational core +
disclosure duty) is tagged computational. `condition` = when the policy
applies, plain language over a turn's conduct; `action` = the checkable
obligation; `source` = the `references/….md` wiki page it traces to (the
author gate refuses dead pages). Count backstop `OKF_POLICY_MAX_RULES`
(default 60) and a 50k-char doc cap guard against enumeration pathology.
**Migration is the self-heal**: a pre-split document (no `type`) fails the
parse, which flags the row stale and re-authors it — no code path needed.

**Authoring lifecycle.** ALWAYS ON per dataset (the v1-era `ar_enrolled`
opt-in is retired; the only switch left is the deploy-wide
`OKF_POLICY_BUILD_ENABLED`). The triggers, not a flag, bound the work: the
runner's post-complete follow-on build authors on every committed bundle
change (full, incremental, annotation — AFTER the terminal status write; see
"Harvest status"), `policy_rebuild` events cover the rest (the
Reasoning page's manual Sync, the repromote accelerator — which also
first-authors a dataset restored to a version without a document — and the
chat check's stale discovery), and the nightly reconcile re-verifies ONLY
datasets whose lifecycle has begun (`ar_build_status` present). A dataset
predating the feature is therefore never bulk-backfilled: its first document
comes from a manual Sync, its next harvest/increment, or a repromote.
Authoring runs on the harvest runtime — as the runner's follow-on step
(best-effort, AFTER the commit marker AND the terminal status write) and on
`mode="ar_rules"` dispatches — as a small ReAct agent with full reasoning
(`harvest.ar_author`): it reads the sources (with per-file unified diffs
against the previous authoring's copies), then submits through a validating
`write_policies` tool. The pipeline is short: gather sources →
fingerprint-skip (unchanged sources + live document = zero model calls) →
flip `building` (THE per-dataset lease; conditional UpdateItem with a
`BUILD_LOCK_STALE_SECONDS` takeover so a dead author's row cannot lock
authoring out) → author → persist the document + author state → `stamp_ready`
with the **gather-time** fingerprint, then one freshness re-check (a wiki
that moved mid-authoring flags the row stale and queues a rebuild — never a
mislabelled document). Authoring IS
completion — there is no build workflow and no completion authority.

**From-scratch runs are map-reduce** (first authoring + forced Sync — any run
with no prior document). Measured 2026-08-03 (hifa): three single-pass
from-scratch runs over identical sources each produced a DIFFERENT incomplete
rule set — extraction recall is an attention problem, not a model-tier one.
So: `harvest.ar_clusters` splits the sources into deterministic topic
clusters (path + frontmatter-tag routing; ~8 files / ~25k tokens max, thin
topics fold into a catch-all, the confidentiality/disclosure topic and the
`usage_guardrails.md` contract stay ISOLATED however thin — isolation is the
remedy for the pages single-pass kept dropping); a fleet of per-cluster
extractors (classifier contract: thinking off / reasoning `"none"`, FORCED
`submit_rules` tool call, schema + source-attribution validation with
bounded retry, salvage-on-exhaustion) mines candidate rules in parallel;
and the authoring agent becomes the SYNTHESIZER — it receives the candidate
union, dedupes/merges/prunes (proportionality is its explicit job, backed by
the `OKF_POLICY_MAX_RULES` cap), verifies doubtful candidates via
`read_source`, and owns the single `write_policies` gate as before. The
gate adds a one-shot **coverage nudge** on full-mode submissions: a valid
document that leaves non-index sources uncited is staged AND answered with
the uncited list — one extract-or-affirm round trip. Everything is
fail-open (fleet failure → plain single-pass authoring); UPDATE runs never
fan out (minimal diffs + stable ids are exactly right there). Code-level
env: `OKF_POLICY_FANOUT` (default on; `false` kills the fleet),
`OKF_POLICY_EXTRACT_CONCURRENCY` (default `4`).

**The rebuild authority** (`incremental.ar_rebuild`, reached by
`policy_rebuild` events and the nightly reconcile) makes only
deterministic decisions: registration/bundle-ready/fingerprint checks,
re-dispatch on change/missing document, **reaping** rows stalled at `building` past a 1h
grace period, and a **deterministic recovery** — when the persisted document's
sources manifest fingerprint matches the live wiki but the ready stamp was
lost, it re-stamps instead of re-authoring. Dispatches always mint a FRESH
runtime session id (`runtime_session_id(..., unique_token=uuid)`): an
AgentCore session is pinned to the runtime *version* it started on, so a
deterministic id would reattach a post-deploy retry to pre-deploy code. Dedup
comes from the `building` flip, never from session affinity.

**Off-mount artifacts** — under a top-level prefix SIBLING to `okf/` (the
`benchmark/` precedent): outside the reindex rule's filter and unreachable by
any LLM role's file tools.

```
policy/<data_domain>/<dataset>/
├── policies.yaml          # the authored policy document (the judges' rubric
│                          #   and the Reasoning page's per-policy list)
├── rules_schema.json      # {version, databases: {db: {table: [columns]}}} —
│                          #   the QUALIFICATION schema for `rules:` blocks,
│                          #   snapshotted at authoring time from the harvest's
│                          #   .metadata/columns.tsv (the schema the wiki was
│                          #   authored against, NOT live Glue). Absent = the
│                          #   author gate refuses rules; the chat tier
│                          #   evaluates nothing and the judges carry the load
├── sources_manifest.json  # {fingerprint, files: {rel: sha256}} — what the
│                          #   author last SAW…
└── sources/<rel>          # …and its exact copies: the DIFF BASE. The next
                           #   authoring run hands the agent per-file unified
                           #   diffs against these, so an incremental/
                           #   annotation harvest yields a surgical EDIT of
                           #   policies.yaml (stable ids), not a rewrite
```

All derived and rebuildable from the bundle; deleting a dataset purges the
prefix.

**Per-dataset state lives on the registry `DATASET#` row** (`pk =
"DOMAIN#<data_domain>"`, `sk = "DATASET#<dataset>"` — the mapping row), as
flat scalars: `{ar_build_status, ar_source_hash, ar_pending_source_hash,
ar_build_started_at, ar_built_at, ar_build_detail}`.
Statuses: `building` (the lease) / `ready` / `failed` / `stale`. An absent
`ar_build_status` means the lifecycle has never begun — the reconcile skips
such rows (no bulk backfill) and the chat check's dataset discovery ignores
them. Retired attrs (`ar_enrolled`, the v1 Bedrock AR set) may linger on
rows written by older deploys; nothing reads them, and dataset deletion
removes the row itself.

**The document is a DERIVED artifact of the bundle, exactly like the vector
index.** S3 markdown is the only truth; a document authored from anything but
the CURRENT wiki state must never be judged against. Enforcement is by
**fingerprint, not by trusting event hooks**: the check recomputes the live
source hash (`okf_core.ar_sources` owns which files are policy material and
the fingerprint) and compares it to the stamped one. A mismatch renders NO
verdict — the dataset reports "rebuild pending", the row is flagged `stale`,
and a `policy_rebuild` event is published (both best-effort), so the click
starts the repair.

**The checks** (`chat/policy_check.py`, v3 — the post-turn panel,
`policy_check` request type, `POLICY#` report rows, and synthesizer are all
REMOVED; the shield timeline steps are the only policy surface):

- **Per-run opt-in.** The composer's "+" menu gains a **Policy** field with
  side options Computational / Behavioural / Strict (= both), carried on the
  run envelope as `features: ["sql", "policy:computational" |
  "policy:behavioural" | "policy:strict"]`. The Policy field REQUIRES the
  SQL feature — the UI disables it while SQL is off and unchecking SQL also
  unchecks Policy; the server enforces the same dependency independently
  (`chat.sql.normalize_features` drops orphaned `policy:*` values). The
  deploy flag `OKF_CHAT_POLICY_CHECK_ENABLED` remains the master gate above
  the opt-in. No selection ⇒ no checker constructed ⇒ zero cost.
- **One turn-scoped checker serves both tracks**
  (`policy_check.PolicyChecker`, built by the server per run — it needs the
  thread id): handed to the SQL tool (computational) and the behavioural
  middleware; shared state = the curated question, policy caches, budgets,
  an 8-worker pool with a lock over the shared caches (concurrent checks —
  parallel tool-called queries, a behavioural eval alongside — must not
  serialize while their callers' wait budgets burn). Fleet mechanics are
  shared: policies shard ≤
  `OKF_CHAT_POLICY_SHARD_SIZE` (default 10) per mini-judge, packed by
  SOURCE page (`shard_policies` keeps one wiki page's policies in one
  judge's shard — shared vocabulary/context; a group never straddles a
  boundary, so grouping can cost an extra shard); judges are CLASSIFIERS on
  EVERY family — thinking OFF + temperature 0 on a Converse (Anthropic) id,
  reasoning `"none"` on an openai.* id — with a FORCED `report_violations`
  tool choice (legal on Anthropic exactly because thinking is off), single
  fast pass. They answer
  through exactly ONE tool, `report_violations`, whose
  contract is **violated policy IDS ONLY** — no evidence, no explanations
  (saves judge output tokens, and the reminder the main agent reads is built
  from the AUTHORED policy text — condition/action/source — so judge prose
  never pollutes its context). Ids outside the judged set are dropped; a
  judge that won't call the tool after one retry fails open as a missing
  shard. **No second verification round on either track** — the flags ship
  hedged ("CAN include false positives — use your own judgment") and the
  receiving agent, holding the wiki pages and the live context, is the
  verifier.

**The rolling curated question** (both tracks judge against it): turn 1 is
the raw question at zero model cost — but it is still PERSISTED as the
curated question (the seed the next turn chains from). Curation runs on
EVERY gated turn, armed or not (the checker is built whenever the deploy
gate is on; an unarmed run's checker exists solely as the chain's keeper),
so enabling Policy mid-conversation chains from real state instead of
starting over. A later turn runs ONE minimal-effort
rewrite — (previous curated question, previous final answer, current raw
question) → curated question — kicked ASYNC at run start (the server's
`prewarm()`, right after any clarification fold) so it races the model's
own thinking and the queries; **evaluators never wait on it** — a verdict
fired before the rewrite lands judges against the raw question, and the
rewrite still persists in the background for the next turn. A previous
ANSWER alone (an armed turn
that ran no SQL never seeded the question) still triggers the rewrite. An answered `ask_human` clarification IS
folded in, inline within the turn (the resume rebuilds the agent; the server
hands the checkpoint's raw question + the Q&A to the fresh checker, which
re-runs the rewrite) — the ask-first evidence lives in the behavioural steps
track, and the computational judges need the clarified intent. Durability:
the state lives on the THREAD row (`policy_curated_question`, written when
the rewrite lands; `policy_last_answer`, ~2k chars, written at stream end;
`policy_question_history`, the last 3 curated questions rolled forward with
each landing), read with one lazy GetItem — so reloading a chat days later
still chains context. The history widens the REWRITE's resolution context:
the questions before the previous one (at most 2) render as an
earlier-questions section in the rewrite input, so a fragment jumping back
past the immediately previous topic still curates to the right question. The
judges never see them — both tracks judge against the single current curated
question. Every piece is best-effort and fail-open: absent attributes mean
raw-question (turn-1) semantics.

**The computational track** (the `run_sql` race):

- **Which dataset's policies**: an @-scoped run pins it — which covers every
  REDSHIFT run by construction (the Redshift engine can only be built from
  the scope's source descriptor). An UNSCOPED Athena run has no default
  database, so the model must schema-qualify every table — the checker reads
  the schemas in FROM/JOIN position (`extract_sql_schemas`; aliases, CTE
  names, string literals, and comments can't match) and resolves them
  against the policy-bearing datasets' Glue map (`ar_build_status` present;
  `glue_database` attr, dataset id fallback — same contract as scope
  enrichment; one filtered registry scan, cached per turn). A cross-dataset
  join is judged against each matched dataset; a query touching only
  policy-less data is skipped for free.
- **Race, don't block**: `run_sql` submits the check before dispatching the
  query (the judges spend the query's own execution time), then waits at most
  `OKF_CHAT_POLICY_QUERY_TIMEOUT_S` (default 60) AFTER the results are back —
  but ONLY for analytical queries (`should_wait`); exploration probes submit
  without waiting. The cold start is PREWARMED at run start (the server
  calls `checker.prewarm()` right after the clarification fold): each piece
  — judge-model build, the curated-question rewrite, the freshness gate /
  policy-dataset map — is its own parallel pool task behind its own
  once-guard, and **nothing ever waits on the rewrite**: evaluators fire
  the moment they are triggered and take the curated question only if it
  has already landed, else the turn's raw question (`_curated_now`). A
  verdict that isn't ready in time is dropped — advisory, never a
  bottleneck.
- **Exploration is free**: a deterministic structural gate
  (`is_analytical_sql`) judges only queries that compute something the
  policies govern (aggregates, windows, joins, unions). Row peeks, DISTINCT
  value enumeration, SHOW/EXPLAIN/DESCRIBE, and `information_schema` probes
  are never judged. Identical (whitespace-normalized) SQL is served from a
  per-turn cache, and at most `OKF_CHAT_POLICY_QUERY_MAX_PER_TURN` (default
  3) queries per turn are judged at all. Only the COMPUTATIONAL subset of
  the document is judged here.
- **Delivery**: flagged violations ride back in the SAME tool result as the
  rows, as a `<system-reminder>` after the payload (the `sql_anomalies`
  channel). The model decides the correction: revise the query, address it
  in the answer, or set a misread flag aside. For display, the server splits
  the reminder back OUT of the tool content (`split_policy_reminder`) into a
  typed `{"type": "policy"}` chunk — live and on history reload — which the
  UI renders as its own thinking-timeline step (shield marker, finding lines
  only); the model always sees the full string.

**The behavioural track** (`policy_check.BehaviouralPolicyMiddleware`, a
`before_model` hook in the same slot as `SteeringMiddleware`): `before_model`
runs exactly once after ALL parallel tool results return, so two analytical
queries fired in parallel produce ONE evaluation covering everything so far
— batching for free, no debounce. When new successful analytical `run_sql`
results exist since the last eval, the middleware kicks ONE eval over the
steps-so-far (the curated question, the ask_human Q&A verbatim, tool calls +
result summaries — never thinking blocks or injected notes), judged against
the BEHAVIOURAL subset with committed-conduct framing (computing on an
unresolved ambiguity is committed; not-having-asked-YET is not) — then
**WAITS for the verdict** (bounded by the same
`OKF_CHAT_POLICY_QUERY_TIMEOUT_S` budget; `asyncio.to_thread` on the async
path so the event loop never blocks) and injects it before THIS model call
as a `HumanMessage` carrying `additional_kwargs["okf_policy"]` (the steering
pattern — merged into the tool-result user message, never a cache
invalidation, never a user bubble on reload), surfacing as the same typed
`policy` chunk / shield step. The wait is the delivery guarantee: the model
step right after an analytical batch is usually the one that writes the
FINAL answer, so a fire-and-forget eval loses that race and its verdict
never reaches the model at all (observed live 2026-08-02). The checker is
prewarmed at run start (see "Race, don't block" above; the middleware's
first hook remains a second, idempotent caller), so the wait is normally
just the fleet. A verdict slower than the budget rolls
to the next hook, or drops at turn end (advisory, logged). One eval in
flight at a time; a policy flagged once this turn is never re-flagged (no
nagging).

**Shared invariants**: the freshness gate runs once per turn per dataset
(stale ⇒ never judged + `flag_stale` + `policy_rebuild` publish — the same
self-heal that migrates pre-v3 documents), mid-turn framing on both prompts
(the answer doesn't exist yet; answer-stage obligations are not violations),
the ids-only judge contract, and fail-open everywhere: any failure returns
the results/turn untouched.

**Control API routes** (Cognito-authed, same Lambda):

- `GET /reasoning/{domain}/{dataset}` — everything the Reasoning page renders:
  `wiki_ready`, status, `up_to_date` (the fingerprint gate for humans), the
  live source list, and the document's policies (id/condition/action/source).
- `POST /reasoning/{domain}/{dataset}/sync` — one FORCED `policy_rebuild`
  event (refused without a complete wiki). Sync is "author now": the event
  carries `force: true`, which bypasses the rebuild-iff-changed skip at BOTH
  hops (rebuild authority and harvest-side trigger) — a manual Sync's sources
  may be unchanged while the authoring itself moved on (model/effort/prompt),
  and without force a Sync on a ready dataset acknowledged "queued" then
  silently did nothing (live 2026-08-03). A forced run also authors FROM
  SCRATCH — the prior document is withheld (update mode's "minimally edit"
  would hand it straight back); automatic rebuilds keep update mode (minimal
  diffs, stable ids). Doubles as the manual FIRST
  authoring for a dataset predating the feature (there is no bulk backfill).
  The building lease still wins (an in-flight run is never preempted; the UI
  disables Sync while a build is live), and a stalled `building` row is
  reaped and re-dispatched.
- `GET /reasoning/{domain}/{dataset}/document` — `{exists, text}`: the live
  `policies.yaml` verbatim (kept for raw API access; the UI page renders the
  parsed per-guardrail list from the status call instead).

### Deterministic policy rules

A computational policy entry may carry an optional `rules:` list — the
mechanically checkable core of its `action`, evaluated on the agent's SQL by
`okf_core.policy_rules` (sqlglot) at `run_sql` call time. The evaluation is
SUBMITTED to the checker pool before the engine dispatches (its gate loads
race the query, never serialize in front of it) and JOINED after the results
return; the judge path waits on the in-flight evaluation so shard exclusions
stay deterministic.

**Tri-state contract.** Per rule: `violation` (proven — every implicated
column resolved to a real `(database, table)` through the schema and the
forbidden shape is present / the required one absent), `pass` (proven clean),
or `unknown` (anything unprovable: parse/qualify failure, a column flowing
through a CTE or derived source whose name overlaps the bindings, `OR` at the
top of a `WHERE`, a set-operation `ORDER BY`, no sqlglot). Renames don't
evade: same-scope aliases (`SUM(points) AS total`, `ds.points`) are resolved
away by qualification and still prove violations, while a derived-scope
projection that renames a bound column (`points AS pts` in a CTE, `SUM(pts)`
outside — including expression wrappers like `points + 0 AS pts`) is tracked
as an export alias and reads `unknown`, never a decisive pass. A policy is
`violation` if any of its rules
proves one, else `unknown` if any is undecidable, else `pass`. **The tier only
ever flags-with-proof or stays silent** — its worst case is coverage loss,
never a wrong flag.

**The closed dimension catalog** (`okf_core.policy_rules.DIMENSIONS` — the
author may only BIND these; an unknown dimension is a gate error and a policy
fitting none stays prose-only): `forbidden_aggregation`, `forbidden_usage`,
`forbidden_function`, `required_predicate`, `required_guard`,
`forbidden_grouping`,
`forbidden_sequencing_key`, `required_distinct`. Two evaluator families —
expression-local (one ancestor walk after qualification) and scope analysis
(conjuncts of `WHERE` + `HAVING` + INNER JOIN `ON`; outer-join `ON` never
satisfies a required predicate, since it does not filter the preserved side).
Matching is engine-faithful and equivalence-aware: literals compare the way
Athena compares them (numbers numerically, strings case-SENSITIVELY, booleans
case-insensitively), `NOT (col = v)` satisfies a `neq`, negative literals
resolve, function names match Trino spellings underscore-insensitively
(`date_diff` ⇒ sqlglot's `datediff`), cast types accept ANSI aliases
(integer/int, real/float, decimal/numeric — REAL is in the default guard
set), comparison conjuncts IMPLY `is_not_null` (NULL fails every
comparison, so `d = date(…)` satisfies a required not-null — no false flag
on bounded queries), `required_predicate` has a `bounded` op (any positive
equality/range/IN conjunct on the column, whatever the value — the
"must pin an explicit window" shape), `forbidden_grouping` fires on GROUP
BY keys only (ordinals and expressions over the target included; reading/
filtering stays legal, DISTINCT deliberately out of scope), `||` is
transparent to usage contexts (`MIN(time || 'x')` still
aggregates the text form), cast chains are walked whole (an inner `TRY_CAST`
never shields an outer forbidden cast, but IS the guard for
`required_guard`), a guard only counts as a positive conjunct or inside the
enclosing CASE/IF (an OR-branch or NOT-wrapped guard reads `unknown`), and
`COUNT(*)` triggers table-scoped `required_predicate` rules by table
presence when no `when_columns` narrows the trigger.

**Two semantics that look like bugs but are not.** (a) `forbidden_sequencing_key`
fires on a window `ORDER BY` and on `ORDER BY … LIMIT 1` ("the latest row") but
NOT on a plain ordered listing with any other LIMIT — the `run_sql` tool
description asks the model to LIMIT every query, so an any-LIMIT reading would
refuse ordinary paging. (b) `required_distinct` normally carries a
`when_filtered` trigger (the predicate that makes the reading wrong, e.g. the
winner predicate) and never fires on `COUNT(DISTINCT …)` of ANY column —
count-distinct is fan-out-safe by construction, and counting a different column
is a different, legitimate question.

**Every rule carries its own self-test** (`examples: {violation, pass}`) and
the author gate EXECUTES it at submit time: the violation example must trip
the rule and the pass example must not (`unknown` fails too), on top of the
schema-contract check that every bound table/column exists. A rule cannot
enter the document without proving it fires and does not over-fire — the same
move as qgen's submit-time gold execution.

**Advisory only — rules never refuse.** A proven violation rides back after
the results as one `<system-reminder>` per dataset (deterministic wording —
no false-positive hedge, unlike the judges' — with the policy's own `action`
as the fix-it). The query always executes; there is no verification
lifecycle, no overlay, and no blocking path. The trust anchor is entirely
the author gate's executed self-test (above).

**Cross-dataset queries** get the UNION of both datasets' rules. Bindings are
database-qualified via the sidecar, and a rule only fires on columns proven to
belong to ITS dataset's tables, so cross-talk is impossible by construction; a
table from an unregistered database contributes no schema, which makes bare
columns near it unresolvable and lands `unknown`.

**Judge-shard exclusion.** Policies the tier decided (violation or pass) for a
given normalized SQL are removed from that query's judge shards — the fleet
only ever sees `unknown` and rule-less policies, which also saves judge tokens
on every flagged-clean query. `run_computation` is deliberately UNTOUCHED: a
computation is already frozen, human-verified SQL with its own trust lane.

**Which statements it evaluates** (`is_checkable_sql`) is deliberately WIDER
than the judge track's `is_analytical_sql`: that gate requires an
aggregate/join/window/group-by because a fleet round costs tokens, while
several dimensions fire on plain single-table reads (an unguarded numeric
CAST, `ORDER BY <text time>`, `ORDER BY <opaque id> … LIMIT 1`, a sentinel
filter). Reusing the judge gate would disarm the tier for exactly those. Only
non-reading statements (SHOW/DESCRIBE/EXPLAIN — opaque `Command` nodes) and
`information_schema` probes are skipped, which also keeps this synchronous
path off the policy gate's I/O.

**Athena/Trino only in practice.** The sidecar's table names come from the
harvest snapshot, and rule targets are `table.column` (unqualified). A
Redshift-backed dataset addresses tables as `schema.table`, which no target can
express and whose 2-part references resolve as `db.table` — so on Redshift the
tier reads `unknown` everywhere and the judges carry the whole load. Fail-safe,
but a real coverage gap: schema-qualified bindings are the follow-on.

**Fail-open, everywhere.** No sqlglot, no sidecar, a raising evaluator:
every path yields `unknown`/no-op and today's judge-only behavior. A `rules`
block the runtime's own catalog can't parse (okf_core is vendored per
artifact, so a newer harvest image can author a dimension an older chat
image doesn't know) degrades THAT policy to prose
(`parse_policies(drop_invalid_rules=True)`) instead of silencing the whole
document for both tiers. The tier arms under exactly the conditions the
computational track already does (deploy gate + the run's
`policy:computational`/`policy:strict` opt-in) — no new feature flag.

**Tracing (live testing).** Every tier log line is prefixed `[policy-rules]`
(logger `chat.policy_check`, INFO+), so ONE CloudWatch filter on the chat
runtime log group yields the whole trace per query: the evaluated SQL +
dialect/default-db, one line per rule — `<label> P010 rule[0]
forbidden_aggregation(driverstandings.points) -> VIOLATION: SUM(…)` — the
per-policy outcome (`VIOLATION -> advisory reminder to the model` /
`PASS -> excluded from the judge shard` / `UNKNOWN -> deferred to the judge
fleet`), a
`done in Nms: X violation / Y pass / Z unknown; …` tally, and the judge-side
hand-off (`… already decided deterministically — N policies left for the
judges`). Every SKIP names its reason at WARNING/INFO — no rule-bearing
policies, missing `rules_schema.json` sidecar (with the fix: Guardrails →
Sync), a FAILED sidecar load (transient S3 error — named as such, never
misdiagnosed as "never authored"), sqlglot missing from the image — because
a silently inert tier is the
one failure mode an operator can't diagnose from behavior alone. The
`evaluate_policies` result also carries the per-rule breakdown (`rules:
[{index, label, dimension, verdict, detail}]`) for any future UI surface.

**The `policy_rebuild` event** (`okf_core.policy_rebuild`): custom source
`okf.policy`, detail-type `policy_rebuild`, detail
`{data_domain, dataset, reason?, force?}`. Rides the SAME EventBridge → SQS →
incremental-handler path as the Glue events. Published by the Control API
(manual sync / repromote) and the chat runtime (stale discovery). `force`
(manual sync ONLY — omitted otherwise) makes the rebuild unconditional;
automatic publishers never force, for them fingerprint equality genuinely
means nothing to do. Duplicates are harmless: the runtime's conditional `building` flip collapses
them. Events are a freshness ACCELERATOR; the fingerprint gate is
correctness.

## S3 Vectors (one bucket, one index)

See `okf_core.embedding`.

- 512 dims, cosine, float32. Non-filterable metadata keys are `title`,
  `description`, `s3_key`. These are immutable in S3 Vectors.
- Filterable metadata: `data_domain`, `dataset`, `table`, `type`, `tags`.
- Embed text and metadata come from `build_embed_text`,
  `build_filterable_metadata`, and `build_non_filterable_metadata`.
- Any query that filters or returns metadata needs both
  `s3vectors:QueryVectors` and `s3vectors:GetVectors`.

## DynamoDB tables

Four tables (plus the checkpoint table the LangGraph saver owns); names come from
env vars, with the defaults shown.

### `okf-registry` — domain registry, harvest status, credentials

Partition key `pk` (S), sort key `sk` (S). Item shapes:

**Declared domain.** `pk = "DOMAIN#<data_domain>"`, `sk = "META"`, attrs
`{data_domain, description, context, created_at, updated_at}`. A first-class,
operator-declared entity: domains must be declared before Glue databases can be
mapped into them. `description` is a short one-liner; `context` is richer prose
(used in the harvest prompt and exposed to agents over MCP). Listing
(`GET /domain-defs`): scans `pk begins_with "DOMAIN#"` with `sk = "META"`.
Deletion (`DELETE /domain-defs/{domain}`) is blocked (409) while `DATASET#`
mappings still exist under the same partition. On declare/update, a derived
concept doc is written through the harvest mount at
`okf/<domain>/_domain/overview.md` (see S3 layout below) so the domain is
embedded and semantically searchable.

**Domain mapping.** `pk = "DOMAIN#<data_domain>"`, `sk = "DATASET#<dataset>"`,
attrs `{data_domain, dataset, source, glue_database, created_at}` plus optional
dataset-guidance attrs `{guidance, guidance_updated_at, guidance_applied_version}`
(shared authoring instructions; see `okf_core.guidance` + the harvest payload's
`dataset_guidance` above) plus the optional **policy-check** attrs
(`ar_build_status`, `ar_source_hash`, `ar_pending_source_hash`,
`ar_build_started_at`, `ar_built_at`, `ar_build_detail` — the build state and
lease described under "Policy checks (LLM-judge engine)" above; retired
attrs like `ar_enrolled` or the v1-era `ar_policy_arn`/`ar_guardrail_id` may
linger on legacy rows — nothing reads them, and deleting the dataset deletes
the row).
Requires a pre-existing `META` row for the same `pk` (enforced by
`assert_domain_declared` in the upsert adapter).

A legacy `recursive_improvement` map may still sit on old `DATASET#` rows; it is
**dead** — the in-harvest recursive-improvement loop is retired (harvests never
benchmark; see the Benchmark Studio sections below) and nothing reads or writes
the attribute anymore.

`source` is the first-class, future-extensible **source descriptor** — a nested
map `{type, ...type-specific config}` naming WHERE the dataset's data lives and
how the harvester reads it. The vocabulary lives in `okf_core.sources`
(`SUPPORTED_SOURCE_TYPES`, `DEFAULT_SOURCE_TYPE`). Supported types:

- **glue** — `{"type": "glue", "glue_database": "<db>"}`.
- **redshift** — `{"type": "redshift", "redshift_database": "<db>"}` plus
  **self-describing** connection routing: `cluster_identifier` OR `workgroup_name`
  (exactly one) + `secret_arn` (the Secrets Manager secret that authenticates to
  it). The operator picks these in the UI, so any cluster/workgroup in the account
  is harvestable with **no deploy-time connection config** — the harvest reads the
  connection entirely from the descriptor. Registration REQUIRES the full
  connection (target + secret; a `400` otherwise) — a db-only descriptor can't be
  harvested, so it's rejected at the boundary rather than failing deep in the
  async run. (`normalize_source` still tolerates a stored db-only row when
  READING, so legacy rows never break readers.) The secret must hold a
  **read-only DB user** and be named with the deployment's secret prefix
  (`var.redshift_secret_name_prefix`, default `okf-`) — the IAM grants are scoped
  to that name pattern. See `docs/DATA_SOURCES.md`.

Config keys are stored generically on the item, so a new source type (BigQuery, …)
adds a type + config keys with no item-schema migration. For a **glue** source the
flat top-level `glue_database` attribute is **also written** as a back-compat
mirror: the harvest invocation payload and the incremental scan
(`incremental/store.py`, which filters on `glue_database`) read it directly. A
non-glue source writes no such mirror, which is what (correctly) scopes the
`aws.glue`-event incremental path to Glue datasets. Readers go through
`okf_core.normalize_source`, which reconciles the nested shape and pre-`source`
rows (flat `glue_database` only) into one `{type, ...config}` dict. The Control API
validates on write (`PUT /domains/{domain}/datasets/{dataset}` accepts either a
`source` object or a bare `glue_database`), rejects an unsupported type with `400`,
and applies per-source registration rules (`assert_source_registrable`): a **glue**
dataset name must equal its `glue_database` and the database must exist; a
**redshift** dataset name is independent of `redshift_database`, must carry the
full connection (target + secret, see above), and gets no live existence probe
(the harvest verifies the connection when it first runs).

The UI's mapping dialog fills a Redshift descriptor dynamically: `GET
/redshift/clusters` lists provisioned clusters + Serverless workgroups
(control-plane, no DB connection), and `GET
/redshift/databases?cluster=…|workgroup=…&secret_arn=…[&database=…]` lists
databases within a chosen target via the Redshift Data API (`ListDatabases`, which
connects, hence the secret; `database` is the bootstrap DB to connect through — a
provisioned cluster's `DBName` hint from `/redshift/clusters` — defaulting to
`dev`).

**Harvest status.** `pk = "HARVEST#<data_domain>#<dataset>"`, `sk = "STATUS"`,
attrs `{status: queued | running | complete | failed | cancelled, mode,
started_at, updated_at, detail, runtime_session_id, model, effort}`. The
harvest is DONE at the bundle commit marker: the runner flips the row
terminal right after `finalize_bundle` returns, and the policy document then
authors as its own follow-on step (still in the same runtime session, but a
separate process semantically). The terminal `complete`/`failed` writes are
conditional on the row still being in flight AND still being THIS run's
(`runtime_session_id` must match when the runner knows its own — a hung
run's late write must not clobber a successor that took the lease via the
staleness escape, nor trigger a ghost policy build against its mid-write
bundle). The session pin alone can't protect the INCREMENTAL path, whose
session id is deliberately deterministic per dataset (two successive
incremental runs share it) — so the runner also pins on `started_at`
(`status.read_run_identity`): every lease acquire rewrites it and no later
write touches it, making it a per-run identity the condition checks too. That step runs only when the terminal
write actually landed (`report_status` returns False when a cancel won the
race — no authoring for a cancelled run), is SKIPPED outright when the
policy feature is off (`OKF_POLICY_BUILD_ENABLED` unset — no flush wait for
a no-op), first WAITS for the bundle's S3
flush to settle (the S3 Files mount's write-back can lag the terminal write
by a minute — gathering early fingerprints a partial wiki and the fresh
document immediately reads "out of date"; the wait polls for the commit
marker plus a settle margin, bounded, and matches the marker's
`completed_at` against the one THIS run's finalize wrote — on a re-harvest
the PREVIOUS run's `complete` marker is still the visible object until the
mount flushes), and is serialized by ITS OWN lock —
the mapping row's `ar_build_status = "building"` flip with its
`ar_build_started_at` stamp (`okf_aws.ar_policy.build_lock_active`; staleness
escape `BUILD_LOCK_STALE_SECONDS` = 1h, honored symmetrically by the flip
itself, so an abandoned `building` row neither wedges harvests nor locks
authoring out — the reconcile still reaps it). The gate lives INSIDE the
Control API's lease acquirers (`acquire_harvest_lease` /
`acquire_repromote_lease` — every trigger path gets it for free) and answers
with `409` ("guardrails are being authored…"); dataset deletion, which takes
no lease, calls it directly; the incremental orchestrator checks the shared
helper and returns `skipped_guardrails_building` without recording the new
version (the change is re-detected once the build lands). The REVERSE gate
also holds: the Reasoning page's Sync refuses (`409`) while the harvest
lease is held — the finished harvest authors on its own (the lease-held
mirror honors BOTH staleness escapes below, so a dead repromote frees Sync
after 120s, not 8h). The Reasoning status GET reports `wiki_rewriting` only
while the marker reads `in_progress` AND that lease is live — a failed run
leaves the marker at `in_progress` forever, and without the cross-check the
page would promise an auto re-author that is never coming (it reports "the
last harvest did not complete" instead, with no freshness verdict). A follow-on build
that loses the flip race leaves a `policy_rebuild` trigger behind, and every
authoring run ends with a freshness re-check (a wiki that moved mid-build is
flagged stale + re-queued), so the gate stays a courtesy, never a
correctness requirement. The
status GET exposes the live lock as a top-level `guardrails_building`
boolean — knowably false (and not re-read) while the row itself is still
`queued`/`running`, since the build only starts post-terminal and an active
build would have refused the lease. A
`mode = "cross"` row additionally carries `cross_target`
(`"<domain>/<dataset>"`, stamped at lease time — the counterpart the discovery
run is against, surfaced by the status GET so the UI shows WHO, not just the
mode; mirrors the repromote rows' `repromote_target`). Mode strings are wire
values — the UI maps them to display labels (e.g. `cross` renders as
"Cross-dataset discovery", `annotated` as "Apply annotations"). `model` and
`effort` record the RESOLVED LLM config the run actually used (override or
deploy-time default); the runtime stamps them on the `running` transition
(`harvest.status.report_status`), so they're empty on a still-`queued` row.
`cancelled` is a terminal
status set by the Control API's `cancel_harvest` (`POST
/harvest/{domain}/{dataset}/cancel`): it `StopRuntimeSession`s the
`runtime_session_id` and flips the row with a conditional update
(`status IN (queued, running)`) so it never clobbers a `complete`/`failed` the
runner wrote first. Being terminal, it satisfies the lease-free predicate below,
so a retrigger is immediately allowed.

This row also serves as a per-dataset harvest lease. Every path that starts a
harvest — the Control API's `trigger_harvest` and the incremental orchestrator /
nightly reconcile's `process_event` — acquires the lease with a conditional
`PutItem` before invoking the runtime:

```
attribute_not_exists(pk) OR NOT (status IN (queued, running)) OR started_at < <now − 8h>
```

If a harvest for the dataset is already in flight, the second one is refused: the
Control API returns `409`, and the incremental path returns `skipped_locked`
without recording the new Glue version, so the change is picked up again by the
next event or the nightly reconcile. This keeps two runs from writing the same
bundle directory at once (one run's `clean_authored_output` deleting files while
the other writes them). A lease older than 8 hours
(`HARVEST_LEASE_STALE_SECONDS`, the AgentCore session cap) can be taken over, so
a dead job whose final status write was lost doesn't wedge the dataset forever. A
failed invoke marks the row `failed` to release the lease.

**Annotations: unanchored + agent-submitted.** `quote` is OPTIONAL on an
annotation item: empty = an UNANCHORED note (page-level general feedback),
which the orphan sweep resolves only if its doc is gone. `concept_id` may be
the `_dataset` sentinel (underscore-pseudo, like `_domain`) for DATASET-level
feedback — never orphaned while the dataset exists. `submitted_via` records
provenance: `"ui"` (default) or `"agent"` — the chat agent's per-run
`submit_annotation` tool files on the user's behalf (the run's verified sub
keys the partition; chat role has PutItem-only on the annotations table).

A **cross-dataset run** (`mode = "cross"`) takes only ITS OWN dataset's lease —
the target is read via a start-time snapshot and never written, so no lease is
ever taken on it and no cross-bundle write window exists. Concurrent X→Y and
Y→X cross runs are therefore independent by construction (each harvests its own
dataset). The initiating bundle's fresh `complete` marker carries
`cross_target: "<d>/<ds>"` provenance.

A **repromote** (bundle version restore, below) takes this SAME lease with
`mode = "repromote"` and rides the existing `queued → complete | failed`
lifecycle — no new status value. A **bundle import** ("Bundle export & import"
below) does the same with `mode = "import"`. EVERY acquirer — the Control
API's harvest and repromote/import acquires AND the incremental
orchestrator's twin — carries one extra takeover clause for these two modes:

```
OR (mode IN ("repromote", "import") AND status = queued AND started_at < <now − 120s>)
```

Both run synchronously inside the 30s-capped Control API Lambda, so such a row
still `queued` after 120s is provably dead, and anything (a retry, a harvest
start, an incremental event) may take it over immediately instead of waiting
out the 8h harvest staleness — harvest rows are unaffected. The mode list and
threshold have ONE owner, `okf_core.session.SYNC_LEASE_MODES` /
`SYNC_LEASE_STALE_SECONDS`, shared by all acquirers and the read-side
`harvest_lease_held` mirror. The repromote row also carries
`repromote_target` (the marker VersionId being restored) so the status GET's
`stalled_lease` answer can offer one-click retry; a stalled IMPORT row is
reported on the same GET with `can_retry: false` (its retry is re-applying
the import, not re-POSTing a repromote).

**Repromote convergence manifest.** `pk = "HARVEST#<data_domain>#<dataset>"`,
`sk = "REPROMOTE"`, attrs `{started_at, completed_at, target_version_id,
new_version_id, requested_by, copied: [vector_key...], deleted: [vector_key...],
total}` — written once per repromote (overwriting the previous one) after the S3
writes land. It exists because deleted keys are unlistable after the fact: the
convergence check needs the exact touched-key set captured at write time.
`GET /bundle/{d}/{ds}/repromote` reports a key converged when its `VEC#<key>`
freshness row's `updated_at` (which reindex advances only AFTER the vector work
succeeds) is `>= started_at − 2s`; the UI declares a repromote done only when
every key converged — matching the definition that *current is what the vector
index serves*.

**Import provenance row.** Same shape on `sk = "IMPORT"` for a bundle import
(`{started_at, completed_at, archive_sha256, new_version_id, requested_by,
copied, deleted, total}`), written once per apply. No convergence poll reads
it yet — it captures the touched-key set at write time (the same
deleted-keys-are-unlistable reason) so one can. Dataset deletion removes the
`REPROMOTE` and `IMPORT` rows along with `STATUS` (the ghost-row rule: a
re-registered same-named dataset must not inherit a previous owner's
convergence manifests).

**Harvest live step feed.** Separate from the coarse status row, the harvest
runtime narrates its progress at message granularity. As the agent runs, a
LangChain callback (`harvest.steps.StepEmitter`, attached via
`config["callbacks"]` so it also observes every sub-agent) emits one stdout line
per step: `OKF_STEP <json>` where the JSON is
`{ts, data_domain, dataset, session_id, seq, kind, label, agent, tool?, ok?, error?, full?, result?}`.
`kind` ∈ `agent | tool_call | tool_result | subagent | usage`; `seq` is a 1-based
monotonic counter; `label` is a human phrase (tool calls are shaped, e.g.
"Reading `tables/races`", "Started `table-author`: …") — tool RESPONSE bodies are
never emitted, only success/failure, with THREE exceptions, each with its own bound (~500 chars for the error snippet, ~8KB for the lint report and agent `full`, ~64KB for sub-agent dispatch I/O): a FAILED `tool_result`
(`ok: false`) carries **`error`**, a whitespace-collapsed snippet of the failure
text (bounded ~500 chars). That text exists nowhere else (it goes back to the
model, not the logs), so without it a failed call — e.g. a provider 400 killing
a sub-agent — is undiagnosable after the run. And a SUCCESSFUL `lint_bundle`
`tool_result` carries **`lint`**, the lint gate's structured report
`{ok, errors, warnings, steps, findings, hidden?, note?}` (bounded ~8KB — the
emitter drops tail findings past the budget into a `hidden` count; error/warning
totals come from the per-step counters so they survive truncation). The UI
badges the feed row with the counts and opens the findings in a modal on click.
And a SUB-AGENT DISPATCH carries its I/O (bounded ~64KB each, not 8 — a
table-author brief carries its context-digest slice and routinely exceeds 8KB;
each event is one CloudWatch line, hard limit 256KB): the `task` tool_call (and a `subagent`
start event) carries **`full`** — the complete dispatch brief, where the label
keeps only a teaser — and its successful `tool_result` carries **`result`**, the
sub-agent's FINAL answer. QuickJS `task()` squares get the same two texts via
`subagent` events with `phase: "update"` (a mid-flight patch carrying `full`
right after start and `result` right before complete, correlated by `sub_id`):
the library's own lifecycle events truncate the brief to 200 chars and never
carry the answer, so `harvest.subagent_io` shims the dispatch choke point to
emit them. Never the sub-agent's internal steps — those stay
filtered. The UI opens both in the fleet square's drill-in sheet (Output/Input
tabs). An `agent` event also carries **`full`**
(the complete markdown of the AIMessage, whitespace preserved, bounded ~8KB) when
it exceeds the one-line `label`; the UI renders `label` as inline markdown and
opens `full` in a modal on click. `tool_call`/`tool_result` share a `call_id`
so the UI folds them into one row. **`subagent`** events power the UI's fleet
squares (the dynamic reviewer/table-author fan-out): they carry
`{phase: start|complete|error|update, batch, sub_id, subagent_type?, error?, full?, result?}` (an
`error`-phase event carries the same bounded `error` snippet — the
langchain_quickjs `SubagentErrorEvent`'s failure string) where `batch` is
the top-level `eval` tool-call id grouping one fan-out wave (NOT the event's own
`eval_id`, a REPL-local counter that resets to `call_0` on every `eval()` and so
can't tell one wave from the next — the emitter correlates each sub-agent to the
current top-level `eval` call_id) and `sub_id` is the per-dispatch id.
They come from `langchain_quickjs`'s custom stream (the run loop uses
`.stream(stream_mode=["custom"], subgraphs=True)`, since `.invoke()` drops these
into a no-op writer). The UI grows a row of squares as sub-agents START (there is
no reliable pre-start count — the model builds the fan-out list dynamically).
**`usage`** events carry a `usage` object with the **cumulative** token counts for
the whole run — `{input, output, cache_read, cache_write, total}` (`total` =
input+output) — accumulated across EVERY model turn including sub-agents (they
emit no feed row but dominate the spend). Fields mirror LangChain's normalized
`usage_metadata` (`cache_write` is its `cache_creation`, the Anthropic
prompt-cache WRITE; `cache_read` is a cache HIT). **`input` is the FULL input
count and already INCLUDES `cache_read` + `cache_write`** (per langchain_aws
`_extract_usage_metadata`, which sums `bedrock_input + cacheRead + cacheWrite`
into `input_tokens`), so `total` = `input` + `output` and cache is a *breakdown*
of input, never additive — the UI shows cache read/write as indented "of which"
children under Input, not sibling rows (listing them alongside double-counts). Counts are absolute, so the UI
renders the latest snapshot as a running total (a missed/re-ordered poll can't
corrupt it) and shows no feed row for the event. Metering is wired differently
from the other kinds: it rides a `UsageForwarder` callback on the **shared model
instance** (`build_harvest_agent(step_emitter=…)` → `_build_model(callbacks=…)`),
NOT the run-config `StepEmitter`. This is deliberate — QuickJS `task()` sub-agents
run on their own asyncio tasks and never reach the parent run's callbacks, but
they invoke the same inherited model, so only a model-instance callback sees
every turn. (`on_llm_end` on the run-config emitter must NOT meter, or sub-agent
turns are undercounted and supervisor turns double-counted.)
AgentCore ships stdout to the runtime's CloudWatch log group, so
this reuses existing storage (no new event store). The Control API's
`GET /harvest/{domain}/{dataset}/events?since=<seq>&since_ts=<ms>` reads it back
with `FilterLogEvents`, correlating by the run's `runtime_session_id` (on the
STATUS row), and returns `{events, next, next_ts, done}` (`done` once the status
is terminal). Two cursors the UI echoes back: `since`/`next` is the `seq`
high-water mark (exact dedup); `since_ts`/`next_ts` is the highest CloudWatch
event timestamp (ms), which bounds `FilterLogEvents`' `startTime` so each live
poll scans only a recent window instead of the whole run. On first load
(`since_ts=0`) the floor is the run's `started_at`, so a viewer who opens the
page mid-run backfills the whole current run. `OKF_STEP` is a frozen marker
shared by `harvest.steps` and `control_api.handlers`.

**Benchmark report index (Benchmark Studio).** `pk =
"HARVEST#<data_domain>#<dataset>"`, `sk = "REPORT#<report_id>"` — one row per
standalone benchmark run (`okf_core.benchmark_report` owns the shapes; the
retired RI loop's `BENCH#` rows have no successor and old ones are ignorable).
Report ids are time-prefixed (`r<UTC compact>-<hex>`, charset-locked by
`is_valid_report_id`) so the sk RANGE ordering is chronological — the report
list is one `Query begins_with(sk, "REPORT#")`, `ScanIndexForward=False`, no
GSI. **Flat scalars only** (structure lives in the S3 report JSON): `status`
(`queued → running → complete | failed`), `created_at`/`started_at`/
`completed_at`/`updated_at`, `detail` (failure reason), config summary
(`checks` as a CSV string, `runs`, `solver_model`/`solver_effort`,
`judge_model`/`judge_effort`, `version_id`, `question_count`, `count_<check>`),
`runtime_session_id`, `requested_by`, live progress stamps (`phase`,
`progress_check`, `progress_run`, `total_runs`, `progress_current`,
`progress_total` — throttled UpdateItems from the runtime; the Benchmark list
POLLS rows for live progress, there is no benchmark CloudWatch feed), headline
KPIs once complete (`<check>_raw`, `<check>_adjusted`, `<check>_graded`,
`total_tokens`, `annotation_candidates`), and the annotation-aggregation
sub-lifecycle (`agg_status`: `idle | running | complete | failed`,
`agg_detail` — the aggregation's OWN failure reason, separate from the run's
`detail` so an agg failure never clobbers the run's; cleared on every agg
start — and `annotation_final_count`). The Control API writes the QUEUED row
(conditional PutItem) and invokes the runtime; the runtime owns everything
after via UpdateItem — terminal statuses (`complete`/`failed`, agg included)
retry a transient DynamoDB fault a few times, since the terminal write has no
later write to correct it. Starting an aggregation flips `agg_status` with a
CONDITIONAL UpdateItem ("not already running unless the `updated_at` heartbeat
is past the 8h stale cutoff" — the same escape as report deletion), so a dead
aggregator is retryable and two concurrent POSTs can't both start one. **No lease semantics**: benchmark runs never touch the `STATUS`
row — they write nothing to the bundle, so they run concurrently with harvests
and with each other. Rows persist until the user deletes the report (no TTL).

**Cross-dataset reference signal (derived).** `pk = "DOMAIN#<target_domain>"`,
`sk = "XREF#<target_dataset>#<source_domain>#<source_dataset>"`, attrs
`{target_data_domain, target_dataset, source_data_domain, source_dataset,
updated_at}`. One row per documented PAIR, recording that
`<source>`'s bundle holds `external/<target_domain>/<target_dataset>/…` docs.

**Derived, never authored.** The reindex worker maintains these rows from the
bundle's S3 object events — the same events that drive the vector index (see
`reindex.handler._upsert_xref` / `_clear_xref_if_pair_empty`): a concept doc
under `okf/<sd>/<sds>/external/<td>/<tds>/` upserts the row; a delete whose
pair prefix no longer holds ANY concept doc (checked with a live listing, so
out-of-order events self-correct) removes it — with a CONDITIONAL delete
(`updated_at` older than the listing's start), so a concurrent worker's fresh
upsert for newly authored docs can never be erased by a stalled delete-path
worker. Pair components that fail OKF segment validation (e.g. a `#`, which
would collide two pairs onto one sort key) produce no row at all. Because it is event-derived it
survives full-harvest wipes and repromotes with no writer having to remember it,
and it is rebuildable by replay — the same "S3 markdown is truth, everything
else is derived" rule as the vectors. Whether the pair prefix is empty is judged
by `parse_bundle_key`, so a leftover generated `index.md` does not keep a row
alive.

**Why it exists.** Pair docs live only in the initiating bundle, so a consumer
scoped to the referenced dataset would otherwise never learn the relationship
exists. `list_domains` (both the Control API's `GET /domains` and the
consumption MCP tool) reads these rows alongside the mapping listing (one
`entity="xref"` Query on the entity index — see "Registry entity index"
below) and adds two optional fields per dataset:
`cross_references` (datasets this one holds pair docs FOR) and
`cross_referenced_by` (datasets whose bundle holds pair docs about this one —
read them under `<that dataset>/external/<this_domain>/<this_dataset>/`). Both
are omitted when empty. The reindex role therefore holds `PutItem` +
`DeleteItem` on the registry table (and `ListBucket` on the bundle bucket).

**MCP credential.** `pk = "CRED#<client_id>"`, `sk = "META"`, attrs
`{name, client_id, created_by?, created_at}`. Metadata only — the client secret
is returned once at creation and never stored. This backs the credentials UI
(list and revoke); the credential itself is a Cognito M2M app client.
`created_by` is the owner, stamped from the caller's verified JWT identity
(`email`, falling back to `sub`), not the request body. Revoking
(`DELETE /credentials/{client_id}`) requires a matching `CRED#` row — so an
arbitrary app client, such as the public SPA login client, can't be deleted — and
when a caller identity is present it must equal `created_by`.

**Registry entity index.** Mapping rows, declared-domain `META` rows, and
`XREF#` rows carry two extra attributes — `entity` (`"dataset"` | `"domain"`
| `"xref"`) and `pair` (`"<domain>/<dataset>"`, or `"<domain>"` for META) —
keying the `by-entity` GSI (projection ALL). Listings **Query** it instead of
Scan-with-filter: a Scan reads the whole table (harvest status + `REPORT#`
rows included), so its cost grows with usage rather than with the dataset
count. Every writer stamps the attributes (`upsert_domain_mapping`,
`declare_domain`, reindex's XREF upsert — all via the `okf_aws.registry_entity`
vocabulary); rows written before the index existed need
`scripts/backfill_registry_entity.py` (deploy.sh runs it at the END of the
compute stage — after the writers that stamp the attributes are deployed,
so no row can land unstamped once the marker exists — idempotent), whose
LAST step writes the readiness
marker row `pk = "REGISTRY"`, `sk = "ENTITY_INDEX_READY"`. Readers Query the
index ONLY once that marker exists (`okf_aws.registry_entity` — the one
shared read protocol) and use the legacy filtered Scan until then: a GSI
only contains rows stamped with its keys, so on a partially-stamped registry
any result-shape heuristic would return the fresh rows and silently hide
every pre-index dataset. The only OTHER condition that may fall back to the
Scan is "the index does not exist" (marker stamped before the terraform
apply); mid-pagination errors must surface instead — a silent Scan there
would re-return pages the caller already consumed. Consumers: the Control
API's `list_domains`, the consumption MCP `list_domains`, and the chat
policy check's `_policy_glue_map`.

Listing: the consumption MCP `list_domains(domain?, query?, cursor?, limit?)`
is PAGINATED — it returns `{"datasets": [...], "next_cursor": ...}` with a
soft `limit` (default 100, max 500; DDB requests are Limit-bounded so tiny
rows can't defeat the cap), an opaque base64 cursor, a `domain` partition
filter (base-table Query) and a case-insensitive `query` substring filter
over `"<domain>/<dataset>"`; the legacy-scan fallback returns everything with
`next_cursor = null`. `list_declared_domains` scans with `sk = "META"`;
`list_credentials` scans `pk begins_with "CRED#"`.

### `okf-chat` — conversation index (+ the policy checks' rolling context)

Partition key `pk` (S), sort key `sk` (S). Isolation is **structural** — the
caller's Cognito `sub` is baked into the pk, so a user's Query can only ever
return their own rows (the same argument as `okf-annotations`). Key shapes live
in `okf_core.chat_threads`.

**Conversation.** `pk = "CHAT#<user_sub>"`, `sk = "THREAD#<thread_id>"`, attrs
`{title, model, effort, data_domain?, dataset?, created_at, updated_at,
expires_at?}`. `thread_id` is the CLIENT-FACING id the browser sends, not the
`<sub>:<thread_id>` checkpoint-namespaced form. Written best-effort by the chat
runtime on each turn (`chat.threads.touch_thread` — `created_at`/`title` via
`if_not_exists` so a UI rename survives); read/renamed/deleted by the Control
API. `expires_at` (epoch seconds, `DELETED_TTL_SECONDS` = 1 day) is set ONLY on
delete, so an active conversation never expires; the list also skips any row that
already carries one, because TTL is eventually consistent.

The THREAD row also carries the policy checks' **rolling context** (see
"Policy checks" above): `policy_curated_question` (the latest turn's curated
standalone question, written when the rewrite lands and overwritten inline on
an ask_human fold), `policy_last_answer` (that turn's final answer, ~2k
chars, written at stream end), `policy_question_history` (a DynamoDB List
of the last 3 curated questions, most recent last, rolled forward with each
landing — the entries before the previous question feed the next rewrite as
earlier-questions resolution context), and `policy_history_last_raw` (the
RAW question whose curated form is the history's last entry — the durable
same-turn signal an ask_human fold's re-curation uses to REPLACE that entry
instead of appending; cleared to `""` by the stream-end answer write so it
never outlives its turn, which is what lets an IDENTICAL question re-asked
on a later turn append normally). All optional, best-effort, no TTL
(thread rows have none) —
reloading a chat days later still chains context; absent attrs mean
raw-question semantics with no background (`chat.threads.read_policy_state` /
`write_policy_state`).

**Memory pause blob.** THREAD rows also carry an optional `memory_pending`
attr (S, JSON `{obs, qa}`): the long-term-memory context of a turn paused on
`ask_human` — the harness observation so far plus the clarification rounds
already folded. Written at every pause (each pause overwrites), read by the
resume (`chat.threads.write_memory_pending` / `read_memory_pending`), never
cleared — the next pause's overwrite is the freshness contract. Best-effort
like all memory touchpoints.

**Memory datasets ledger.** THREAD rows also carry an optional
`memory_datasets` attr (S, JSON list, capped at 32): every dataset the
harness has observed a tool touch in this thread, merged at each turn's
memory-event write (`chat.threads.read_memory_datasets` /
`merge_memory_datasets`). It is the citation-validation set: a later no-tool
turn's answer citation counts toward the annotation's `datasets-cited` line
only if its dataset appears here, was observed this turn, or is the
conversation pin. Advisory and best-effort.

**Memory settings.** `pk = "CHAT#<user_sub>"`, `sk = "SETTINGS#memory"`
(`okf_core.chat_threads.MEMORY_SETTINGS_SK`), attr
`memory_enabled` (BOOL). The per-user switch for long-term memory: **missing
row (or attr) = the deploy default** (`OKF_CHAT_MEMORY_DEFAULT_ON`, from
`var.chat_memory_default_on` — true means opt-out/default-on, false means
opt-in/default-off; flipping the var never touches rows users already set).
Written by the Control API (`PUT /memory/settings`, the Memory page's master
switch); read by the chat runtime at turn start
(`chat.memory.ChatMemory.user_enabled`, which also treats a FAILED read as
the deploy default — an unreadable row must not silently flip a user's
memory in either direction). Off = the runtime neither recalls nor writes events; existing
memory records are untouched — though the server still STRIPS the previous
turn's recall injection from ongoing threads, so switching off silences
already-injected context too. Because this row shares the `CHAT#<sub>`
partition, the conversation list Query constrains
`begins_with(sk, "THREAD#")` (an unconstrained Query would surface it as a
phantom conversation).

**Legacy: policy-check reports.** v2's post-turn panel persisted one
`sk = "POLICY#<thread_id>#<turn_key>"` row per checked turn. v3 removed the
panel — these rows are **no longer written or read**; existing ones are inert
(invisible to the list, which constrains `begins_with(sk, "THREAD#")`). The
`POLICY#` key helpers stay in `okf_core.chat_threads` so any cleanup tooling
can still address them.

`okf-chat-checkpoints` is a fifth table owned entirely by
`langgraph-checkpoint-aws`'s `DynamoDBSaver`: `PK` (S, HASH) + `SK` (S, RANGE) —
UPPERCASE — TTL attribute `ttl` (lowercase, written only when the saver is
constructed with `ttl_seconds`), no GSI, with checkpoints and pending writes
distinguished by PK prefix (`CHECKPOINT_<thread>` vs
`WRITES_<thread>#<ns>#<ckpt>`). The chat runtime namespaces the thread id with
the caller's sub before it reaches the saver.

### `okf-freshness` — reindex and incremental dedup state

Partition key `pk` (S), sort key `sk` (S). Item shapes:

**Reindex dedup.** `pk = "VEC#<vector_key>"`, `sk = "SEQ"`, attrs
`{last_sequencer, updated_at}`. S3 `object.sequencer` values compare
lexicographically per key, so an event at or below `last_sequencer` is a
duplicate or replay and is ignored. `last_sequencer` is advanced (conditional
`PutItem`) only after the embed and `PutVectors`/`DeleteVectors` succeed, never
before — otherwise a transient failure would leave the marker ahead of the work,
and the SQS retry would skip the record as a duplicate and silently drop the
vector.

**Table version.** `pk = "TABLE#<data_domain>#<dataset>#<table>"`,
`sk = "VERSION"`, attrs `{version_id, update_time, last_seen_at}`. The
incremental path uses this to confirm a real change before re-harvesting.
Iceberg data commits (empty column diff, +1 version) are *absorbed* — the
new version is recorded without a re-harvest — so this row advances with
every commit while re-harvests fire only on schema changes; see
`OKF_INCREMENTAL_ICEBERG_COMMITS`.

### `okf-annotations` — user feedback on the wiki

Partition key `pk` (S), sort key `sk` (S). A separate table (not registry/
freshness) so its DynamoDB **TTL sweep** — on `expires_at` — can never reap a
durable row; the worst a stray `expires_at` can do is delete an annotation.

**Annotation.** `pk = "ANNO#<data_domain>#<dataset>#<user_sub>"`,
`sk = "<concept_id>#<annotation_id>"`, attrs `{data_domain, dataset, concept_id,
annotation_id, author?, quote, prefix?, suffix?, block_line?, note, status,
outcome?, resolution?, report_id?, created_at, updated_at, expires_at?}`
(`report_id` only on `submitted_via: "benchmark"` items — the report the note
was applied from, provenance only).

**Isolation is structural.** The author's immutable Cognito `sub` is baked into
the partition key, so a user's `Query` can only ever return their OWN annotations
— there is no cross-user read path (readers pass `user_sub` from the verified JWT,
never the body). `sub` (not `email`) is used because it never changes and is
`#`-delimiter-safe. `author` is the human-facing label (email) for display only.

**Anchoring is a quote, not a coordinate.** `quote` is the selected passage; the
UI grows `prefix`/`suffix` (see `okf_core.annotations.normalize_text` / the UI's
`minimalUniqueContext`) only until the `(prefix+quote+suffix)` window is unique in
the doc, so two identical quotes on a page are distinguishable. `block_line` is a
body-relative source-line HINT (from react-markdown's `node.position`, stamped as
`data-sl`) the agent can jump near — never the source of truth. A re-harvest
rewrites the doc, so any coordinate would go stale; the quote is what survives.

**Lifecycle.** `status` ∈ `open | in_review | resolved`; `outcome` (set with
`resolved`) ∈ `applied | rejected | orphaned`. `expires_at` (epoch seconds, 7-day
TTL — `okf_core.annotations.HISTORY_TTL_SECONDS`) is set ONLY at resolution, so an
open/in_review annotation never expires. The Control API's `run` pre-flight
(`POST /harvest/{domain}/{dataset}/annotations/run`) takes the per-dataset lease,
then for each of the caller's open annotations loads the target doc from S3 and
re-anchors the `quote` (`is_orphaned`): a note whose passage is gone is
auto-resolved `orphaned` (with `ORPHAN_RESOLUTION_MESSAGE`) and the agent never
sees it. If EVERY open note orphans (or none are open), the run is **skipped** —
the status row is set `complete` and the runtime is NOT invoked. Otherwise the
survivors are flipped `in_review` and sent in the `annotated` payload; on invoke
failure the Control API reverts them to `open` so no feedback is stranded. After
the run, the harvest RUNNER (not the agent — it has no DynamoDB tools) reconciles
the agent's on-mount verdict file to `resolved` with `outcome`+`resolution`, and
reverts any survivor the agent didn't rule on back to `open`.

CRUD: `GET|POST /annotations/{domain}/{dataset}` and
`DELETE /annotations/{domain}/{dataset}/{annotation_id}?concept=<id>` (the concept
id has slashes, so it rides in the query string, not a path segment).

**Annotation scope (cross-dataset docs).** The run endpoint accepts an optional
body `{"scope": "dataset" | "cross"}`: `cross` applies only notes whose
`concept_id` is under `external/` (the cross-dataset docs), `dataset` only the
rest; absent = everything. `_dataset`-WIDE notes are general feedback and pass
BOTH filters — whether one rides a given run is the `annotation_ids`
selection's call (the UI offers them in every scope, preselected only in the
dataset one). Out-of-scope OPEN
notes stay untouched for a later run of the other scope; an out-of-scope
`in_review` STRAGGLER (from a dead prior run) is reverted to `open` rather than
dropped, preserving the reclaim invariant even for users who always pick one
scope. A `cross`-scoped run also IGNORES dataset guidance AND the saved
`recursive_improvement` settings (both operate on the dataset's own docs, which
the scope excludes). When surviving notes reference `external/<d>/<ds>/…` docs,
the Control API derives the counterpart datasets from the concept ids and sends
their Glue database names as `extra_glue_databases` in the payload — the
runtime widens the run's scoped session policy to them so the agent can
actually verify cross claims with qualified SQL (without it, every check would
be AccessDenied and the notes would be falsely rejected). The UI's picker modal
offers one cross scope PER TARGET PAIR (`Cross-dataset · <domain>/<dataset>`,
from the bundle's `external/` listing plus any pending note that targets one) —
never a generic "all external" bucket — so a run names exactly which target it
verifies against. The pair choice rides in `annotation_ids` plus an optional
`cross_target: "<domain>/<dataset>"` body field (the wire `scope` stays
`"cross"`; `cross_target` is refused with any other scope): the selected notes'
concept ids widen the session policy, and `cross_target` guarantees the
target's Glue database is granted even when the selection carries only
`_dataset`-wide general notes (whose ids name no pair).

**Partial selection.** The run endpoint also accepts an optional body
`annotation_ids: [<id>, …]` (the UI's annotation picker): only the listed notes
ride the run. Unselected OPEN notes stay open for a later run; an unselected
`in_review` straggler reverts to `open` — the same stranding argument as the
scope filter. An empty list is valid: with a dirty guidance the run still fires
guidance-only, otherwise it short-circuits as "nothing to apply". Composes with
`scope` (the id filter applies within the scope).

## Harvest invocation payload

`InvokeAgentRuntime(agentRuntimeArn=<harvest arn>, runtimeSessionId=<per-dataset
id>, payload=json.dumps({...}).encode())`, where the payload is either:

```json
{ "data_domain": "sales", "dataset": "orders", "mode": "full",
  "source": { "type": "glue", "glue_database": "orders" },
  "model": "openai.gpt-5.6-sol", "effort": "xhigh",
  "domain_description": "Revenue & order pipelines",
  "domain_context": "Covers all B2C sales; refunds excluded." }
```

`source` (optional, all modes) is the first-class source descriptor
(`okf_core.sources`, `{type, ...config}`) the Control API resolves from the mapping
row and threads through so the runtime dispatches on the source type
(`harvest.clients.build_source`) instead of assuming a Glue database named by the
dataset. A `glue` source carries `glue_database`; a `redshift` source carries
`redshift_database` plus its cluster/workgroup + `secret_arn` connection routing
(self-describing — the harvest connects from the descriptor, no deploy-time env).
**Absent → the runtime defaults to a glue source named by `dataset`** (back-compat:
older payloads and the provision/write-domain-doc modes carry no `source`). The incremental path is
Glue-only (it fires on `aws.glue` catalog events) and always sends a `glue` source.

(`model`/`effort` optional — see below.) Or, for an incremental run:

```json
{ "data_domain": "sales", "dataset": "orders", "mode": "incremental",
  "changed_table": "customers",
  "diff": { "added": [], "removed": [], "retyped": [] },
  "domain_description": "Revenue & order pipelines",
  "domain_context": "Covers all B2C sales; refunds excluded." }
```

or, for a cross-dataset run (Roadmap §5 — author `external/` pair docs):

```json
{ "data_domain": "sales", "dataset": "orders", "mode": "cross",
  "source": { "type": "glue", "glue_database": "orders" },
  "target": { "data_domain": "crm", "dataset": "customers",
              "source": { "type": "glue", "glue_database": "customers" },
              "domain_description": "Customer master data",
              "domain_context": "…" },
  "domain_description": "Revenue & order pipelines" }
```

The Control API resolves + validates `target` from the UI's flat
`target_data_domain`/`target_dataset` body fields (`resolve_cross_target`):
registered mapping (404), glue-backed on BOTH sides (400 — v1 verification is
qualified Athena SQL, so cross-source pairs have no common engine), distinct
from the dataset itself AND resolving to a DIFFERENT Glue database (400 — the
same dataset name under two domains is the same physical data), and BOTH
bundles published (409). A cross payload deliberately carries **no
`dataset_guidance` and no `recursive_improvement`** — guidance is
dataset-scoped steering and the pair docs are shared with another dataset's
readers. The runtime validates the target components as path segments
(`okf_core.paths.external_pair_prefix` — they become destructive paths and the
XREF key), widens the run's scoped session policy to the pair's two Glue
databases (never more), snapshots the target's catalog + published docs (minus
the target's own `external/` subtree — another run's pair docs are not the
target's own verified facts) into `.metadata/external/<d>/<ds>/`, and confines
writes to `external/<d>/<ds>/` (guard-enforced, INCLUDING the cross-mode
reviewer's middleware). Ordering is load-bearing: the target-readiness
re-checks (the trigger-time check only covered trigger time) and the required
target snapshot all run BEFORE the first destructive step, so a failure there
leaves the bundle untouched and READY — prior pair docs intact. Uses a fresh
session id per trigger, like `full`.

or, for an annotation run (apply a user's wiki feedback in place):

```json
{ "data_domain": "sales", "dataset": "orders", "mode": "annotated",
  "user_sub": "<cognito sub>",
  "annotations": [
    { "annotation_id": "…", "concept_id": "tables/orders",
      "quote": "one row per order", "prefix": "", "suffix": "",
      "block_line": 12, "note": "grain is per line-item, not per order" }
  ],
  "model": "openai.gpt-5.6-sol", "effort": "high",
  "subagent_model": "…", "subagent_effort": "…",
  "reviewer_model": "…", "reviewer_effort": "…",
  "domain_description": "Revenue & order pipelines",
  "domain_context": "Covers all B2C sales; refunds excluded.",
  "dataset_guidance": "Ignore the staging_* tables; status is decoded in the dictionary.",
  "dataset_guidance_version": "2026-07-17T09:00:00+00:00" }
```

Applying annotations is a harvest like any other, so `model`/`effort` (+ the
`subagent_*`/`reviewer_*` pairs) are the SAME optional per-harvest override
triple `mode: "full"` accepts — same three scopes (supervisor / sub-agents /
reviewer), same catalog validation at the Control API trust boundary, same
fallback when omitted (the runtime's deploy-time `OKF_HARVEST_MODEL`/
`OKF_HARVEST_EFFORT`). The UI's harvest picker sends its current selection on
an annotation run, so applying annotations honors whatever model an operator
had chosen for full harvests of this dataset, rather than silently reverting
to the deploy-time default.

`dataset_guidance` (optional, on every mode) is the dataset's shared authoring
guidance — persistent, editable operator instructions (registry
`DATASET#` row: `guidance`, `guidance_updated_at`, `guidance_applied_version`).
It steers the harvest prompt; on a SUCCESSFUL run the runner stamps
`guidance_applied_version = dataset_guidance_version` so the guidance clears its
DIRTY state (`okf_core.guidance.is_dirty`). An `annotated` run is invoked when
there are live annotations **or** the guidance is dirty — so editing guidance and
re-running applies it even with zero annotations (a guidance-only re-harvest,
`annotations: []`).

**Benchmark Studio invocation (`mode: "benchmark"`).** A standalone,
human-triggered evaluation on the harvest runtime — NOT a harvest: it takes no
lease, doesn't use the S3-Files mount (the wiki snapshot is GET straight from
S3, live or pinned to a bundle version), and writes nothing to the bundle. The
harvester itself can no longer benchmark — the in-run recursive-improvement
loop, its `run_benchmark` tool, and the `recursive_improvement` payload block
are retired end to end. The payload (`okf_core.benchmark_report` field names):

```json
{ "data_domain": "sales", "dataset": "orders", "mode": "benchmark",
  "report_id": "r20260729t101500-1a2b3c4d",
  "checks": ["sql", "behavior"],
  "runs": 3,
  "version_id": "",
  "questions_key": "benchmark/sales/orders/questions.csv",
  "questions_version_id": "3sL4kqQJlcpXroDTDmJ+rmSpXd3dIbrHY+MTRCxf3vjVBH40Nr8X8gdRQBpUMLUo",
  "solver_model": "global.anthropic.claude-sonnet-5", "solver_effort": "high",
  "judge_model": "global.anthropic.claude-opus-5", "judge_effort": "xhigh",
  "behavior_live_sql": false,
  "source": {"type": "glue", "glue_database": "orders"} }
```

`questions_version_id` (optional) is the CSV's S3 VersionId as the Control API
validated/counted it at start — the bundle bucket is versioned, so the runtime
GETs exactly that version and a re-upload between start and the fetch can't
swap the graded set. Absent (older payloads, or a CSV written while the bucket
was unversioned) → the runtime reads the latest object.

`behavior_live_sql` (optional, default false; also a `BOOL` on the REPORT#
row's config summary when true) hands the BEHAVIOR solver read-only `run_sql`
against the live dataset — a truer consumer simulation (real agents can
query); its prompt flips from "you CANNOT query" to wiki-leads-SQL-verifies
(`harvest/benchmark/checks.py solver_protocol`). It never applies to the SQL
EX check, whose solver stays data-blind by design. Reports carry the flag —
scores are not comparable across different settings of it.

`questions_key` is the uploaded CSV — one gold column per check
(`question,gold_sql,expected_behavior`; a question participates in a check iff
its gold cell is non-blank; unrecognized columns are ignored — the retired
`gold_answer` no longer resolves). It lives under the
**off-mount** `benchmark/<domain>/<dataset>/` prefix — deliberately NOT under
`okf/` — so the gold is invisible to every LLM role; the runtime GETs it into
process memory (needs `s3:GetObject` on `<bundle-bucket>/benchmark/*`).
`checks` ⊆ `{sql, behavior}` (≥ 1); `runs` is clamped to 1–5; models are
validated against the harvest catalog by the Control API; `version_id`
(optional) pins the wiki to a published bundle version — **the WIKI, not the
DATA**: grading always executes against live Athena. Question count is
hard-capped at 100. `sql` (shown as "Accuracy") grades deterministically —
BIRD-style result-set equality: rows compared as POSITIONAL tuples (column
order matters, row order doesn't), with numeric-looking cells normalized to
`Decimal` so Athena's stringified `3` vs `3.0` compare equal; `behavior` is
**judge-graded**: `expected_behavior` is
free-form prose (refusals, policy adherence, "should say it isn't tracked"),
the solver answers in free-form text, and the judge rules on EVERY
(question, run) attempt independently — so `behavior` has NO judge-adjusted
score and its failed pairs never enter the overturn review (the grader already
was the judge; its score block carries `adjusted: null`, and the REPORT# row
omits `behavior_adjusted`). Each failed behavior pair instead gets ONE
question-level SYNTHESIS review (all graded runs together): it supplies the
pair's `judge` block — comment + one consolidated annotation (the annotation
candidate) — and never changes outcomes.
Failures are LOUD: a run that can't fetch/parse its questions or materialize
its snapshot fails the REPORT# row with the error (no silent degradation). The
judge phase is always on; there is no stop target and no loop. Every benchmark
ReAct role (solver, the judge's hats, the annotation aggregator) is built via
``harvest/benchmark/react.py`` — LangChain ``create_agent`` with the chat
agent's ``BedrockPromptCachingMiddleware`` — so on a Converse Claude model the
per-turn re-sent conversation bills as cache READS (a Mantle GPT caches
implicitly server-side, where the middleware no-ops). The judge hats DELIVER
their ruling through a tool call (``submit_verdict`` / the reviewer's
``submit_review`` — args are the output, structured by construction, no fence
parsing), and a ``SubmitToolNudgeMiddleware`` steers a hat that tries to
finish without submitting — at most twice, then the unparseable-output path
rules the case a fail with ``judge_error`` set. The judge hats see
everything the solver can't: their file tools are rooted at the JUDGE tree
(wiki docs + `.metadata/` + `.context/` + the `.traces/` solve traces), and
when that tree carries `.context/` files the run also opens a Code
Interpreter sandbox session (`harvest/code_interpreter.py sandbox_session` —
the same `OKF_CODE_INTERPRETER_ID` interpreter the harvester uses) and hands
the hats `run_code`, so binary context uploads (PDF/DOCX/PPTX/XLSX) that
`read_file` only base64-encodes stay readable evidence. Best-effort like the
harvest's: no interpreter configured, or a start/upload failure, just means
the judge reads text context only.

**Report artifacts (off-mount, human-facing).** The run persists
`benchmark/<d>/<ds>/reports/<report_id>/report.json` — config recap, per-check
scores (raw + judge-adjusted, per-run + mean ± spread), per-question stability,
per-question detail (gold, every attempt's outcome/reason/prediction, the
judge's `{verdict: pass|fail, comment, annotation}`), telemetry (per-tool call
distribution, tokens by role, wall time), and the judge's annotation
`candidates` (+ the aggregator's `final` set once generated) — and a companion
`traces.json` (EVERY attempt's bounded solver trace, passing and failing, keyed
`{q_id, check, run}`; shape per `harvest/benchmark/trace.py`). Both carry gold,
so they are served ONLY via the Cognito-authed Control API: `POST/GET
/benchmark/{d}/{ds}/runs`, `GET/DELETE .../runs/{report_id}`,
`GET .../runs/{report_id}/traces`, `POST .../runs/{report_id}/aggregate`.
An artifact past the 4 MiB inline cap (a Lambda response tops out at 6 MB;
multi-run `traces.json` routinely exceeds it) is answered as a short-lived
presigned S3 GET instead of the document — `report_url` on the report
response, `{report_id, traces_url}` on traces — which the UI api client
follows transparently. `DELETE .../runs/{report_id}` is refused (409) while
the run or an aggregation is genuinely active, but a row whose `updated_at`
heartbeat predates the harvest-lease stale cutoff (8 h) is deletable — a
killed runtime must not leave an immortal zombie — and the runtime's row
writes are conditional on the row existing, so a late finish can't resurrect
a deleted report (the runtime also re-checks the row before persisting the S3
artifacts, so a delete mid-run doesn't leave orphaned gold behind). Because
the bundle bucket is VERSIONED, deletion purges **every object version and
delete marker** under the report prefix (`list_object_versions` + per-version
deletes) — a plain delete would leave the gold readable as noncurrent
versions. Deleting the DATASET purges the whole
`benchmark/<d>/<ds>/` prefix the same versioned way (questions.csv + all
report artifacts) and every `REPORT#` row along with the bundle. `POST
.../runs/{report_id}/aggregate`
kicks `mode: "aggregate_annotations"` — the ReAct aggregator dedupes the
candidates into the final set on the report — and `POST
.../runs/{report_id}/annotations` batch-creates the human-selected set as
normal annotations with `submitted_via: "benchmark"` (validated whole-batch
before anything is written — no partial commits; the path's `report_id` must
name an existing report row and is stamped on each created annotation for
provenance); an unscoped annotation
harvest then applies them. On both start routes (run + aggregate) the Control
API reads the invoke's synchronous ack: a `{"status": "rejected"}` from the
runtime flips the row to `failed` (run `detail` / `agg_detail`) and answers
502 — an accepted invoke API call is not an accepted payload. The judge reads each solver's trace — what it
searched, which docs it opened — which is what separates "the wiki never
says this" from "the wiki says it and the solver never found it"; beyond the
per-case inline summaries, ALL traces are laid into the judge's file tree as
`.traces/<check>/q<id>-run<n>.md` so it can `grep` across solvers for
systemic patterns.

**Synthetic question banks** (`mode: "generate_questions"` — `okf_core.qbank`
+ `harvest/benchmark/qgen.py`) share the Benchmark Studio posture exactly: no
lease, no mount, nothing written to the bundle, loud failures on the row. The
index row is `sk = "QBANK#<qbank_id>"` on the same `HARVEST#<d>#<ds>`
partition (ids `qb<timestamp>-<token>`, time-prefixed and chronologically
sortable like report ids), same `queued → running → complete/failed`
lifecycle, heartbeat, and stale-delete escape; the artifact —
`benchmark/<d>/<ds>/qbank/<id>.json`, gold-carrying, off-mount — holds
`{config, questions[], dropped[], counts, telemetry}`. The author agents see
ONLY the dataset's ground truth (`.metadata/` + `.context/`, physically
materialized without the wiki docs — the wiki is the system under test) plus
the live source tools; the config (`count` 20–100, `checks` + `sql_share`,
`dimensions`) is validated by `okf_core.qbank.validate_config` at the Control
API AND in the runtime, and the deterministic allocator turns it into
explicit (dimension, tier, check) slots. Questions are delivered through the
`submit_question` tool, which validates at submit time (shape, cross-author
dedup, business-language leakage lint, and a live gold execution under the
grading caps — an applied bank can never produce DISCARDED questions); a
quota middleware refuses a silent finish (every slot filled or explicitly
forfeited), one backfill round — chunked into round-1-sized author batches so
a large leftover set doesn't blow one agent's step budget — retries the
leftovers, and the rest are dropped WITH reasons in the artifact. Routes: `POST/GET
/benchmark/{d}/{ds}/qbanks`, `GET/DELETE .../qbanks/{qbank_id}`, `POST
.../qbanks/{qbank_id}/apply`, `POST .../qbanks/{qbank_id}/cancel`. Cancel
mirrors the harvest cancel (best-effort `StopRuntimeSession` on the row's
session, then a CONDITIONAL flip to `cancelled` that never clobbers a
terminal state) with one stronger guarantee: **partial work does not
survive** — the cancel purges every artifact version under the qbank key,
the runtime discards instead of persisting when it finds the row cancelled,
EVERY runtime row write (the initial `running` flip included — a cancel
landing while the row is still queued cannot be undone by the cold start,
which also exits without generating) is conditional on NOT-cancelled (a
blocked write is dropped, closing the resurrection race), the start path's
failure flip is conditional the same way (a hung invoke can neither
overwrite a cancel nor upsert a ghost of a deleted row), and `GET`/`apply`
serve the artifact for COMPLETE rows only, so even a race-surviving orphan
is unreachable. Dataset deletion purges QBANK# rows with the REPORT# rows.
A bank past the inline response cap ships as a presigned `bank_url` (the
canonical CSV stays inline). Apply renders the canonical CSV
(`question,gold_sql,expected_behavior,tier,dimension`; the extra columns are
ignored by `load_questions` per its documented contract) and REPLACES
`benchmark/<d>/<ds>/questions.csv` (reversible: versioned bucket, runs pin
their CSV version).

or, for writing/refreshing a domain's concept doc through the mount:

```json
{ "data_domain": "sales", "mode": "write_domain_doc",
  "description": "Revenue & order pipelines",
  "context": "Covers all B2C sales; refunds excluded." }
```

The `annotated` payload carries only the LIVE annotations (the Control API's
pre-flight already resolved any orphans) plus the `user_sub` needed to reconstruct
each annotation's DynamoDB key for the runner's write-back. The agent assesses
each note against live data, edits the doc when it holds up (augmentation guard
applies), and writes a per-annotation `{outcome, comment}` verdict to
`.harvest/annotation_results.json` on the mount; the runner reconciles that to the
`okf-annotations` table. It reuses the incremental path's scoped, in-place
approach (no `clean_authored_output`) and the deterministic per-dataset session id.

`domain_description` and `domain_context` are optional enrichment keys added by
the Control API (and the incremental orchestrator) from the `DOMAIN#/META` row.
They are threaded into the harvest prompt so authoring is domain-aware. The
`write_domain_doc` mode writes `<mount>/<domain>/_domain/overview.md` through
the mount (uid 1000 safe) and returns synchronously.

`model` and `effort` are optional per-harvest overrides for the LLM (chosen in
the UI's harvest-settings picker; `full`/`incremental` only). When present the
runtime uses them; when absent it falls back to the deploy-time `OKF_HARVEST_MODEL`
/ `OKF_HARVEST_EFFORT` env. `subagent_model` and `subagent_effort` are the same
kind of override for the run's SUB-AGENTS — the table/reference authors,
reviewers, and context-extractors; when absent the sub-agents run on the
supervisor's config.
The Control API **validates each pair against the model catalog**
(`OKF_HARVEST_MODEL_CATALOG`, from `var.harvest_model_catalog`) before
invoking — an unknown model or an effort not offered for that model is a `400`,
and an effort without its model key is a `400`. This is the trust boundary: both
model values reach `bedrock:InvokeModel`, and the runtime deliberately does not
allow-list effort itself. The catalog (a JSON array of `{model, label, efforts,
default_effort}`) is the single source of truth, shared by the Control API
(validation, raw JSON env) and the UI (`VITE_HARVEST_MODEL_CATALOG`, base64 —
see below) and defined in `okf_core.harvest_models`.

`reviewer_model` and `reviewer_effort` are a third override for the adversarial
`reviewer` sub-agent ONLY — cross-model review improves coverage (a fresh model
family doesn't share the authoring model's blind spots). Absent ⇒ the reviewer
runs on the sub-agents' config (which itself falls back to the supervisor's).

The runtime always builds **three model instances** — the supervisor's, the
sub-agents' (authors/extractors), and the
reviewer's (identical configs when no overrides were sent) — each carrying its
own scope-tagged usage callback. That is what makes the step feed's `usage`
snapshot splittable: it carries the cumulative run totals plus a `by` object
(`{supervisor: {...}, subagents: {...}, reviewer: {...}}`, same counter names)
the UI renders as the per-agent token drill-down. The resolved supervisor pair
is stamped on the status row as `model`/`effort` at the `running` transition;
`subagent_model`/`subagent_effort` and `reviewer_model`/`reviewer_effort` are
stamped only when the respective override was chosen (absence means "same as
the tier above").

Build `runtimeSessionId` with `okf_core.runtime_session_id(...)`, not a bare
`"<domain>__<dataset>"` — AgentCore requires 33–256 characters, so the helper
appends a sha256 suffix to a readable `okf-<domain>-<dataset>-` prefix.

- **Incremental** uses a deterministic id (`runtime_session_id(domain, dataset)`)
  for one session per dataset and microVM affinity. It re-authors the changed
  table and its backlinks in place and leaves the rest of the bundle alone.
- **Full** uses a fresh id per trigger (`unique_token=uuid4().hex`), because a
  one-shot batch job wants a new microVM with a clean S3 Files mount rather than
  reattaching to a warm one (AgentCore reuses a microVM per session id until it
  stops). A full harvest is a clean rebuild: `run_full_harvest` marks the bundle
  in-progress, then `fsutil.clean_authored_output` deletes all prior authored
  output (`datasets/`, `tables/`, `references/`, `index.md`, `log.md`) before the
  agent re-authors. A table dropped from Glue leaves no stale doc, and its vector
  is pruned through the S3 write-through → `ObjectRemoved` → reindex
  `DeleteVectors`. `.context/` (user input) and `.harvest/` (the commit marker)
  are preserved. The rule is: delete every top-level entry whose name does not
  start with `.`.

## Reports (chat-authored)

`create_report` (a chat tool; see `chat/reports.py`) lets the MAIN chat agent
compose an immutable HTML report from evidence gathered in the conversation —
there is no delegated runtime and no job row; the chat loop itself is the
research loop. The pure contract (block schema, composer, id/key helpers)
lives in `okf_core/reports.py`.

- **Blocks are config, not code**: `markdown | chart | table | kpi`; chart
  blocks carry the same declarative `renderChart` spec as chat charts.
  Figure blocks carry `provenance` — `{kind: "computation", slug, params?}`
  (VERIFIED badge) or `{kind: "adhoc_sql", sql}` (EXPLORATORY badge, SQL
  disclosed in the report).
- **The save is atomic and render-verified**: lint → every chart rasterized
  through the baked harness page in headless Chromium (a chart that fails
  refuses the save) → one self-contained HTML (inline CSS, data-URI PNGs,
  print stylesheet) → PDF printed from that same HTML → S3 puts.
- **No database row.** The public report id is composite —
  `rep~<domain>~<dataset>~<YYYYMMDDTHHMMSSZ>~<hex8>` — so serving resolves S3
  keys from the id alone (`okf_core.reports.parse_report_id`).
- **S3 layout** (bundle bucket, OUTSIDE the mounted `okf/` prefix — never
  meets the harvest lease or reindex):
  `reports/<domain>/<dataset>/<stamp>-<suffix>/{blocks.json, report.html,
  report.pdf}`. `blocks.json` is the self-describing source (title, request,
  coordinates, `created_by`, blocks) — future re-rendering never needs the
  agent.
- **Serving**: `GET /report/{report_id}` (control API) returns presigned GET
  urls `{html_url, pdf_url, blocks_url}` — presigned because a composed
  report can exceed the Lambda response cap; `pdf_url` is `""` on a
  deployment without Chromium. The UI lifts `create_report` /
  `present_report` tool calls into inline report cards; the viewer renders
  the HTML in an iframe with NO `allow-scripts`.
- **Env**: `OKF_CHAT_REPORT_HARNESS_PATH` (the harness page in the chat
  image; empty = chart blocks refused, text/table/kpi reports still work),
  `OKF_CHAT_REPORT_MAX_BYTES` (composed-HTML cap, default 8000000). IAM: the
  chat role holds `s3:PutObject` on `reports/*` only — the wiki stays
  read-only to chat.

## Long-term chat memory

Per-user memory on **Bedrock AgentCore Memory** (`infra/durable/agent_memory.tf`
— one memory resource + one CUSTOM user-preference strategy whose extraction/
consolidation prompts are the design's rulebook). Memory stores facts about the
USER — never facts about the data (tables/joins/metrics belong to the wiki):

- **Three record kinds, one namespace per user**: `stated` preferences
  (presentation/workflow/language, plus meanings the user assigns to terms),
  `personal` context (name/role/team — only what the user states about
  themselves), and `binding` records (how this user's recurring question maps
  to a governed artifact — a verified computation or metric, with the
  parameter SHAPE, never literal values). The namespace is
  `wiki/<sanitized-sub>` — derived via `okf_core.memory_records
  .memory_namespace` in BOTH services (the strategy resolves `{actorId}` from
  CreateEvent's actorId, which is sanitized to `[a-zA-Z0-9][a-zA-Z0-9-_]*`;
  deriving from the raw sub anywhere would split write and read).
- **Structured record fields are REAL metadata** (`type` / `dataset` /
  `expires_at`, LLM-extracted per the strategy's `metadata_schema` — the ONE
  awscc-managed resource in the repo, because hashicorp/aws doesn't expose
  the schema yet). `type`+`dataset` are indexed → server-side retrieval
  filters; `expires_at` is client-checked (unset-OR-future isn't expressible
  in AND-composed filters). A legacy content-header line
  (`[type:...] [dataset:...] [expires:...]`) is parsed as FALLBACK for
  drifted extractions. Parser shared via `okf_core.memory_records`
  (chat runtime + control API both use it — never parse ad hoc).
- **Runtime touchpoints** (`chat.memory`): at turn START semantic retrieval
  (query = raw prompt + previous turn's curated question; `maxResults`
  explicit — the API's default page is 20), then client-side lazy-TTL
  (expired records are DELETED on recall) and dataset scoping (pinned chat →
  TWO filtered calls: generic `dataset NOT_EXISTS` + the pin `EQUALS_TO`,
  INTERLEAVED dataset-first so neither pool starves the other; unpinned →
  one unfiltered call), injected as a marker-carrying HumanMessage
  (`okf_memory` — skipped by history rebuild and steering's turn accounting).
  The per-turn injection (marker value `recall`) is REPLACED each turn — the
  server strips the previous one from checkpointed state via `RemoveMessage`
  before injecting fresh, so an edit/delete on the Memory page (or the
  switch going off) actually changes ongoing threads. `personal` records are
  the exception: fetched once (`recall_personal`, server-side `type` filter),
  injected on the thread's FIRST turn (marker value `personal`), carried by
  history. The whole turn-start prep runs in a worker thread
  (`asyncio.to_thread`) — a memory-API brownout must never stall the shared
  event loop. At turn END one `create_event` carrying the turn text
  (every piece excerpt-capped) + a `[[okf-harness]]` annotation of observed
  facts so extraction judges only acceptance and phrasing, never the factual
  core. The annotation's dataset resolution is two-level and either/or:
  **level 1** parses the answer's `<c src="dd/ds/…">` citations and keeps the
  pairs the harness can corroborate (observed this turn, in the thread's
  cumulative `memory_datasets` ledger, or the conversation pin) — a
  `datasets-cited` line that REPLACES the touched list (attribution beats
  exploration, and it survives no-tool follow-ups); **level 2** falls back to
  the observed `datasets-touched` list when no citation validates (an
  unbacked citation is a model claim, never laundered into the trusted
  block). `resolved-by` lines (which governed tool resolved the turn) ride
  regardless — bindings stay observation-only evidence. The annotation also
  carries a `curated-question:` line — the policy machinery's
  context-resolved rewrite of the turn's question — whenever it differs from
  the raw text: extraction is asynchronous, so an elliptical turn ("and last
  month?") must be self-contained in its own event; the raw wording stays
  the USER message (meanings extract from the user's words, never the
  rewrite).
  Paused (`ask_human`) turns persist their observation + folded clarification
  rounds in a `memory_pending` JSON blob on the THREAD row
  (`chat.threads.write_memory_pending`; each pause overwrites, the resume
  reads) so multi-round clarifications and pre-pause governed calls survive
  the invocation boundary; cancelled and ERRORED turns write no event.
- **A recalled binding is a HINT**: the injection instructs the model to
  re-verify the artifact (describe it; only VERIFIED may answer) — memory
  degrades to normal exploration, never to a wrong answer.
- **Management** (control API + the Memory page): `GET /memory` (list, with
  parsed header fields + `expired` flag), `GET/PUT /memory/settings` (the
  per-user switch — see the `okf-chat` settings row above), `PATCH
  /memory/{id}` (edit text; the record's real `metadata` is re-supplied
  verbatim and the header re-emitted, and a `failedRecords` entry in the 200
  response surfaces as a 409 — never a fabricated success), `DELETE
  /memory/{id}`. Namespace is always derived from the caller's JWT sub,
  never a parameter.
- **Env**: `OKF_CHAT_MEMORY_ID` on the chat runtime AND the control API
  (empty = feature off: no client, no recall, routes 404). Deploy gate
  `var.enable_chat_memory`; extraction model `var.chat_memory_model`
  (durable stack).

## Environment variables

| Variable | Meaning |
|---|---|
| `AWS_REGION` | region for all clients |
| `OKF_ACCOUNT_ID` | account id (for building Glue ARNs) |
| `OKF_BUNDLE_BUCKET` | S3 bundle bucket name |
| `OKF_VECTOR_BUCKET` | S3 Vectors bucket name |
| `OKF_VECTOR_INDEX` | S3 Vectors index name |
| `OKF_REGISTRY_TABLE` | DynamoDB registry table (default `okf-registry`) |
| `OKF_FRESHNESS_TABLE` | DynamoDB freshness table (default `okf-freshness`) |
| `OKF_ANNOTATIONS_TABLE` | DynamoDB annotations table (default `okf-annotations`) — user-scoped wiki feedback + the harvest runner's resolution write-back |
| `OKF_HARVEST_RUNTIME_ARN` | AgentCore harvest runtime ARN |
| `OKF_ATHENA_OUTPUT` / `OKF_ATHENA_WORKGROUP` | Athena results (glue source) |
| `OKF_GLUE_CATALOG_ID` | Glue catalog id override (glue source; default the runtime account's catalog) |
| `OKF_MOUNT_PATH` | S3 Files mount (default `/mnt/data`) |
| `OKF_CODE_INTERPRETER_ID` | AgentCore Code Interpreter id backing the harvest agent's `run_code` tool (extracts text from binary `.context/` docs). A network-isolated SANDBOX-mode interpreter. Unset → harvest runs without `run_code` (text-only `.context` reading) |
| `OKF_ENABLE_LAKEFORMATION` | Set (`"true"`) when the harvested Glue catalog is Lake Formation-governed → adds `lakeformation:GetDataAccess` to the harvest data role's per-invocation session policy so LF can vend S3 creds for governed table data. Set by `var.enable_lakeformation`; requires adopter-side LF grants + data-location registration (see `docs/LAKE_FORMATION.md`). Unset → plain IAM catalog access |
| `OKF_HARVEST_MODEL` | harvest model id — the **fallback default** used when a harvest request omits `model` (default `us.anthropic.claude-opus-4-8`). An `anthropic.*` id runs on the Bedrock **Converse** API (`ChatBedrockConverse`); an `openai.*` / `gpt-*` id (e.g. `openai.gpt-5.6-sol`) runs on the Bedrock **Mantle** OpenAI-compatible endpoint (`ChatOpenAI`, bearer-token auth via `aws_bedrock_token_generator`). The prefix selects the provider; see `agent._build_model` |
| `OKF_HARVEST_MODEL_CATALOG` | (Control API) JSON array of `{model, label, efforts, default_effort}` — the models + efforts the UI picker offers and the Control API validates a per-harvest `model`/`effort` against. From `var.harvest_model_catalog`; unset → `okf_core.harvest_models.DEFAULT_CATALOG`. The UI receives the same catalog **base64-encoded** as `VITE_HARVEST_MODEL_CATALOG` (base64 so it survives `deploy.sh`'s `eval "export k=v"`) |
| `OKF_HARVEST_MANTLE_REGION` | AWS region for the Bedrock Mantle endpoint when `OKF_HARVEST_MODEL` is a GPT id (default `us-east-2`). **Independent of `AWS_REGION`** — GPT-5.x on Mantle is only in us-east-2/us-west-2, while the harvest runtime may deploy elsewhere. Drives both the Mantle base URL and the region the bearer token is minted for. Ignored on the Converse path |
| `OKF_HARVEST_MANTLE_USE_RESPONSES_API` | selects the Mantle API surface (default `true` → OpenAI **Responses** API on the `/openai/v1` path, which is what GPT-5.x requires). Set `false` for a gpt-oss model (Chat Completions on `/v1`). GPT path only |
| `OKF_HARVEST_MANTLE_BASE_URL` | override for the Mantle base URL (default `https://bedrock-mantle.<region>.api.aws/openai/v1` for Responses, `.../v1` for Chat Completions; region from `OKF_HARVEST_MANTLE_REGION`). GPT path only |
| `OKF_HARVEST_MANTLE_READ_TIMEOUT` / `OKF_HARVEST_MANTLE_MAX_ATTEMPTS` | httpx read timeout (s) and retry budget for the `ChatOpenAI` Mantle client (defaults `600` / `5`, mirroring the Converse knobs). The botocore `OKF_HARVEST_BEDROCK_*` knobs do NOT apply to the GPT path |
| `OKF_HARVEST_EFFORT` | reasoning effort. On Converse, passed verbatim to Bedrock `output_config.effort` (default `xhigh`; valid values are model-specific). On the GPT path it maps onto OpenAI's `reasoning_effort` scale — verbatim on GPT-5.6 (which added `max` above `xhigh`), so `low`/`medium`/`high`/`xhigh`/`max` all pass through unchanged. Which efforts a given model accepts is model-specific (an older GPT id rejects `max`); the model catalog is the trust boundary that only offers a level a model supports |
| `OKF_HARVEST_MAX_TOKENS` | harvest model max output tokens. Default is provider-aware when unset: `128000` for Converse (Opus 4.8), `32000` for GPT. An explicit value always wins |
| `OKF_HARVEST_MAX_SUBAGENT_CONCURRENCY` | how many dynamic subagents run at once on a `task()` fan-out (default `5`). This lowers langchain_quickjs's per-REPL `task()` semaphore, so a `Promise.all` keeps at most this many crawls in flight and queues the rest. It is not `config.max_concurrency` — the fan-out is a QuickJS `Promise.all`, not a LangGraph batch, so only the semaphore bounds it. The same value bounds the `run_review` tool's in-flight cluster pipelines. |
| `OKF_HARVEST_REVIEW_DISPATCH_TIMEOUT_S` | wall-clock cap per `run_review` dispatch (one reviewer or one fixer; default `1800`). On timeout the dispatch is cancelled and its cluster is recorded as `failed` — retryable via `run_review(cluster_ids=[...])`. |
| `OKF_HARVEST_REVIEW_CLUSTER_SIZE` | docs per `run_review` review cluster (default `7`). The supervisor-owned hubs — `datasets/*` overview docs and `references/usage_guardrails` — are excluded from clustering entirely (they'd hub-steal unrelated spokes, and only the supervisor may edit them; corrections reach it as propagation notes). |
| `OKF_HARVEST_BEDROCK_READ_TIMEOUT` | botocore read timeout in seconds for the harvest bedrock-runtime client (default `600`). Botocore's 60s default is too low: one xhigh Opus 4.8 turn can generate for minutes, and a slow Converse response would otherwise raise `ReadTimeoutError` and fail the harvest. |
| `OKF_HARVEST_SQL_MAX_ROWS` | soft row cap on the agent-facing `run_sql` tool's result (default `200`). Collection stops at the cap and the tool reports `truncated: true` — a hint to the agent to add a LIMIT or aggregate — instead of buffering an unbounded result into its context. Distinct from the benchmark grader's hard `OKF_BENCHMARK_GRADER_MAX_ROWS` (which raises: a truncated set can't be equality-graded) |
| `OKF_HARVEST_PROFILE_ENABLED` | `"0"` disables the snapshot-time column profiles (`.metadata/profile/<table>.md` — null share, ~distinct, min/max, top-K values; see `harvest/profile.py`). Default on. Profiles are best-effort: any failure downgrades to a manifest note, never fails the snapshot |
| `OKF_HARVEST_PROFILE_SAMPLE_ABOVE_BYTES` | byte-size threshold above which a table is profiled from a sample instead of a full scan (default 1 GiB). The size comes from the catalog's size hint; a hint-less ICEBERG table is sized exactly from its `$files` metadata sum (the source's `iceberg_data_bytes` capability — Iceberg Glue Parameters never carry Hive stats); any other table with NO hint is treated as large. Sampled sheets are stamped INDICATIVE — value lists from a sample are never treated as closed enums |
| `OKF_HARVEST_PROFILE_TARGET_SAMPLE_BYTES` | the scan budget one sampled profile aims for (default 256 MiB) — the sample percent is `target/size`, clamped to 0.01–100 |
| `OKF_HARVEST_REL_QUERY_TIMEOUT_S` | per-query timeout in seconds for the relationship pass's SKETCH scans (default `60`). Doubles as the cost bound on each unknown-size last-resort sketch attempt — a cancelled attempt bills at most this long a scan and produces nothing rather than wrong evidence. The join/grain probe queries keep the source default |
| `OKF_HARVEST_REL_SMALL_TABLE_BYTES` | the feasibility-FREE zone of the full-scan policy (default 10 GiB — the pre-band gate value): tables measured at or under it are probed/sketched with no rate requirement; between it and `OKF_HARVEST_REL_MAX_TABLE_BYTES` a proven rate is required (see that row). Clamped to the ceiling when set above it |
| `OKF_HARVEST_PROFILE_ENUM_MAX_DISTINCT` / `OKF_HARVEST_PROFILE_TOPK` / `OKF_HARVEST_PROFILE_MAX_ENUM_QUERIES` | value-list bounds: only columns with ~distinct ≤ the first (default `50`) get a value list, capped at TOPK values (default `20`), at most MAX_ENUM_QUERIES per table (default `15`, most enum-like first). Higher-cardinality columns report the count only ("not enumerated") |
| `OKF_HARVEST_PROFILE_MAX_COLUMNS` / `OKF_HARVEST_PROFILE_BUDGET_S` / `OKF_HARVEST_PROFILE_QUERY_TIMEOUT_S` | remaining cost caps: columns profiled per table (default `100`), overall wall-clock budget for the whole profiling pass (default `1800`s — tables past it are `skipped-budget` in the manifest), and per-query timeout (default `60`s — reliably tables to ~50 GB at Athena's dependable scan rates; a cancelled query bills its partial scan and yields neither a sheet nor the size measurement, so RAISE it per deployment — ~`300`s buys the 100–500 GB band — when such tables are worth profiling to completion; every query's timeout is CLAMPED to the budget remaining, so no setting lets one query overrun the pass). The pass-1 scan doubles as a size AND throughput measurement: its `data_scanned_bytes` and `engine_ms` (bytes/time = the table's layout-aware scan rate, powering the relationship pass's proven-band feasibility check) plus per-column `{distinct, null_pct}` persist as sibling keys on the table's `domains.json` entry (cache-carried) and feed the relationship pass — observed bytes as a sizing rung, column stats to rank sketch columns. Reuse makes re-runs cheap: profiles persist on the mount and are fingerprint-keyed (catalog update time + version + column set), so incremental runs re-profile only the changed table and cross runs only mismatches; a full harvest always re-profiles |
| `OKF_HARVEST_BEDROCK_CONNECT_TIMEOUT` | botocore connect timeout in seconds (default `10`) |
| `OKF_HARVEST_REL_ENABLED` | `"0"` disables the snapshot-time RELATIONSHIP EVIDENCE pass (`.metadata/relationships/` — `joins/<a>__<b>--<key>.md` with match rates both ways / cardinality / orphan samples, and `grain/<table>.md` with key uniqueness; see `harvest/relationships.py`). Candidates are enumerated mechanically (shared key-like column names — `_id`/`_key`/`_nbr`/`_sk` suffixes and table-naming prefixes, holder→home-table pairing) and probed by the SAME SQL cores as `validate_join`/`check_grain` (`harvest/probes.py`) — a plain loop with no model, so authors read verified evidence instead of spending agent turns re-probing. Best-effort: failures are manifest rows, never a failed snapshot. Default on |
| `OKF_HARVEST_REL_BUDGET_S` / `OKF_HARVEST_REL_MAX_PAIRS` / `OKF_HARVEST_REL_MAX_TABLE_BYTES` / `OKF_HARVEST_REL_MAX_TABLES_PER_KEY` / `OKF_HARVEST_REL_MAX_GRAIN_PER_TABLE` | the evidence pass's cost caps: wall-clock budget (default `1800`s — subjects past it are `skipped-budget`; GRAIN probes run FIRST — one cheap aggregate per table — so a candidate-flooded join loop can never starve them, and the sketch scan is hard-capped at HALF the budget so collection can't eat the probes' time — both seen live on MusicBrainz: 3,469 skipped, 0 grain), join-pair cap (default `100`; it counts PROBED pairs only — cache reuses, TYPE MISMATCH sheets, and size-skips are free — and when it binds the probe budget is SPREAD, least-covered table first with sketch nominations winning ties (a skipped renamed-key pair is unrecoverable live); pairs beyond it are `skipped-cap` without even being sized), per-side FULL-scan policy in three tiers — at or under `OKF_HARVEST_REL_SMALL_TABLE_BYTES` (default 10 GiB) unconditionally; over `OKF_HARVEST_REL_MAX_TABLE_BYTES` (default 50 GiB) never; in the PROVEN band between them only when a profile-measured scan rate (the table's own `scanned_bytes/scan_ms`, else the catalog's conservative p25) predicts completion within 90% of the probe's query timeout — no measurement in the band means it does not run (`skipped-slow` on grain rows, distinct from the hard `skipped-size`), so the raised ceiling bounds worst-case SPEND while the proof bounds worst-case TIME; the measured rates and implied byte ceiling are logged as operator guidance (job logs only, no feed line); when exactly ONE side is refused a full scan — the enterprise fact→dim shape (or a proven-band side without proof: sampling makes it feasible by construction) — the pair is probed with that side SAMPLED toward `OKF_HARVEST_REL_SAMPLE_TARGET_BYTES` (default 256 MiB, Athena `TABLESAMPLE SYSTEM` so billed bytes actually shrink) against the FULL small side, reporting ONLY the unbiased sampled→full direction on an INDICATIVE sheet (`ok-sampled` in the manifest) with every sampled-side number from ONE scan (each `TABLESAMPLE` reference draws an independent sample, so the matched count rides a LEFT JOIN inside the same query); both sides over, an unmeasurable size, or a source without `sql_sampled_ref` still `skipped-size` — and grain probes are never sampled (sample-uniqueness proves nothing); a table with NO byte-size hint is MEASURED — first for free from the profile pass's observed `data_scanned_bytes` (a COMPLETE-scan measurement of the profilable columns, never an early-exit lower bound, and the honest number for what a probe would bill), then via the source's `estimate_table_bytes` — for ICEBERG tables the exact `$files` metadata sum first (a manifests-only query through the engine's own permissions; an S3 listing would overcount every retained snapshot until VACUUM), then an S3 listing of the table's location, no query, early-exit at the gate — because DDL-registered tables carry no `totalSize` Parameter (only crawlers/ETL write it) and pure assume-large skipped every probe on such catalogs; only when even the measurement can't tell (a view, listing denied, a partitioned table whose root lists empty) does the table count as oversized. The SAMPLE PERCENT is never sized from the gate measurement (it early-exits, so it is only a lower bound that would over-sample a huge table by orders of magnitude) — the pass re-lists to completion for the sampled side and skips the pair when even that can't tell. The workgroup has no scan cutoff, so this gate is the only bound), fan-out cap for a key shared by many tables (default `6`; beyond it only holder→home pairs are probed, or none when no home table exists), and grain candidates per table (default `2`). Reuse mirrors the profile cache: sheets are fingerprint-keyed on BOTH tables' catalog fingerprints, incremental runs re-probe only pairs touching the changed table, a full harvest re-probes everything |
| `OKF_HARVEST_REL_SKETCH_ENABLED` / `OKF_HARVEST_REL_SKETCH_K` / `OKF_HARVEST_REL_SKETCH_MIN_CONTAINMENT` / `OKF_HARVEST_REL_SKETCH_MAX_COLUMNS` | the NAME-BLIND value-sketch nominator inside the relationship pass (full harvests only; default on): per key-ish/text column a KMV bottom-k sketch (`k` default `256`, error ~1/√k) is computed during the snapshot — Athena in ONE columnar scan per table via `min(DISTINCT from_big_endian_64(xxhash64(...)), k)` (both tokens matter: `min(x, n)` is a row-level order statistic so a multiset sketch collapses on fact-side FKs, and Trino's `xxhash64` returns VARBINARY so without the decode every cell is unparseable and the nominator silently disables itself), Redshift one cheap per-column query via `FNV_HASH` — and all cross-table column pairs are compared in memory; pairs whose estimated value containment ≥ the threshold (default `0.5`) are nominated into the SAME probe/verdict pipeline as name/role candidates (`via="sketch"` note on the sheet). This is what catches renamed keys (`cds` = `cdscode`). Guardrails: the CONTAINED side must exceed the enum-domain size (a tiny domain is inside everything — the dense-integer trap) and must NOT be its own table's PK (bare `id` or a self-naming key — FKs point AT PKs, so a PK contained elsewhere is the dense-surrogate coincidence: 301 junk `id↔id` nominations on MusicBrainz), containment INTO a dense INT PK additionally requires the contained column to NAME the containing table (`artist` → `artist.id`, `begin_area` → `area.id` — a 1..N surrogate numerically contains every smaller int column, so values alone prove nothing there; hash/UUID PKs and non-PK containers like `cdscode` stay name-free since sparse domains don't collide by chance), same-named holder pairs collapse toward the column's HOME table exactly like the name source (`tag` in fifteen `*_tag` tables pairs each holder with `tag`, never pairwise; widely-shared same-named columns with no home are refused), max `12` columns sketched per table (eligibility is TYPE-based — int/text families only, name-blind; under the cap columns RANK by name tier first, then by the profile pass's per-column stats — high-distinct, low-null first, with columns whose OBSERVED distinct count is at or under the enum-domain floor excluded outright: the guardrails refuse their nominations anyway, so a slot spent on them buys nothing), dedup against name/role nominations, and the shared budget/size gates apply — tables MEASURED over the gate are never sketched, while UNKNOWN-size tables get bounded LAST-RESORT attempts (visited after every measured table, each capped by `OKF_HARVEST_REL_QUERY_TIMEOUT_S`, abandoned after two consecutive failures; a timed-out sketch yields NOTHING, never wrong evidence, and a nomination whose probe the size gate still refuses persists as a no-verdict NOMINATED sheet routing the author to a live `validate_join`), and a sketch is always its OWN full scan of just the sketched columns, never the profile pass's row sample (per-VALUE facts don't survive row sampling; a sampled sketch would publish deflated containment that reads as "unrelated"). Sketch nominations are the one candidate source that does not re-enumerate from the catalog, so full runs persist them in `relationships/candidates.json` and incremental/cross runs (which never sketch) revalidate + merge them — without it the first non-full run would silently drop every sketch-discovered sheet. Known blind spot: key pairs whose distinct counts differ by more than ~`k`/8 are beyond the sketch's resolving power — counted in the harvest log, falls back to live probing |
| `OKF_HARVEST_BEDROCK_MAX_ATTEMPTS` | botocore `retries.max_attempts` in adaptive mode (default `5`); retries transient throttles and timeouts instead of failing the run |
| `OKF_BENCHMARK_MAX_CONCURRENCY` | how many benchmark solver ReAct loops (and judge reviews) run at once in a Benchmark Studio run (default `10`). Its own `asyncio.Semaphore` — each solver is one in-flight model request at a time, so this is the peak concurrent Bedrock requests from the benchmark. Raise on generous quota, lower on `ThrottlingException`. `mode: "benchmark"` runs only. |
| `OKF_QGEN_MAX_CONCURRENCY` | peak concurrent question-author agents in a `mode: "generate_questions"` run (default `4` — authors are long-lived explorers, not one-question solvers). Gold validation reuses the grader's `OKF_BENCHMARK_GRADER_TIMEOUT_S`/`_MAX_ROWS` caps so a generated gold can never be DISCARDED at grading time. |
| `OKF_BENCHMARK_ATHENA_CONCURRENCY` | how many benchmark grading queries (gold/predicted SQL EX executions) run against Athena at once (default `15`); size under the Athena workgroup's concurrent-DML limit. `mode: "benchmark"` runs only |
| `OKF_BENCHMARK_GRADER_TIMEOUT_S` | per-query timeout in seconds for the SQL EX grader's Athena executions (default `60`). A timed-out query is best-effort cancelled (`stop_query_execution`) so it doesn't keep holding a workgroup slot; the timeout classifies as TRANSIENT (retried with backoff, never memoized). Grading only — the harvest's own `sample_rows`/`run_sql` keep their own timeout. `mode: "benchmark"` runs only |
| `OKF_BENCHMARK_GRADER_MAX_ROWS` | row cap on one grading query's result set (default `50000`). Past the cap collection stops and the outcome is classified — gold → DISCARDED ("gold result exceeds N rows"), prediction → FAIL — instead of buffering an unbounded result. Grading only. `mode: "benchmark"` runs only |
| `OKF_INCREMENTAL_ICEBERG_COMMITS` | (incremental) how the change handler treats an Iceberg **data commit** — a Glue version bump with an EMPTY column diff, no changed partitions, and a version delta of exactly one (every Iceberg write bumps the version by swapping `metadata_location`, so per-commit re-harvests on a busy table would run continuously). Default `skip`: record the new version (so the nightly reconcile doesn't re-detect it) and absorb the event — Iceberg table docs then refresh on schema changes and full harvests, not on data writes. `review` restores the old always-invoke behavior. A wider version gap, a non-empty diff, or a never-seen table always invokes (a dropped event could hide a schema change the two-latest diff can't see); non-Iceberg version bumps are untouched — a Hive property-only bump still re-reviews |
| `OKF_POLICY_BUILD_ENABLED` | (harvest, incremental) `"true"` → author and refresh policy documents from the wiki (default `false`). The runner's post-complete follow-on build and the rebuild authority both no-op when unset — an ALREADY-authored document keeps being usable, since usability is decided by the row's `ar_build_status` + fingerprint, not by this flag. Set from `var.enable_policy_build` |
| `OKF_COMPUTATIONS_ENABLED` | (consumption MCP + Control API) `"true"` → `run_computation` EXECUTES (Athena / Redshift Data API); unset/false → it returns the rendered SQL with a note, and no engine client is built (the IAM grant is absent too). From `var.enable_attested_computations`, default **false** — machine-vended MCP creds must not gain source-data read from a routine apply. The CHAT surface deliberately ignores this flag: its `run_computation` is ALWAYS bound (the sanctioned path never costs the raw-SQL opt-in) and EXECUTES only when `var.enable_chat_sql` granted the clients — without them it returns the rendered SQL un-executed |
| `OKF_COMPUTATION_MAX_ROWS` / `OKF_COMPUTATION_TIMEOUT_S` | `run_computation` result-row cap (default `200`; the rest truncate with `truncated: true`) and per-execution timeout in seconds (default `120`, best-effort engine cancel on expiry). Row cap set from `var.computation_max_rows`; the timeout is env-read only |
| `OKF_USER_POOL_ID` | Cognito user pool id (the Control API vends and revokes M2M app clients in this pool) |
| `OKF_MCP_SCOPE` | the custom scope (`okf-mcp/invoke`) granted to vended M2M clients; must match the consumption authorizer's `allowed_scopes` |
| `OKF_HARVEST_LOG_GROUP` | the harvest runtime's CloudWatch log group the Control API reads to serve the live step feed (`GET /harvest/{domain}/{dataset}/events`). Derived by Terraform as `/aws/bedrock-agentcore/runtimes/<runtime-id>-DEFAULT` (overridable via `var.harvest_log_group`). Unset/incorrect → the feed returns an empty batch; status polling is unaffected |
| `OKF_WEB_SEARCH_ENABLED` | (chat runtime) `"true"` → offer the agent the public `web_search` tool. Set from `var.enable_web_search`. Requires `OKF_WEB_SEARCH_GATEWAY_URL` too: with either missing the tool is simply not wired (and the role carries no `bedrock-agentcore:InvokeGateway` grant anyway) |
| `OKF_WEB_SEARCH_GATEWAY_URL` | the AgentCore Gateway MCP endpoint fronting the built-in `web-search` connector (`https://<gateway-id>.gateway.bedrock-agentcore.us-east-1.amazonaws.com/mcp`). `/mcp` is appended if absent. The runtime speaks MCP JSON-RPC to it (`initialize` → `tools/call`) signed with SigV4 |
| `OKF_WEB_SEARCH_REGION` | region the gateway lives in, i.e. the region `web_search`'s SigV4 is signed for (default `us-east-1`). **Independent of `AWS_REGION`** — the web-search connector is offered in `us-east-1`/`eu-west-1`/`ap-northeast-1` only (`var.web_search_region` picks one), so a query may leave the deployment's region (it never leaves AWS) |
| `OKF_WEB_SEARCH_TOOL_NAME` | the gateway-side tool name, `<target-name>___WebSearch` (AgentCore prefixes every tool with its target's name, joined by THREE underscores). Set by Terraform to save a round trip; empty → the runtime discovers it via `tools/list` and caches it |
| `OKF_WEB_SEARCH_MAX_RESULTS` | default results per search when the agent doesn't pick a count via the tool's `max_results` arg (default `10`; 1–25). The connector ranks by relevance — the agent steers time through the query text (and, with the filters flag below, the date bounds), reading each result's `publishedDate` |
| `OKF_WEB_SEARCH_FILTERS_ENABLED` | `"true"` → `web_search` also offers the connector-1.2.0 request-level filter args (`published_after`/`published_before`, `include_domains`/`exclude_domains`). Set by Terraform **through the target's version-pin step** (`web_search.tf`), never by hand: a pre-1.2.0 target does not reject a `filters` argument, it **silently ignores** it, so an ungated runtime would hand the model constraints that quietly don't apply. Unset → the tool keeps the plain `query`+`max_results` surface |
| `OKF_CHAT_MEMORY_ID` | (chat runtime + Control API) the AgentCore Memory resource id for long-term chat memory (see "Long-term chat memory"). Set from `var.enable_chat_memory ? chat_memory_id : ""` — empty disables the feature end to end: the runtime builds no memory client (no recall, no event writes) and the Control API's `/memory*` routes return 404. The extraction/consolidation prompts + model live on the DURABLE stack's strategy resource (`var.chat_memory_model`), not in env |
| `OKF_CHAT_MEMORY_DEFAULT_ON` | (chat runtime + Control API) what a MISSING per-user memory switch row means: `true` (default) = opt-out — memory on until the user switches it off; `false` = opt-in — off until explicitly enabled on the Memory page. Set from `var.chat_memory_default_on` on BOTH services so the switch the page shows agrees with what the runtime does; rows users already set are never affected |
| `OKF_CHAT_GUARDRAILS_GATE_ENABLED` | (chat runtime, env-read — not Terraform-plumbed) default `true` → `read_page` on any page of a dataset is DENIED at the tool boundary until that dataset's `references/usage_guardrails` has been read in the thread (`chat.guardrails_gate.GuardrailsGateMiddleware`). Per-dataset marks live in CHECKPOINTED agent state (`guardrails_read` channel, dict-merge reducer), so resumes remember; the guardrails read itself always passes and marks on any completed attempt (a guardrails-less legacy bundle can't lock out). Browse/search tools are never gated. `"false"`/`"0"`/`"no"`/`"off"` disables |
| `OKF_CHAT_POLICY_CHECK_ENABLED` | (chat runtime) `"true"` → the mid-turn policy checks may be armed per run via `features: ["sql", "policy:*"]` (default `false`). Unset → no checker is ever constructed, whatever the client sends — the master gate above the per-run opt-in, set from `var.enable_policy_checks` |
| `OKF_CHAT_POLICY_CHECK_MODEL` | (chat runtime) the policy checks' model id, serving BOTH the curated-question rewrite (no reasoning pass — extraction) and the JUDGE fleets. Default `global.anthropic.claude-sonnet-5` — judges run classifier-style on every family (thinking off + temperature 0 on Anthropic, reasoning `"none"` on openai.* ids like `openai.gpt-5.6-terra`, forced `report_violations` either way); from `var.chat_policy_check_model`, which also drives the chat role's Mantle grants, so an openai.* value needs no extra wiring. Code-level companions (env-read, not Terraform-plumbed): `OKF_CHAT_POLICY_SHARD_SIZE` (default `10` — policies per mini-judge), `OKF_CHAT_POLICY_QUERY_TIMEOUT_S` (default `60` — residual wait after the query returns) and `OKF_CHAT_POLICY_QUERY_MAX_PER_TURN` (default `3` — judged analytical queries per turn). Deploy-time only — deliberately NOT validated against `OKF_CHAT_MODEL_CATALOG`, which is the trust boundary for CLIENT-supplied models |
| `OKF_POLICY_PREPROCESS_MODEL` | (harvest runtime) model id for the `policies.yaml` AUTHORING AGENT (`harvest.ar_author` — full reasoning; policy distillation is judgment work). Default `global.anthropic.claude-sonnet-5`, from `var.policy_preprocess_model`, which also drives the harvest role's Mantle grants. Code-level companions (env-read, not Terraform-plumbed): `OKF_POLICY_AUTHOR_EFFORT` (default `high`), `OKF_POLICY_AUTHOR_THINKING_BUDGET` (pre-adaptive models like Haiku 4.5 — e.g. `48000`), and `OKF_POLICY_MAX_RULES` (default `60` — the author gate's policy-count BACKSTOP against enumeration pathology; the prompt's proportionality guidance, not this cap, is what sizes a document to its dataset). The incremental Lambda runs NO models: authoring dispatches to the runtime |
| `VITE_CHAT_POLICY_CHECK` | (UI, from `ui_env`) shows the composer's Policy feature (the "+" menu field with Computational / Behavioural / Strict) and the Reasoning sidebar page. Defaults ON — only the literal `"false"` hides them (an unset var must not silently drop the affordance). A DISPLAY gate only, same pattern as `VITE_CHAT_SQL_ENABLED`; the server-side `OKF_CHAT_POLICY_CHECK_ENABLED` + feature normalization are the real boundary |

## HTTP and auth

- Control API and MCP requests carry `Authorization: Bearer <Cognito token>`.
- The API Gateway HTTP API JWT authorizer uses audience = app client id, issuer =
  `https://cognito-idp.<region>.amazonaws.com/<poolId>`.
- The consumption MCP AgentCore authorizer uses
  `discoveryUrl = <issuer>/.well-known/openid-configuration`. Inbound trust is
  scope-based (`allowedScopes = ["okf-mcp/invoke"]`), not a client allowlist, so a
  newly vended machine client is accepted with no infra change. `allowedAudience`
  is unusable here because Cognito M2M `client_credentials` access tokens carry no
  `aud`.

### MCP machine credentials (apps and agents)

- An `okf-mcp` resource server defines the `invoke` scope, giving the full scope
  string `okf-mcp/invoke`. The web SPA also carries this scope, so human sessions
  pass the same authorizer check.
- The Control API vends credentials as Cognito M2M app clients
  (`client_credentials` grant, `GenerateSecret=true`, scope `okf-mcp/invoke`):
  `POST /credentials {name}` returns `{client_id, client_secret}` once;
  `GET /credentials` returns metadata from the registry;
  `DELETE /credentials/{client_id}` deletes the app client and revokes it
  immediately. This needs IAM `cognito-idp:{Create,Delete,Describe}UserPoolClient`
  on the pool.
- To get a token, an app POSTs to the Cognito token endpoint with HTTP basic auth
  `client_id:client_secret` and body
  `grant_type=client_credentials&scope=okf-mcp/invoke`, then sends the resulting
  access token as `Authorization: Bearer <token>` to the MCP server. Tokens are
  short-lived (60 minutes) and meant to be cached; the token endpoint is capped at
  150 RPS per account and Region.
