> [!IMPORTANT]
> **AI Assist Note (Knowledge Heritage)**:
> This document is part of the "Sovereign Reality" documentation.
> - **@docs ARCHITECTURE:Documentation**
> - **Failure Path**: Information drift, legacy terminology, or documentation mismatch.
> - **Telemetry Link**: Cross-reference with `execution/parity_guard.py` results.
>
> ### AI Assist Note
> Automated governance and architectural tracking.
>
> ### 🔍 Debugging & Observability
> Traceability via `parity_guard.py`.

# 🧠 Knowledge & Memory: The Hybrid "Split-Brain" Architecture

**Intelligence Level**: High (ECC Optimized)  
**Last Hardened**: 2026-04-27 (MFS & Vector Parity)  

> **@docs ARCHITECTURE:Retrieval**

Tadpole OS implements a powerful **Split-Brain Memory** system that separates deterministic data from high-dimensional semantic search.

---

## 🏗️ Memory Strategy: Deterministic vs. Semantic

Intelligence requires both exact record-keeping and fuzzy contextual intuition.

| Layer | Technology | Data Type | Purpose |
|:--- |:--- |:--- |:--- |
| **Deterministic** | **SQLite (sqlx)** | Logs, Budgets, Registry, Metadata, OKF Metadata | Legal Audit & Operational Truth |
| **Semantic** | **LanceDB (Vector)**| Embeddings (768-dim), RAG snippets | Contextual Awareness & Discovery |

---

## 🛰️ The Data Ingestion Pipeline (4-Phase Model)

The engine converts raw documents into "Intelligence Assets" through a disciplined background pipeline.

### Phase 1: Multi-Factor Scoring (MFS)
Tadpole OS doesn't just rely on raw vector similarity. It uses a **Multi-Factor Scoring (MFS)** engine to rank context snippets based on three primary signals:
1.  **Vector Similarity**: Uses cosine distance (via LanceDB) for high-dimensional semantic mapping.
2.  **Mission Affinity**: Contextually boosts results that match the active `Mission-ID`, ensuring relevant domain knowledge is prioritized.
3.  **Temporal Recency**: Applies a decay factor to older memories, ensuring the agent uses the most up-to-date information while retaining long-term historical context.

The MFS engine reranks these signals into a unified relevance score (0.0 - 1.0) before passing the top-K snippets to the LLM.

### Phase 2: Automated Connectors (`connectors.rs`)
Data sources are automatically synchronized via background workers:
- **`IngestionWorker`**: A Tokio daemon that crawls configured folders.
- **Incremental Sync**: Uses MD5-checksums to avoid re-embedding unchanged files.
- **Connectors**: Supports File-system watching with absolute path validation.

### Phase 3: Deterministic SOP Workflows (`workflows.rs`)
Beyond fuzzy RAG, the engine can execute **Structured SOPs**:
- **Markdown-to-State**: Parses `.md` files into executable steps.
- **Sequential Execution**: Ensures compliance-heavy tasks follow a strict order of operations.

### Phase 4: Layout-Aware Parsing & Dual-Write Atomicity (`store.rs`)
- **Transactional Dual-Write Atomicity**: `add_entry` wraps SQLite metadata inserts in an explicit transaction (`pool.begin()`) and commits only after LanceDB vector storage succeeds. If vector storage fails, the SQLite transaction rolls back, eliminating orphaned "ghost" metadata entries.
- **Vector Predicate Sanitization**: `get_peers` enforces strict input ID character validation (`c.is_alphanumeric() || c == '-' || c == '_'`) before constructing LanceDB filter predicates, preventing injection attacks.
- **Bounded OKF Playbook Cache (`okf_gate.rs`)**: Enforces maximum cache capacity (`MAX_CACHE_ENTRIES = 256`) with automatic TTL eviction for playbook requirements, eliminating static map memory leaks during large-scale swarm runs.
- **Zero-Allocation Requirement Parser**: Hot-loop allocation-free requirement extraction using precomputed static matchers (`pattern_colon`, `pattern_space`).

---

## 🧪 Vector Memory Management

### LanceDB + Apache Arrow
- **Zero-Copy Performance**: Uses Apache Arrow memory layout for ultra-low latency searches.
- **Orphan Sweeping**: A background daemon purges mission-specific temporary vector "scopes" once tasks are complete to prevent disk bloat.

### Local-First Embeddings
By default, the engine utilizes the **BGE-Small-EN-v1.5** model via ONNX for zero-latency local vectorization, ensuring no data ever leaves the bunker for memory processing.

### Swarm Context Compaction
To avoid token bloat and CPU load spikes during long-running agent missions, dialogue compaction is performed dynamically:
- **$O(N)$ Linear Compactor**: Iteratively filters dialogue history in linear time, stripping large codeblocks from older messages while preserving the last 4 reasoning turns fully intact.
- **Resilient Fallback**: Falls back to a regex-based parser if LLM summarization fails, guaranteeing zero loss of structural turn coherence.

[//]: # (Metadata: [Knowledge_Memory])
