[繁體中文](README.md) | **English**

# AI Knowledge Retrieval Benchmark: Hybrid RAG vs. Graph RAG vs. OKF-Wiki on Taiwan Building Regulations

End-to-end benchmark of three AI retrieval architectures on the Taiwan *Building Technical Regulations: Building Design and Construction Edition* (389 articles). Tested with 15 questions covering multi-chapter reasoning, numerical thresholds, and exception clauses.

---

## 📊 Leaderboard (Benchmark V4.1)

Evaluated with 5-dimension RAGAS + human-verified Ground Truth:

| System | Context Precision | Context Recall | Faithfulness | Answer Relevancy | Answer Correctness | **5D Avg** |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| 🏆 **OKF-Wiki** (Markdown Tree Navigation) | 0.52 | 0.59 | 0.56 | **0.98** | **0.89** | **0.71** |
| 🥈 **Hybrid RAG** (Vector + BM25) | **0.58** | **0.69** | **0.58** | 0.71 | 0.62 | 0.64 |
| 🥉 **Graph RAG** (Knowledge Graph Triplets) | 0.29 | 0.40 | 0.25 | 0.57 | 0.51 | 0.40 |

| System | Strengths | Limitations |
| :--- | :--- | :--- |
| OKF-Wiki | Highest correctness (0.89); strong cross-chapter synthesis | Multi-step directory traversal; slower on broad queries |
| Hybrid RAG | Highest recall (0.69); lowest build cost; ~0.9s latency | Chunk boundaries can break legal context; introduces noise |
| Graph RAG | Fastest queries (~0.6s); precise entity-numeric constraints | High upfront extraction cost; granular exception clauses at risk |

> Full transcripts: [V4.1 Benchmark Report](docs/benchmark_v4_report.md) | Historical: [V3 Report](docs/benchmark_v3_report.md)

---

## 📂 Document Index

| Document | Description |
| :--- | :--- |
| [📋 Q&A Reference Sheet](docs/qa_reference.md) | 15 evaluation questions + human Ground Truth answers, human-readable |
| [V4.1 Detailed Benchmark Report](docs/benchmark_v4_report.md) | Per-question scores, retrieved context, and 3-stage evaluation transcripts |
| [Three-Architecture Infrastructure Guide](docs/three_rag_infrastructure_guide.md) | Build workflow, model selection, and Mermaid architecture diagrams |
| [V4 Upgrade Roadmap](docs/v4_upgrade_plan.md) | Technical specs for next-phase upgrades of all three systems |
| [Project Status Handoff](docs/PROJECT_STATUS.md) | Quick-start for new Agents — current state and next tasks |
| [V3 Baseline Report](docs/benchmark_v3_report.md) | Earlier human-scored evaluation (historical reference) |

---

## 📐 RAGAS 5-Dimension Metrics

1. **Context Precision** — Signal-to-noise ratio of retrieved text; irrelevant articles lower the score.
2. **Context Recall** — Whether all required articles and exception clauses were retrieved.
3. **Faithfulness** — Answer must be 100% grounded in retrieved text; using parametric memory is penalized.
4. **Answer Relevancy** — Answer directly addresses the question without deviation.
5. **Answer Correctness** — Numerical and semantic alignment with human Ground Truth.
6. **Abstention Penalty** — "No information" responses force Faithfulness / Relevancy / Correctness to 0.0.

---

## 🔍 Implementation Stack

1. **Hybrid RAG** — ChromaDB (dense) + Rank-BM25 (sparse) + RRF reranking. Chunks follow the native legal hierarchy (*Chapter / Section / Article / Paragraph / Subparagraph*) instead of arbitrary token limits.
2. **Graph RAG** — LLM extracts triplets from 389 articles into a NetworkX graph. Vector search finds entry nodes; 1-hop expansion retrieves linked regulation texts.
3. **OKF-Wiki** — Pure Markdown file tree + `_index.md` Maps of Content + bidirectional links. Agent navigates with file-reading tools only — no vector index.

---

## 🚀 Benchmark Pipeline

```
[15 Questions] → 45 Generator Subagents → retrieve + answer → Answer JSONs
[Ground Truth] → 45 Evaluator Subagents → Initial Review → Strict Verification → Consensus → RAGAS Scores
                                                                              ↓
                                                                    V4.1 Benchmark Report
```

Three stages: **Generation → Evaluation (3-stage audit per question) → Aggregation (script parses logs, outputs report)**

---

## 🎯 Architecture Selection

| Use Case | Recommended | Reason |
| :--- | :--- | :--- |
| Prototyping, PoC, budget-constrained | **Hybrid RAG** | Fastest setup, recall 0.69, no graph extraction cost |
| High-compliance (legal, medical, financial) | **Graph RAG** | Precise entity constraints; maintain raw text mappings |
| Cross-chapter research, enterprise assistants | **OKF-Wiki** | Correctness 0.89; Markdown maintained directly by domain experts |

---

## 🔮 Roadmap (details: [V4 Upgrade Plan](docs/v4_upgrade_plan.md))

1. **Hybrid RAG — Contextual Chunk Enrichment (Phase 1, priority)**: Inject chapter context into each chunk at index time; add Self-Querying Metadata Filters and Parent-Child Chunk Strategy.
2. **Graph RAG — Graph-Vector Dual Entry**: Vectorize node descriptions for fuzzy seed lookup, then expand via graph traversal; pre-compute Leiden community summaries for global queries.
3. **OKF-Wiki — Hybrid Routing**: Vector lookup drops Agent into the correct directory; Agent handles remaining navigation.
4. **Long-term: Multi-Architecture Query Router** — Route by question type (exact lookup / cross-chapter synthesis / numerical constraints).
