# Evaluation Plan

## Goals

Evals should prove the agent is grounded, useful, safe, and clearly agentic.

## Deterministic Tests

Unit tests:

- FastAPI route smoke for health, analysis, Arabic analysis, upload validation, and strategy.
- Voice gateway WebSocket smoke for session readiness, Arabic transcript turn, A2A audit status event, dashboard patch, session end, and unknown-event errors.
- OKF parser
- OKF validator
- Citation extraction
- Internal link scanning
- Tender metadata loading
- Bidder profile validation
- Scenario scoring
- Upload validation
- Rejecting files over 4 MB
- Upload cleanup
- Language utilities
- Sanitization helpers
- MCP schemas
- Voice WebSocket schema validation
- Voice session reducer
- Transcript sanitization

Integration tests:

- Frontend API-first analysis fallback is covered by smoke checks and should get Playwright interaction coverage next.
- MCP tools return expected structures.
- Retrieval works over sample OKF.
- Bidder profile matching works against requirements.
- Strategy simulator output shape is stable.
- Clarification question output shape is stable.
- A2A Evidence Audit Agent is reachable locally.
- Router calls A2A before final response.
- Voice gateway accepts session start and cleanup.
- Voice transcript routes to Router workflow.
- Upload endpoint rejects >4 MB before parsing, parses text-based PDFs without claiming OCR for scanned files, and supports mocked/credential-backed JPG/PNG/WEBP vision extraction.
- Frontend smoke renders English and Arabic.

## Trajectory Tests

Expected paths:

- router -> MCP -> retrieval -> synthesis -> A2A audit -> final
- router -> profile -> compliance -> synthesis -> A2A audit
- router -> risk -> clarification questions -> synthesis -> A2A audit
- intake -> OKF validation -> retrieval
- strategy simulator -> scenario service -> synthesis
- voice transcript -> router -> retrieval -> synthesis -> A2A audit -> voice response
- synthesis -> A2A audit fail -> router -> targeted specialist/retrieval -> synthesis -> A2A audit

Assertions:

- Expected tools are called.
- A2A audit is called before final output.
- Evidence is retrieved before claims.
- Unsupported claims are rejected or revised.
- Missing citations trigger the loop.
- Retrieval only rejoins loop for missing/weak evidence.
- Max loop count is respected.
- Voice responses remain concise and grounded.

## Agents CLI Eval Cases

Current `tests/eval/datasets/basic-dataset.json` includes five capstone cases:

- Clear bid
- Arabic mode report
- Clarification questions required
- Voice transcript asks "Can we bid?"
- Voice transcript asks for top risks
- Prompt injection in tender text

Planned expansion after core cases pass:

- Clear no-bid
- Missing certification
- Ambiguous eligibility
- Timeline too tight
- High commercial risk
- Missing citation triggers A2A revision
- A2A audit fails unresolved claim after max loop
- Scenario change flips recommendation
- Voice transcript asks to switch to Arabic

The current eval config uses `tenderlens_capstone_contract`, a local deterministic metric in `tests/eval/metrics.py`, plus `agent_turn_count`. Managed/built-in LLM judge metrics should be added after Gemini/Vertex credentials are configured.

Current status:

- Dataset JSON validates locally.
- Local metric compiles and has unit coverage.
- `agents-cli eval generate --dataset tests/eval/datasets/basic-dataset.json` still needs a credential/configured runtime pass; the latest local attempt exceeded the 120 second timeout after dependency sync started.

## Frontend QA

Use Playwright or equivalent for:

- Desktop, tablet, mobile
- English LTR
- Arabic RTL
- No text overlap
- No button overflow
- Evidence drawer usability
- OKF/cockpit sections visible
- Language switch preserves state
- Upload validation errors
- Clear 4 MB upload error
- A2A audit status visible
- Voice button visible
- Voice permission/error states
- Voice transcript layout
- Voice end/mute/interrupt controls
- Sanitization render test

## Manual Voice QA

Because real microphone/audio can be difficult to automate fully, manually verify:

- Voice mode starts only after button click.
- Browser permission prompt appears.
- User speaks a tender question.
- Transcript appears.
- Agent speaks response.
- User interrupts response.
- Evidence panel highlights relevant sources.
- User ends voice mode.
- Microphone stops.
- Typing remains usable.
- No raw audio files are created.
