A retrieval evaluation dataset contains queries, a document collection, and relevance judgments that identify which documents answer each query. Queries should cover varied intents, wording, and difficulty. The evaluation set should include lexical matches, paraphrases, multi-hop questions where appropriate, and unanswerable questions if abstention is evaluated. Relevance labels should be checked independently where possible. A tiny or repetitive dataset can produce misleadingly perfect metrics.
