What kind of metrics and non-functional requirements I can be asked in a RAG system ?

I would evaluate a RAG system at three levels.

First is retrieval quality. I would measure Recall@K to determine whether the system retrieved the necessary evidence, Precision@K to measure how much of the retrieved context was relevant, and MRR or nDCG to evaluate ranking quality.

Second is answer quality. Important metrics include answer correctness, relevance, completeness, groundedness, citation accuracy, and hallucination rate. I would also measure unanswerable-question detection, because a reliable system must refuse to answer when the available evidence is insufficient.

Third is production reliability. My non-functional metrics would include p50, p95, and p99 latency; time to first token; throughput; availability; timeout and error rates; cost per successful query; index freshness; ingestion success rate; and mean time to recovery. For security-sensitive systems, I would also measure unauthorized-document retrieval, prompt-injection success, and sensitive-data leakage, with a target of zero.

For example, initial targets might be Recall@10 above 90%, groundedness and citation accuracy above 95%, hallucination below 2%, p95 latency below three seconds, and availability above 99.9%.

I would create a representative evaluation dataset containing answerable, unanswerable, ambiguous, adversarial, and permission-sensitive questions. I would run it as an offline regression suite before each release and combine it with production monitoring and sampled human reviews. Separating retrieval, generation, and infrastructure metrics allows us to identify exactly where a RAG failure occurred.