Deliverable 4g · Performance Measurement & Drift Monitoring
Performance & Drift
KPIs, live performance dashboards, drift detection, and ROI — the GAO Monitoring principle, made operational. Models are watched continuously; when input distributions shift, the framework reacts.
Model performance · trailing 12 months
Drift detection · Mail-Theft Image Classifier
Population Stability Index (PSI) monitored against a 0.2 breach threshold.
What drifted
Peak holiday-season parcel volume introduced high proportions of new poly-mailer and oversized flat packaging types not well-represented in the original training set; the shift in packaging material textures and label placement patterns pushed PSI above the 0.20 breach threshold.
Automated response
Automated drift alert triggered a mandatory human-review escalation for all high-confidence theft flags; a targeted retraining job was queued using 60 days of recently labeled images; the retrained model was validated offline and promoted to production on 2026-04-11, restoring PSI to the stable band.
RAG Triad · retrieval-augmented generation quality
The RAG Triad evaluates the two failure modes of retrieval-augmented generation: retrieving the wrong context, and generating beyond what was retrieved. It is scored continuously on a sampled, human-graded evaluation set. For an oversight agency, groundedness is the load-bearing metric: an answer that cannot be traced to source evidence has no place in an audit or investigation.
Context Relevance
0.91question to retrieved context
Share of retrieved passages judged relevant to the query, guarding against reasoning over off-topic or stale material.
Groundedness
0.96retrieved context to answer
Share of answer claims fully supported by the retrieved context. The hallucination check, and the metric an inspector general weights highest.
Answer Relevance
0.93question to answer
Degree to which the final answer actually addresses the question asked rather than being evasive or partial.
Any answer scoring below the groundedness target is withheld and routed for human review rather than returned; ungrounded responses are logged for retraining.
Return on investment