Model reliability / Aug 2025
Disagreement-Aware RAG
Answer when grounded. Abstain when uncertain.
Predicts disagreement risk from generation and evidence signals and abstains when that risk is high.
~97%coverage
3–4%hallucination rate
What I built
- Built a disagreement-aware RAG system that abstains when predicted disagreement risk is high.
- Trained a lightweight logistic regression model over self-consistency variance (k-resampled generations), evidence overlap (ROUGE-L–style), and an token-level entropy
- Engineered a full-stack app that allows querying of questions on a test dataset and plots coverage vs. hallucination; achieved ~97% coverage @ 3–4% hallucination (τ≈0.55–0.60).
- Implemented a zero-shot entailment auditor with BART-MNLI to flag likely hallucinations in evaluation.
Explore the implementation
Source code, setup instructions, and project documentation are available in the repository.