An Empirical Study of Knowledge Graph–Augmented Retrieval for Hallucination Mitigation in University Academic Assistance Systems

Large Language Models (LLMs) used in university chatbots often create false or misleading answers—a problem known as "hallucination". When students ask about course prerequisites, credit transfers, examination rules, or graduation policies in open and distance learning systems, standard AI systems frequently confuse similar policies or guess missing details. Standard document-search methods (Vector Retrieval-Augmented Generation or Vector-RAG) help by pulling relevant text passages, but they still struggle when answers depend on interconnected rules spread across multiple handbook pages. We built and tested a Knowledge Graph–Augmented Retrieval (KG-RAG) system. This approach connects traditional keyword/vector search with a structured "knowledge map" (a Knowledge Graph) that explicitly links courses, degree tracks, examination regulations, and prerequisite dependencies. We constructed a real-world multi-institutional benchmark dataset containing 2,850 policy passages compiled from 20 leading open, distance, and online universities globally (including The Open University UK, IGNOU, BAOU, University of London, and ASU Online) along with a structured graph of 3,400 entities and 8,200 relations. We evaluated 220 verified student queries across three systems: a base LLM, a standard Vector-RAG system, and our hybrid KG-RAG framework. The proposed KG-RAG system achieved 92.0% factual accuracy, outperforming standard Vector-RAG (84.0%) and the base LLM (61.0%). Hallucination rates dropped from 16.0% in standard Vector-RAG to just 6.0% in KG-RAG (p<0.001, d=1.42). On complex multi-hop queries requiring multi-step reasoning across academic policies, KG-RAG achieved 87.0% accuracy compared to 74.0% for Vector-RAG and 43.0% for the base LLM. Integrating a structured knowledge map provides deterministic boundaries that stop AI from guessing based on simple word similarity. The system traces exact policy pathways, significantly lowering hallucination risks in higher education administrative support while remaining computationally efficient for institutional deployment.

Knowledge Graph–Augmented Retrieval, Hallucination Mitigation, Retrieval-Augmented Generation (RAG)

1. Baidoo-Ananu, D., & Owusu, E. (2023). Synthetic evaluation of conversational artificial intelligence in higher education systems. Journal of Educational Technology Systems, 51(4), 412–430. [Google Scholar] [Crossref]

2. Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Yan, Y., ... & Fung, P. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), 1–38. [Google Scholar] [Crossref]

3. Zhang, Y., & Dong, M. (2025). Empirical evaluation of hallucination metrics in Retrieval-Augmented Generation architectures. Computer Science Review, 55, 100680. [Google Scholar] [Crossref]

4. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., ... & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33, 9459–9474. [Google Scholar] [Crossref]

5. Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., ... & Wang, H. (2023). Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997. [Google Scholar] [Crossref]

6. Qian, L., Wang, Y., & Liu, Z. (2024). Limitations of vector chunking in domain-specific retrieval-augmented generation. IEEE Transactions on Knowledge and Data Engineering, 36(5), 2105–2119. [Google Scholar] [Crossref]

7. Es, S., James, J., Espinosa-Anke, L., & Schockaert, S. (2023). Ragas: Automated evaluation of retrieval augmented generation. arXiv preprint arXiv:2312.10997. [Google Scholar] [Crossref]

8. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., ... & Scialom, T. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. [Google Scholar] [Crossref]

9. Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, L., Wilie, B., ... & Fung, P. (2023). A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023. [Google Scholar] [Crossref]

10. Kang, M., & Lee, J. (2023). Dense passage retrieval and semantic vector search in specialized domain QA. Information Processing & Management, 60(3), 103280. [Google Scholar] [Crossref]

11. Hogan, A., Blomqvist, E., Cochez, M., d’Amato, C., de Melo, G., Gutierrez, C., ... & Zimmermann, A. (2022). Knowledge graphs. ACM Computing Surveys, 54(4), 1–37. [Google Scholar] [Crossref]

12. Pan, S., Luo, L., Wang, Y., Chen, C., Wang, J., & Wu, X. (2024). Unifying large language models and knowledge graphs: A roadmap. IEEE Transactions on Knowledge and Data Engineering, 36(7), 3180–3199. [Google Scholar] [Crossref]

13. Zhao, Y., Zhang, X., & Chen, H. (2023). Graph-based retrieval-augmented generation for structured factual reasoning. Proceedings of the Association for Computational Linguistics (ACL), 1420–1435. [Google Scholar] [Crossref]

14. Chen, H., & Xu, L. (2024). Knowledge graph-augmented retrieval for domain-specific large language models: A survey. IEEE Transactions on Knowledge and Data Engineering, 36(2), 845–861. [Google Scholar] [Crossref]

15. Agrawal, A., Sansford, R., & Taylor, P. (2024). GraphRAG: Leveraging graph-based efficiency to minimize hallucinations in LLM-driven document retrieval. ACL Anthology / GenAI K-1, 54–68. [Google Scholar] [Crossref]