A RAG and Reciprocal Rank Fusion-Based Chatbot for New Student Admission (SPMB) Information Services at SMK Muhammadiyah 1 Wonosobo

Iqbal Ali Ar-Ridho; Nahar Mardiyantoro; Saifu Rohman

Authors

Iqbal Ali Ar-Ridho Informatics Engineering Study Program, Faculty of Engineering and Computer Science, Universitas Sains Al Qur’an Jawa Tengah di Wonosobo, Indonesia
Nahar Mardiyantoro Informatics Engineering Study Program, Faculty of Engineering and Computer Science, Universitas Sains Al Qur’an Jawa Tengah di Wonosobo, Indonesia
Saifu Rohman Informatics Engineering Study Program, Faculty of Engineering and Computer Science, Universitas Sains Al Qur’an Jawa Tengah di Wonosobo, Indonesia

Keywords:

chatbot, Natural Language Processing, Retrieval-Augmented Generation, Reciprocal Rank Fusion, SPMB

Abstract

The New Student Admission (SPMB) information service at SMK Muhammadiyah 1 Wonosobo faces a high volume of inquiries—987 registrants were recorded in 2025—handled through seven official channels limited to working hours, resulting in repetitive questions and inconsistent answers. This study aims to develop a Natural Language Processing (NLP)-based chatbot using the Retrieval-Augmented Generation (RAG) and Reciprocal Rank Fusion (RRF) methods to automate the SPMB information service on the school’s official website. The system was built using the Prototype development method. The designed RAG pipeline adds three retrieval-enhancement mechanisms on top of naive RAG, namely query expansion, RRF with an asymmetric 4:1 weighting, and cross-encoder reranking, with Gemini 3 Flash Preview as the answer generator and a knowledge base of 44 official school text documents. Testing was conducted through Black Box Testing and RAGAS evaluation measuring faithfulness, answer relevancy, and context precision. Black Box Testing of 17 scenarios achieved a 100% functional success rate. RAGAS evaluation on the full (Enhanced) configuration obtained faithfulness 0.9812, answer relevancy 0.9146, and context precision 0.7991. Compared to the Baseline configuration, answer relevancy increased by 12.40% and context precision by 14.16%, while faithfulness remained in the “Very Good” category. This study concludes that applying these three techniques positively contributes to document relevance and the answer quality of the SPMB information chatbot.

References

D. Khurana, A. Koli, K. Khatter, and S. Singh, "Natural language processing: state of the art, current trends and challenges," Multimedia Tools and Applications, vol. 82, no. 3, pp. 3713–3744, 2023, doi: 10.1007/s11042-022-13428-4.

R. R. Rafi, A. Setiawan, A. A. Rosadi, and A. Sanjaya, "Rancang bangun chatbot penerimaan murid baru berbasis cosine similarity pada SMK Intensif Baitussalam," Prosiding Seminar Nasional Teknologi dan Sains, vol. 4, 2025.

F. M. Naufan, "Pengembangan chatbot AI berbasis NLP dengan metode Retrieval-Augmented Generation (RAG) untuk otomatisasi layanan informasi pajak dan keuangan daerah di website BPPKAD Kabupaten Wonosobo," Undergraduate thesis, Universitas Sains Al-Qur’an Jawa Tengah di Wonosobo, 2025.

Z. Rackauckas, "RAG-Fusion: a new take on retrieval-augmented generation," International Journal on Natural Language Computing, vol. 13, no. 1, pp. 37–47, 2024, doi: 10.5121/ijnlc.2024.13103.

Y. Gao et al., "Retrieval-augmented generation for large language models: a survey," arXiv preprint, 2023, doi: 10.48550/arXiv.2312.10997.

P. Lewis et al., "Retrieval-augmented generation for knowledge-intensive NLP tasks," in Advances in Neural Information Processing Systems (NeurIPS), 2020, doi: 10.48550/arXiv.2005.11401.

S. Es, J. James, L. Espinosa-Anke, and S. Schockaert, "RAGAS: automated evaluation of retrieval augmented generation," in Proc. 18th Conf. European Chapter of the ACL (EACL): System Demonstrations, 2024, doi: 10.48550/arXiv.2309.15217.

R. S. Pressman and B. R. Maxim, Software Engineering: A Practitioner’s Approach, 9th ed. New York: McGraw-Hill Education, 2020.

J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, and Z. Liu, "M3-Embedding: multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation," in Findings of the ACL 2024, 2024, doi: 10.48550/arXiv.2402.03216.

M. Douze et al., "The Faiss library," arXiv preprint, 2024, doi: 10.48550/arXiv.2401.08281.

G. V. Cormack, C. L. A. Clarke, and S. Büttcher, "Reciprocal rank fusion outperforms Condorcet and individual rank learning methods," in Proc. 32nd Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, 2009, pp. 758–759, doi: 10.1145/1571941.1572114.

L. Wang, N. Yang, and F. Wei, "Query2doc: query expansion with large language models," in Proc. 2023 Conf. on Empirical Methods in Natural Language Processing (EMNLP), 2023, doi: 10.48550/arXiv.2303.07678.

C. Li, Z. Liu, S. Xiao, Y. Shao, and D. Lian, "Llama2Vec: unsupervised adaptation of large language models for dense retrieval," in Proc. 62nd Annual Meeting of the ACL, 2024, doi: 10.48550/arXiv.2312.15503.

S. Nidhra and J. Dondeti, "Black box and white box testing techniques – a literature review," International Journal of Embedded Systems and Applications, vol. 2, no. 2, pp. 29–50, 2012, doi: 10.5121/ijesa.2012.2204.

G. Samudra, A. Turmudi Zy, and Ermanto, "Implementasi Retrieval-Augmented Generation (RAG) pada chatbot layanan informasi kesehatan," JSAI: Journal Scientific and Applied Informatics, vol. 8, no. 1, 2025, doi: 10.36085/jsai.v8i1.7678.