Character-Driven LLMs: Training Role-Consistent Chatbots for Real-World Use

This paper addressed the challenge of persona drift in LLM-based conversational systems by implementing and comparing three persona enforcement approaches: prompt engineering, RAG with LangChain, and LoRA fine-tuning. Two prototype chatbots — Dr. Maya (counselor) and Prof. Alex (tutor) — were developed and tested across 22 categorized test prompts using an automated LLM-as-judge evaluation framework scoring ten persona consistency metrics. The principal findings are: (a) RAG-enhanced prompting achieves the strongest persona consistency among all tested approaches, with a 6.2% improvement in overall mean score (4.43 vs 4.17) and 58.3% improvement in character break resistance (4.75 vs 3.00); (b) Fine-tuning requires carefully curated, persona-aligned datasets, as generic empathy data produces models with casual, off-character communication styles regardless of model capacity (tested on both 1.1B and 3.8B parameter models); (c) Prompt engineering alone is effective for standard interactions (achieving 5.00 on normal conversation) but cannot prevent boundary violations under adversarial conditions (scoring 3.00 on character break resistance); and (d) A modular, hybrid framework combining prompt conditioning with retrieval-augmented persona memory is the most practical architecture for real-world deployment. Future work includes: developing curated professional counseling dialogue datasets for effective fine-tuning; implementing hybrid pipelines combining fine-tuned models with RAG retrieval; conducting user studies with real participants; and extending the framework to support multilingual personas and multimodal conversational systems.

[1] R. Chen, Y. Zhou, J. Wang, and Y. Liu, “The Oscars of AI Theater: A Survey on Role-Playing with Language Models,” arXiv preprint arXiv:2407.11484, 2025. [2] T. Brown et al., “Language Models are Few-Shot Learners,” in Advances in Neural Information Processing Systems (NeurIPS), 2020. [3] H. Touvron et al., “LLaMA: Open and Efficient Foundation Language Models,” arXiv preprint arXiv:2302.13971, 2023. [4] R. Chen, Y. Zhou, J. Wang, and Y. Liu, “Persona Vectors: Monitoring and Controlling Character Traits in Language Models,” arXiv preprint arXiv:2507.21509, 2025. [5] P. Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” in NeurIPS, 2020. [6] E. Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models,” in ICLR, 2022. [7] S. Zhang et al., “Personalizing Dialogue Agents: I Have a Dog, Do You Have Pets Too?” in Proc. ACL, 2018. [8] P. Y. Lo, A. D. Cheong, and S. Dow, “Noel: A Chatbot Persona to Support Children Designing for Others,” in Proc. ACM CHI, 2025. [9] B. Keum, S. Lee, and J. Kim, “Persona-Identified Chatbot through Small-Scale Modeling and Data Transformation,” Electronics, vol. 13, no. 8, MDPI, 2024. [10] S. Abdulla, M. Al-Ayyoub, and Y. Jararweh, “Pre-training on Multimodal Data for Improved Persona Detection,” Springer, 2024.