Chatbot Development Using LangChain: A Case Study to Foster Creativity and Critical Thinking

Critical thinking and creativity are fundamental skills for engineers and computer scientists. The emergence of Large Language Models (LLMs) able to create chatbots that use natural language is an opportunity for educators to foster these skills. The well-known risk of generative AI for potential misinformation offers fertile ground to practice critical thinking. This paper describes a hands-on experience within a database course, where students had to develop a chatbot using the LangChain framework, and to evaluate it from different points of view. The students were free to choose the domain of their chatbot. The learning goal was twofold: on the one hand, to make them practice with state-of-the-art technologies, and on the other hand to stimulate critical analysis on their output. The paper discusses the students’ evaluation of the chatbots under several metrics, including document retrieval, syntax and grammar accuracy, semantic relevance and information reliability. Students’ assessments were also compared to the teachers’ ones, to gain an insight on the critical attitude of the students and to offer a ground for discussion. The experience was stimulating and appreciated by the students. The final results highlight that the majority of students successfully produced chatbot responses that were grammatically and syntactically correct, and that consistently extracted pertinent sections from documents, yielding semantically relevant outputs. Despite these achievements, a significant portion of students expressed reservations about the reliability of the chatbot’s responses to prompts, gaining awareness of LLMs’ capability to generate responses that make sense to humans but may be potentially misleading.