Large Language Models-aided Literature Reviews: A Study on Few-Shot Relevance Classification

Conducting a comprehensive literature review is a critical step in the research process, often requiring significant time and effort to identify and evaluate relevant academic papers. Traditional methods rely heavily on keyword-based searches to filter potentially relevant articles, which are then manually reviewed for inclusion. This process is not only labor-intensive but also susceptible to subjective biases, potentially affecting the consistency and accuracy of the review. This paper explores the potential of using large language models (LLMs) to automate the relevance assessment phase of literature reviews, thereby addressing some of the limitations of traditional methods. Specifically, we investigate the application of few-shot learning with various LLMs to determine the relevance of papers identified through keyword searches. We evaluate the sensitivity of this approach to the number of shots provided and compare the performance across different open-source LLMs, including Llama-3, Mistral, and Phi-3. Our findings aim to provide insights into the effectiveness of using LLMs for literature review processes, potentially transforming how researchers conduct literature reviews.