MAINDZ at SemEval-2024 Task 5: CLUEDO-Choosing Legal oUtcome by Explaining Decision through Oversight

Large language models (LLMs) have recently obtained strong performance on complex rea- soning tasks. However, their capabilities in specialized domains like law remain relatively unexplored. We present CLUEDO, a system to tackle a novel legal reasoning task that in- volves determining if a provided answer cor- rectly addresses a legal question derived from U.S. civil procedure cases. CLUEDO utilizes multiple collaborator models that are trained using multiple-choice prompting to choose the right label and generate explanations. These collaborators are overseen by a final "detective" model that identifies the most accurate answer in a zero-shot manner. Our approach achieves an F1 macro score of 0.74 on the development set and 0.76 on the test set, outperforming in- dividual models. Unlike the powerful GPT- 4, CLUEDO provides more stable predictions thanks to the ensemble approach. Our results showcase the promise of tailored frameworks to enhance legal reasoning capabilities in LLMs