In-Context Unlearning for Text Summarization using Large Language Models

Due to privacy, safety, or usability reasons, there are situations in which information about specific training samples should be removed from pre-trained Machine Learning models. Machine Unlearning (MU) techniques focus on updating the models by forgetting the content of a specific subset of training samples to forget while preserving the information provided by the samples to retain. Unlike classical Large Language Models (LLMs), which can be used in a zero-shot setting, MU solutions require ad hoc training. This can be impractical, especially in low-cost or resource-constrained scenarios. In this work, we explore the use of LLM in-context unlearning to forget specific concepts in text summarization. The aim is to produce concise yet informative summaries of long textual documents while disregarding concepts mentioned in the forget set. We explore the performance of three open-source LLMs on two different datasets under various few-shot learning settings, investigating the impact of LLM pre-training, prompt and context settings, and documents' characteristics. We identify the situations in which LLMs with in-context unlearning could produce summaries as accurate as classical MU approaches with limited computational budget.