Leaner Training, Lower Leakage: Revisiting Memorization in LLM Fine-Tuning with LoRA

Memorization in large language models LLMs makes them vulnerable to data extraction attacks. While pre-training memorization has been extensively studied, fewer works have explored its impact in fine-tuning, particularly for LoRA fine-tuning, a widely adopted parameter-efficient method. In this...

Counterfactual Influence As a Distributional Quantity

Machine learning models are known to memorize samples from their training data, raising concerns around privacy and generalization. Counterfactual self-influence is a popular metric to study memorization, quantifying how the model's prediction for a sample changes depending on the sample's...