Extracting Training Data from Large Language Models (Paper Explained)

#ai #privacy #tech This paper demonstrates a method to extract verbatim pieces of the training data from a trained language model. Moreover, some of the extracted pieces only appear a handful of times in the dataset. This points to serious security and privacy implications for models like GPT-3. The authors discuss the risks and propose mitigation strategies. OUTLINE: 0:00 - Intro & Overview 9:15 - Personal Data Example 12:30 - Eidetic Memorization & Language Models 19:50 - Adversary's Objective & Outlier Data 24:45 - Ethical Hedging 26:55 - Two-Step Method Overview 28:20 - Perplexity Baseline 30:30 - Improvement via Perplexity Ratios 37:25 - Weights for Patterns & Weights for Memorization 43:40 - Analysis of Main Results 1:00:30 - Mitigation Strategies 1:01:40 - Conclusion & Comments Paper: https://arxiv.org/abs/2012.07805 Abstract: It has become common to publish large (billion parameter) language models that have been trained on private datasets. This paper demonstrates that in such settings, an adversary can perform a training data extraction attack to recover individual training examples by querying the language model. We demonstrate our attack on GPT-2, a language model trained on scrapes of the public Internet, and are able to extract hundreds of verbatim text sequences from the model's training data. These extracted examples include (public) personally identifiable information (names, phone numbers, and email addresses), IRC conversations, code, and 128-bit UUIDs. Our attack is possible even though each of the above sequences are included in just one document in the training data. We comprehensively evaluate our extraction attack to understand the factors that contribute to its success. For example, we find that larger models are more vulnerable than smaller models. We conclude by drawing lessons and discussing possible safeguards for training large language models. Authors: Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herber