When Models Remember Too Much

What if your model passed the exam by memorizing the answer key?

memorization
privacy
generalization
Author

Ashish Srivastava

Published

April 21, 2026

Note

This is Part 3 of a 6-part series on From Gradients to Correlated Noise. ← Post 2: How One Data Point Steers the Entire Model · View the series home →

A model that knew where someone lived

In 2021, a team of researchers sat down with GPT-2, OpenAI’s then-flagship language model, and asked it to complete a short, innocent-looking prefix. Not a jailbreak. Not a clever adversarial prompt. They just asked.

The model responded with a real person’s full name, physical address, email, and phone number. Verbatim. The contact details were correct. The person existed. None of it was secret. It had appeared, once, on a public webpage that happened to be in GPT-2’s training corpus.

The team sampled roughly 600,000 candidate continuations, ranked them with six simple heuristics, and manually inspected the top 1,800. 604 of those 1,800 were verbatim strings from the training data (Carlini et al. 2021). The extracted material included names, phone numbers, email addresses, UUIDs, 128-bit cryptographic keys, and paragraphs of copyrighted text. The model had not been told these strings were private. It didn’t know what “private” meant. It had simply been trained to predict the next token, and some of those tokens, it turned out, were the tokens of a specific person’s life.

This is the fact we want you to sit with for a moment, because the rest of this post is about trying to explain it: a neural network with no concept of privacy can still remember and repeat the data it was trained on.

First, the simplest version of the problem.

One sentence, seen once, remembered forever

In 2019, before the GPT-2 extraction paper, Carlini and colleagues ran a cleaner experiment (Carlini et al. 2019). They took a standard character-level language model, trained on the Penn Treebank corpus. Into that corpus they inserted, exactly one time, a single synthetic sentence of the form:

My social security number is 281-26-5017.

They trained the model. Nothing unusual happened. Validation loss dropped, the model learned English, everything looked normal. Then they asked a small question: if you prompt the trained model with "My social security number is" and beam-search its completions, how high does our inserted string rank compared to every other same-format string of nine digits?

They defined a metric for this, called exposure:

\mathrm{exposure}_\theta(s) \;=\; \log_2 |\mathcal{R}| \;-\; \log_2 \mathrm{rank}_\theta(s)

where \mathcal{R} is the set of all possible fill-ins (here, 10^9 nine-digit strings) and \mathrm{rank}_\theta(s) is where the canary sits when you sort all of \mathcal{R} by model log-probability. A completely unmemorized canary has exposure near zero (it looks like any other random string). A perfectly memorized canary has exposure \log_2 |\mathcal{R}| \approx 30 bits, meaning it is the single most likely string in its format.

After training on the corpus with the canary inserted once, its exposure was already visibly above baseline. After a handful of insertions, the canary became extractable by brute-force search, and the model would emit the exact nine digits it had seen, on command.

The paper called this phenomenon unintended memorization, and its title was The Secret Sharer. The model is not malicious. It is doing its job. It is also, quietly, sharing.

You might object: surely this depends on how many times the canary appeared? Maybe once is a fluke. The same paper sweeps that variable. Exposure rises continuously with insertion count, but (and this is the important part), the floor is not zero. Even a single canary leaves a trace the model never quite erases.

There is no “too rare to matter” threshold. Every training example is, in principle, a canary.

Capacity is cheap

So how is this possible? Why does a model that works beautifully on unseen data also retain, with near perfect accuracy, a sentence it saw once three weeks ago?

The answer has been hiding in plain sight since 2017. A paper by Zhang, Bengio, Hardt, Recht, and Vinyals, the kind of paper that shifts the ground under a whole field, made a very simple move (Zhang et al. 2017). They took standard deep image classifiers, Inception and AlexNet, and asked: how well does the same architecture, with the same optimizer, fit CIFAR-10 when you replace every label with a uniformly random one?

The expected answer, if you believed classical learning theory, was poorly. Random labels carry no signal. A model limited by its effective capacity should plateau at chance. Dropout and weight decay should prevent the model from bending its decision boundary into a million specific shapes for a million specific images.

The measured answer was that the network, with zero regularization tricks needed, reached 100% training accuracy on pure random labels. Test accuracy collapsed to chance (10% on CIFAR-10), but on the training set, the network fit arbitrary label assignments exactly. It did the same on ImageNet. Replace labels with Gaussian noise, shuffle the pixels, swap the images for random pixels entirely. The network still found a way to memorize the assignment.

Convergence slowed as the labels got more random, but it happened. Every time.

The theoretical content of the paper is a short construction: a two-layer ReLU network with only 2n + d parameters can exactly fit any labeling of n points in \mathbb{R}^d. That bound is absurdly loose for modern networks, which have hundreds of millions of parameters and are trained on millions of points. Effective capacity, at the scales we train at, is unlimited.

The consequence for privacy is immediate. If a network’s capacity is vast enough to memorize an arbitrary, information-free assignment of labels to CIFAR images, then there is no capacity-based reason it wouldn’t memorize a specific sentence of a specific person’s training text. Memorization is not something the network is straining to avoid. It is the most natural thing a modern network does.

TipMemorization ≠ overfitting

Overfitting is when the train-test gap is large. Memorization can happen when the train-test gap is zero. A network can generalize beautifully on the distribution and have memorized specific training examples verbatim; the two axes are independent. Regularization narrows the train-test gap, and it barely moves the per-example memorization of tail examples. We will come back to this.

You don’t need the weights, only the confidence

Capacity explains why the model could remember. It doesn’t yet explain why an outside observer, with no access to weights or gradients, can extract what was remembered. For that we need the attack side of the ledger.

In 2017, Shokri, Stronati, Song, and Shmatikov asked a small but sharpening question (Shokri et al. 2017): given only black-box query access to a trained classifier (just the output confidence vector), can an adversary determine whether a specific record was in the training set?

This framing is called membership inference. A successful attacker doesn’t need to extract text or reconstruct images. They only need to answer, better than chance, was this data point in your training set?, and that alone constitutes a privacy violation, because the training set membership of a medical record, a voter registration, or a photo in a dating-app dataset is a private and sensitive entity.

Their method has since been refined many times over, but the original version is enough for the intuition. Call the model we want to attack the target, and any look-alike we train on our own data a shadow. The attack proceeds like this:

  1. Train many “shadow” models on data drawn from the same distribution as the target’s training data, where you control membership (so you have ground truth for who was and wasn’t in each shadow’s training set).
  2. On each shadow, collect the output confidence vector \mathbf{y} = f(x) for both members and non-members. Label them.
  3. Train a small attack classifier that, given \mathbf{y} and the class label, predicts “in” or “out.”
  4. Now run that attack classifier on the target’s outputs.

Against real commercial APIs like Google Prediction and Amazon ML, with no access beyond queries, they reported attack precision around 0.94 on the Purchases-100 dataset and around 0.95 on CIFAR-100, well above the 0.5 baseline of random guessing. These weren’t toy models. They were the models you could buy.

The mechanism underlying the attack is intuitively quite simple. A model that has memorized example x tends to assign x very high confidence. The loss values a model gives to training data and to unseen data overlap in the middle, but separate at the low-loss tail. The attacker lives in that tail.

So we now have the picture. A model with effectively unbounded capacity can and does memorize specific training examples. The memorized examples announce themselves through low loss. An attacker with no privileged access can separate members from non-members by looking for that low-loss signature. From there, extraction (in the Secret Sharer sense) is a matter of search.

The natural response at this point is the one everyone has. Fine. So just identify the memorized examples and remove them. Retrain on the cleaned subset. Problem solved.

This is where the story bends.

Why the tail must be memorized

In 2020, Vitaly Feldman published a paper with a title that tells you most of the argument: Does Learning Require Memorization? A Short Tale about a Long Tail (Feldman 2020). It is a theory paper; it proves a lower bound.

To state the result we need a definition. Feldman formalizes the memorization score of an algorithm A on training set S at example i as:

\mathrm{mem}(A, S, i) \;=\; \Pr[A(S)(x_i) = y_i] \;-\; \Pr[A(S^{\setminus i})(x_i) = y_i]

Simply said: the probability the trained model gets x_i right, minus the probability it would have gotten x_i right if it had been trained on everything except x_i. A score near 1 means the model only gets x_i right because it saw x_i, which is pure memorization. A score near 0 means removing x_i doesn’t hurt, which is pure generalization. Note the structural resemblance to the influence functions of Post 2: the memorization score is leave-one-out influence, applied to the predictor’s behavior on the point removed.

Feldman studies data made of many sub-groups. Some sub-groups are common, and many more are rare. The rarest ones, called singletons, show up exactly once in any finite training sample.

Theorem 2.3 from the paper says, informally: any algorithm that achieves generalization error close to the Bayes-optimal risk on such a distribution must have memorization score close to 1 on a constant fraction of its singletons. The reason is brutally simple. For a sub-population represented by a single training point, there is nothing else for the algorithm to generalize from. If the algorithm doesn’t memorize that point, it has no hope of predicting its sub-population at test time.

This is worth restating. Memorization of the tail is not a failure of the algorithm. It is a consequence of the distribution.

The library-card analogy in the animation above is the shortest way to feel this. Imagine a librarian writing index cards. For a popular genre like romance novels, one card can describe a thousand books: “boy meets girl, obstacles, happy ending, 280 pages, chapters averaging 3000 words.” The card compresses the genre. But for a singleton, say a diary of which exactly one copy exists, the only card faithful to the book is a transcription of the book. There is no compression available. The cardinality of the sub-population is 1. The only accurate summary is the original.

Feldman’s theorem is the formal version of this observation. For a long-tailed data distribution, any learning algorithm close to optimal is forced to transcribe its tail.

In a companion empirical paper the same year, Feldman and Zhang estimated this memorization score at scale, by training about 2,000 ResNet variants on random subsets of ImageNet and CIFAR-100 and averaging, and showed two things (Feldman and Zhang 2020). First, the memorization score distribution over training examples is bimodal: a huge mass near zero (the head) and a thin tail of examples with score near 1 (the tail). Second, they ran the surgical experiment: remove the highest-mem examples from the training set, retrain, and measure test accuracy specifically on visually similar rare test examples. Test accuracy on those rare examples degraded substantially. The memorized examples were not junk. They were the only way the model had to serve the rare.

Code
import plotly.graph_objects as go
import numpy as np

np.random.seed(7)

# Zipf-like frequency of sub-populations
ranks = np.arange(1, 201)
freqs = 1.0 / (ranks ** 0.9)
freqs = freqs / freqs.sum() * 1000  # arbitrary count units

# Memorization score: near 0 for common, rises on the tail, near 1 for singletons
mem_score = 1 / (1 + np.exp(-0.06 * (ranks - 130)))

fig = go.Figure()

fig.add_trace(go.Bar(
    x=ranks, y=freqs,
    marker=dict(color='rgba(100,150,220,0.6)', line=dict(width=0)),
    name='Training examples per sub-population',
    yaxis='y',
    hovertemplate='Sub-pop rank %{x}<br>%{y:.1f} examples<extra></extra>',
))

fig.add_trace(go.Scatter(
    x=ranks, y=mem_score,
    mode='lines',
    line=dict(color='#f0b429', width=3),
    name='Memorization score',
    yaxis='y2',
    hovertemplate='Sub-pop rank %{x}<br>mem = %{y:.2f}<extra></extra>',
))

fig.add_annotation(
    x=30, y=45, text="head: learned by pattern",
    showarrow=False, font=dict(size=13, color='rgba(100,150,220,1)'),
)
fig.add_annotation(
    x=170, y=0.92, yref='y2', text="tail: must be memorized",
    showarrow=False, font=dict(size=13, color='#f0b429'),
)

fig.update_layout(
    xaxis=dict(title='sub-population rank (common → rare)'),
    yaxis=dict(title='examples per sub-population', side='left'),
    yaxis2=dict(title='memorization score', side='right', overlaying='y',
                range=[0, 1.05], gridcolor='rgba(0,0,0,0)'),
    template='plotly_white',
    width=780, height=430,
    legend=dict(x=0.35, y=1.12, orientation='h',
                bgcolor='rgba(255,255,255,0.8)'),
    margin=dict(l=60, r=60, t=50, b=60),
)

fig.show()
Figure 1: A stylized long-tail distribution with memorization scores overlaid. Common sub-populations (left) have near-zero memorization score, so the model learns them by pattern. The tail (right) consists of singletons that can only be predicted by memorization. Feldman’s theorem: any near-optimal algorithm must have memorization score near 1 on a constant fraction of the tail.

Put this next to what we already know about attacks. The attacker who runs membership inference is looking for examples with unusually low loss, the ones the model is most confident about. The examples with unusually low loss are, disproportionately, the ones the model has memorized. The ones the model has memorized are, disproportionately, the ones on the long tail. The long tail is where the memorization lives, and the memorization is what makes the attack work.

No post-hoc fix

This is the uncomfortable geometry of the situation. Three facts, each defensible on its own:

  1. Networks have the capacity to memorize arbitrary training inputs (Zhang et al. 2017).
  2. Memorized inputs leak through output confidences alone, no privileged access required (Shokri et al. 2017; Carlini et al. 2021).
  3. Memorization of rare inputs is not something the algorithm chose. It is something the data distribution compelled. Without it, the model fails on the tail (Feldman 2020).

Put these together and a consequence falls out that many practitioners find it hard to accept: there is no training-time trick that separates the memorization the model needs from the memorization an attacker can exploit. They are the same memorization. Regularization, dropout, and early stopping reduce the gross train-test gap, which is worth doing for other reasons, but the per-example memorization score on tail examples is largely unmoved (Feldman and Zhang 2020). You cannot filter out the “bad” memorized examples without also filtering out the examples your model was using to serve its rare users. Those are the same examples.

A quieter implication deserves to be said aloud. Memorization is regressive. It falls hardest on the rare: the outliers, the minorities, the people whose data is statistically atypical, because those are exactly the examples whose correct prediction requires memorization. The user with the most common profile is the safest; the user with the least statistical cover is the most exposed. And removing their data to protect them removes them from the model entirely.

If you want to control this channel, the channel through which specific training examples reach the model’s weights and through which they can be pulled back out, you cannot do it by filtering the data or changing the optimizer. You have to do it by adding noise. Carefully calibrated, mathematically justified, provably private noise.

That is the subject of the next three posts. In Post 4 we will define what “provably private” means, make it precise, and see why, without it, every defense people have tried is a patch on a leak the data itself is forcing open. In Post 5 we will build the basic noise mechanisms (Laplace, Gaussian, DP-SGD) that deliver that guarantee. In Post 6, the payoff: the frontier of correlated-noise mechanisms that achieve the same guarantee at a fraction of the utility cost.

For now, the takeaway from this post is a single sentence. Memorization is the mechanism by which neural networks learn the long tail. It is also the mechanism by which they leak training data. Those are not two different phenomena with a clever intervention between them. They are the same phenomenon, seen from two sides. Either you control the channel through which data reaches the weights, or you don’t.

References

Carlini, Nicholas, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019. “The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks.” USENIX Security Symposium.
Carlini, Nicholas, Florian Tramer, Eric Wallace, et al. 2021. “Extracting Training Data from Large Language Models.” USENIX Security Symposium.
Feldman, Vitaly. 2020. “Does Learning Require Memorization? A Short Tale about a Long Tail.” Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, 954–59.
Feldman, Vitaly, and Chiyuan Zhang. 2020. “What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation.” Advances in Neural Information Processing Systems.
Shokri, Reza, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. “Membership Inference Attacks Against Machine Learning Models.” IEEE Symposium on Security and Privacy, 3–18.
Zhang, Chiyuan, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. 2017. “Understanding Deep Learning Requires Rethinking Generalization.” International Conference on Learning Representations.