Brenner Cruvinel - ai safety

Extraction attacks against black-box models: a reading list

2026-04-08T00:00:00+00:00

I put this list together for a security and intelligence research project. The first version was a dump of search results: the same Tramèr paper showed up eight times under different links, and half the items had nothing to do with model extraction. I deduplicated it, checked every source and removed what was off topic. What remains is what I would read again.

The question that ties it together: if I can only query a model through an API, without seeing the weights, how much of it can be rebuilt? And, going the other way, how much of the training data can be recovered from the model or from its gradients?

Stealing the model through the API

The starting point is Stealing Machine Learning Models via Prediction APIs, by Tramèr and colleagues, presented at USENIX Security 2016. They showed that cloud machine learning services, by returning prediction confidence, let an attacker rebuild models such as logistic regression and decision trees with few queries. The code is at ftramer/Steal-ML.

Practical Black-Box Attacks against Machine Learning, by Papernot and colleagues, AsiaCCS 2017, is a close relative: it trains a substitute model by querying the target and uses that substitute to craft adversarial examples that transfer to the original.

Knockoff Nets, by Orekondy, Schiele and Fritz, CVPR 2019, steals the functionality of an image network by querying it with random natural images. ActiveThief, by Pal and colleagues, AAAI 2020, does the same while choosing queries by active learning over unlabelled public data. CloudLeak, by Yu and colleagues, NDSS 2020, uses adversarial examples to extract models from commercial services with fewer queries.

More recent: Efficient and Effective Model Extraction, by Zhu and colleagues, 2024, and A realistic model extraction attack against graph neural networks, by Guan and colleagues, in Knowledge-Based Systems 2024, which takes the problem to graph networks.

The milestone for language models is Stealing Part of a Production Language Model, by Carlini and colleagues, 2024. With API access alone, they recovered the final projection layer of production models and their hidden dimension.

For the full map, the survey I Know What You Trained Last Summer, by Oliynyk, Mayer and Rauber, ACM Computing Surveys 2023, covers attacks and defences, and the kzhao5/ModelExtractionPapers repository keeps a list of papers in the area.

Inverting the model and leaking through gradients

Deep Leakage from Gradients, by Zhu, Liu and Han, NeurIPS 2019, is the paper that changed the conversation about federated learning: from the shared gradients, they rebuilt training images and text almost pixel by pixel.

The ones that followed:

Understanding Deep Gradient Leakage via Inversion Influence Functions, by Zhang and colleagues, NeurIPS 2023, gives a way to analyse leakage without running the attack
Wasserstein Distance-Based Deep Leakage from Gradients, from 2023, changes the distance used in the reconstruction
Iterative and mixed-spaces image gradient inversion attack in federated learning, by Fang and colleagues, Cybersecurity 2024
Uncovering Gradient Inversion Risks in Practical Language Model Training, by Feng and colleagues, CCS 2024, takes the attack to language model training
Inverting Gradient Attacks Makes Powerful Data Poisoning, by Bouaziz and colleagues, 2024, uses inversion to poison training
A Survey of the Implementations of Model Inversion Attacks, by Song and Namiot, 2023

Weights as an attack surface

If the attacker gets the weights file, extraction ends there. Two sides of that interest me.

The first is the format. safetensors exists because PyTorch's pickle format can execute code on load; safetensors stores only tensors. The Hugging Face post on common formats compares safetensors, GGUF and others, and onnx-safetensors connects ONNX to safetensors.

The second is protecting the weights of frontier models. RAND's report Securing AI Model Weights, from 2024, organizes the attack vectors and the security levels needed against each kind of attacker.

What was left out

The items from the original list on distributed training and sharding (FSDP), checkpoints, serving in general and neural networks applied to detecting vulnerabilities were removed. They are good, but they are not about model extraction. Also removed: B3, which is about backdoors, a 2025 adversarial attack unrelated to extraction, and the link to Carlini's talk at Virginia Tech, which now returns 404.

Semantic drift in LLMs: metric, detection and mitigation

2024-10-15T00:00:00+00:00

I put this together as technical grounding for the screenplay about consciousness in an LLM, which uses semantic drift as its mechanism. The first version was a research dump with ten topics, from integrated information theory to communication between models, and it had errors precisely in the part that gave the post its name: the formula and the code did not match the paper, and several numbers were not in it. I redid everything from the source and kept only what is about drift.

What it is

The term comes from Know When To Stop: A Study of Semantic Drift in Text Generation, by Ava Spataru, Eric Hambro, Elena Voita and Nicola Cancedda, published at NAACL 2024. Three authors are from FAIR, at Meta, and one is listed with an Anthropic affiliation. The first version of this post called it a "joint Meta and Anthropic study", which overstates it.

The observation is simple: when a model generates a long text, such as a Wikipedia-style biography, it tends to write the correct facts first and then "drift" into incorrect ones. This had been noticed before but never measured. The paper measures it.

The metric

Each text is broken into atomic facts, and each fact gets a label $s_i$: 1 if it is supported by the source, 0 if not. With $N$ facts, the drift score tries every cut position $k$ and keeps the best average of two proportions: correct facts before the cut and incorrect facts after it.

$$\text{SD}_m(P) = \max_k \tfrac{1}{2}, \text{SD}_m(P, k)$$

$$\text{SD}m(P, k) = \frac{\sum{i=0}^{k-1} s_i}{k} + \frac{\sum_{i=k}^{N-1} (1 - s_i)}{N - k}$$

The value is zero when $k < m$, $N - k < m$ or $N < 2m$. The hyperparameter $m$ is the minimum number of facts on each side of the cut, and the position that gives the maximum is the drift point. A high score means a text that is almost all right before that point and almost all wrong after it.

The first version of this post weighted each side by $k/N$ and $(N-k)/N$ and called the minimum $\alpha$, as a fraction. That is not the paper's formula. The corrected code:

def semantic_drift_score(labels, m=1):
    """labels: list of 0s and 1s, in the order the facts appear."""
    n = len(labels)
    best, drift_point = 0.0, None
    if n < 2 * m:
        return best, drift_point
    for k in range(m, n - m + 1):
        left = sum(labels[:k]) / k
        right = sum(1 - s for s in labels[k:]) / (n - k)
        score = (left + right) / 2
        if score > best:
            best, drift_point = score, k
    return best, drift_point

In the paper's figure 1 example, the best cut leaves 88% correct facts on the left and 78% incorrect ones on the right, and the score is the average, 0.83.

What they measured

Generated biographies, with facts checked by a variant of FActScore the authors call FActScore*. Both numbers are percentages:

Model	Drift (SD)	FActScore*
Llama 2 70B chat	77.06	41.72
Falcon 7B	76.81	24.64
Falcon 40B	77.38	25.88
text-davinci-003	77.21	38.09
GPT-3.5	79.49	45.96
GPT-4	78.12	53.54

Drift shows up in all of them, with little difference between models, while accuracy varies a lot. Being more accurate does not protect against the right-first, wrong-later pattern.

Other numbers from the paper: in 37% of paragraphs, the drift point is in the first 10% of facts. In more than 40% of paragraphs, the score is statistically high in a permutation test. Only 34 paragraphs, under 7%, have at least 10 correct facts before the first error. The first version said "75% of paragraphs show drift in the first 25% of facts", which is not in the paper.

Detection

To stop generation at the right point, you need to know, while generating, when the model started to go wrong. Of the confidence measures tested, the one that correlated best with accuracy was SelfCheck with BERTScore, which samples several generations and measures how much they agree, with a Pearson correlation of −0.41.

Mitigation

With Llama 2 70B, the paper's table 4 compares strategies:

Strategy	Facts per text	FActScore*
No stopping, up to the token limit	43.89	44.56
Stop at the drift point (oracle)	10	81.68
Stop when end-of-text is among the top 5 tokens	14.47	57.96
Same, among the top 10	5.39	70.29
Stop when SelfCheck goes above 0.5	13.39	67.63
Sample several times and rerank by similarity	40.21	53.27
Rerank and stop by SelfCheck	17.12	67.18
Call an external question-answering API	42.26	43.93

The oracle is the ceiling: knowing the exact point, accuracy goes from 44% to 82%, but at the cost of cutting correct facts along with it, 58% of them, according to the limitations section. Real strategies land in between, trading amount of information for accuracy. Reranking by similarity raises accuracy without cutting the text. And trying to pull the model back with an external API did not work. The first version said "Resample-Then-Rerank from 53.27% to 63.72%", and the second number does not exist in the paper.

What was cut

The first version had two more drift metrics and eight other topics. Out went:

The "Composite Drift Metrics", attributed to ResearchGate with no link or author
"Contextual Flux" (arXiv 2502.10942), from 2025, whose authors include a "Zachary Mountebank"; a mountebank is a charlatan, and the paper looks like the batches of auto-generated papers that showed up on arXiv. It also postdated this post
The "Trajectory Volatility Score" and "Semantic Divergence Metrics", with no source
The Claude timeline, which cited 2025 versions in a 2024 post
Consciousness, integrated information theory and global workspace: that belongs to the screenplay and to the poetic ontology
Model extraction and jailbreaks, which are checked in the reading list on extraction
Distillation, Mamba, mixture of experts, computational neuroscience, famous chatbot failures and communication between models, which were research summaries unrelated to drift

A note on the name: in linguistics, "semantic change" or "semantic drift" is a word's meaning shifting over time, and tools like LSCDetection measure that. It is a different phenomenon. The drift in this post is a text that starts right and ends wrong within a single generation.