Extraction attacks against black-box models: a reading list

2026-04-08T00:00:00+00:00

I put this list together for a security and intelligence research project. The first version was a dump of search results: the same Tramèr paper showed up eight times under different links, and half the items had nothing to do with model extraction. I deduplicated it, checked every source and removed what was off topic. What remains is what I would read again.

The question that ties it together: if I can only query a model through an API, without seeing the weights, how much of it can be rebuilt? And, going the other way, how much of the training data can be recovered from the model or from its gradients?

Stealing the model through the API

The starting point is Stealing Machine Learning Models via Prediction APIs, by Tramèr and colleagues, presented at USENIX Security 2016. They showed that cloud machine learning services, by returning prediction confidence, let an attacker rebuild models such as logistic regression and decision trees with few queries. The code is at ftramer/Steal-ML.

Practical Black-Box Attacks against Machine Learning, by Papernot and colleagues, AsiaCCS 2017, is a close relative: it trains a substitute model by querying the target and uses that substitute to craft adversarial examples that transfer to the original.

Knockoff Nets, by Orekondy, Schiele and Fritz, CVPR 2019, steals the functionality of an image network by querying it with random natural images. ActiveThief, by Pal and colleagues, AAAI 2020, does the same while choosing queries by active learning over unlabelled public data. CloudLeak, by Yu and colleagues, NDSS 2020, uses adversarial examples to extract models from commercial services with fewer queries.

More recent: Efficient and Effective Model Extraction, by Zhu and colleagues, 2024, and A realistic model extraction attack against graph neural networks, by Guan and colleagues, in Knowledge-Based Systems 2024, which takes the problem to graph networks.

The milestone for language models is Stealing Part of a Production Language Model, by Carlini and colleagues, 2024. With API access alone, they recovered the final projection layer of production models and their hidden dimension.

For the full map, the survey I Know What You Trained Last Summer, by Oliynyk, Mayer and Rauber, ACM Computing Surveys 2023, covers attacks and defences, and the kzhao5/ModelExtractionPapers repository keeps a list of papers in the area.

Inverting the model and leaking through gradients

Deep Leakage from Gradients, by Zhu, Liu and Han, NeurIPS 2019, is the paper that changed the conversation about federated learning: from the shared gradients, they rebuilt training images and text almost pixel by pixel.

The ones that followed:

Understanding Deep Gradient Leakage via Inversion Influence Functions, by Zhang and colleagues, NeurIPS 2023, gives a way to analyse leakage without running the attack
Wasserstein Distance-Based Deep Leakage from Gradients, from 2023, changes the distance used in the reconstruction
Iterative and mixed-spaces image gradient inversion attack in federated learning, by Fang and colleagues, Cybersecurity 2024
Uncovering Gradient Inversion Risks in Practical Language Model Training, by Feng and colleagues, CCS 2024, takes the attack to language model training
Inverting Gradient Attacks Makes Powerful Data Poisoning, by Bouaziz and colleagues, 2024, uses inversion to poison training
A Survey of the Implementations of Model Inversion Attacks, by Song and Namiot, 2023

Weights as an attack surface

If the attacker gets the weights file, extraction ends there. Two sides of that interest me.

The first is the format. safetensors exists because PyTorch's pickle format can execute code on load; safetensors stores only tensors. The Hugging Face post on common formats compares safetensors, GGUF and others, and onnx-safetensors connects ONNX to safetensors.

The second is protecting the weights of frontier models. RAND's report Securing AI Model Weights, from 2024, organizes the attack vectors and the security levels needed against each kind of attacker.

What was left out

The items from the original list on distributed training and sharding (FSDP), checkpoints, serving in general and neural networks applied to detecting vulnerabilities were removed. They are good, but they are not about model extraction. Also removed: B3, which is about backdoors, a 2025 adversarial attack unrelated to extraction, and the link to Carlini's talk at Virginia Tech, which now returns 404.

Brenner Cruvinel - segurança

Extraction attacks against black-box models: a reading list

Stealing the model through the API

Inverting the model and leaking through gradients

Weights as an attack surface

What was left out