Brenner Cruvinel - pesquisa

Extraction attacks against black-box models: a reading list

2026-04-08T00:00:00+00:00

I put this list together for a security and intelligence research project. The first version was a dump of search results: the same Tramèr paper showed up eight times under different links, and half the items had nothing to do with model extraction. I deduplicated it, checked every source and removed what was off topic. What remains is what I would read again.

The question that ties it together: if I can only query a model through an API, without seeing the weights, how much of it can be rebuilt? And, going the other way, how much of the training data can be recovered from the model or from its gradients?

Stealing the model through the API

The starting point is Stealing Machine Learning Models via Prediction APIs, by Tramèr and colleagues, presented at USENIX Security 2016. They showed that cloud machine learning services, by returning prediction confidence, let an attacker rebuild models such as logistic regression and decision trees with few queries. The code is at ftramer/Steal-ML.

Practical Black-Box Attacks against Machine Learning, by Papernot and colleagues, AsiaCCS 2017, is a close relative: it trains a substitute model by querying the target and uses that substitute to craft adversarial examples that transfer to the original.

Knockoff Nets, by Orekondy, Schiele and Fritz, CVPR 2019, steals the functionality of an image network by querying it with random natural images. ActiveThief, by Pal and colleagues, AAAI 2020, does the same while choosing queries by active learning over unlabelled public data. CloudLeak, by Yu and colleagues, NDSS 2020, uses adversarial examples to extract models from commercial services with fewer queries.

More recent: Efficient and Effective Model Extraction, by Zhu and colleagues, 2024, and A realistic model extraction attack against graph neural networks, by Guan and colleagues, in Knowledge-Based Systems 2024, which takes the problem to graph networks.

The milestone for language models is Stealing Part of a Production Language Model, by Carlini and colleagues, 2024. With API access alone, they recovered the final projection layer of production models and their hidden dimension.

For the full map, the survey I Know What You Trained Last Summer, by Oliynyk, Mayer and Rauber, ACM Computing Surveys 2023, covers attacks and defences, and the kzhao5/ModelExtractionPapers repository keeps a list of papers in the area.

Inverting the model and leaking through gradients

Deep Leakage from Gradients, by Zhu, Liu and Han, NeurIPS 2019, is the paper that changed the conversation about federated learning: from the shared gradients, they rebuilt training images and text almost pixel by pixel.

The ones that followed:

Understanding Deep Gradient Leakage via Inversion Influence Functions, by Zhang and colleagues, NeurIPS 2023, gives a way to analyse leakage without running the attack
Wasserstein Distance-Based Deep Leakage from Gradients, from 2023, changes the distance used in the reconstruction
Iterative and mixed-spaces image gradient inversion attack in federated learning, by Fang and colleagues, Cybersecurity 2024
Uncovering Gradient Inversion Risks in Practical Language Model Training, by Feng and colleagues, CCS 2024, takes the attack to language model training
Inverting Gradient Attacks Makes Powerful Data Poisoning, by Bouaziz and colleagues, 2024, uses inversion to poison training
A Survey of the Implementations of Model Inversion Attacks, by Song and Namiot, 2023

Weights as an attack surface

If the attacker gets the weights file, extraction ends there. Two sides of that interest me.

The first is the format. safetensors exists because PyTorch's pickle format can execute code on load; safetensors stores only tensors. The Hugging Face post on common formats compares safetensors, GGUF and others, and onnx-safetensors connects ONNX to safetensors.

The second is protecting the weights of frontier models. RAND's report Securing AI Model Weights, from 2024, organizes the attack vectors and the security levels needed against each kind of attacker.

What was left out

The items from the original list on distributed training and sharding (FSDP), checkpoints, serving in general and neural networks applied to detecting vulnerabilities were removed. They are good, but they are not about model extraction. Also removed: B3, which is about backdoors, a 2025 adversarial attack unrelated to extraction, and the link to Carlini's talk at Virginia Tech, which now returns 404.

Prompt templates for research with agents

2025-09-15T00:00:00+00:00

I've been testing a few prompt templates that push crawlers and LLMs into delivering research more accurately. Sharing what I have, still in testing.

WIP, template under test. Still tuning it, not a final version.

The structure is always the same: role, objective, style, discovery sources, mandatory metadata and a benchmark rule. The most important line is the one that says to write "no reliable public benchmarks" instead of making up a number.

And even with it, that's not enough. Variation A produced my survey of the Rust ecosystem. In 2026, when I checked every row of that table against the GitHub API, I found a repository that doesn't exist and a row pointing to the wrong project, one with a similar name. The prompt forbade making things up, and the agent made things up anyway. The template organizes the delivery, but the one who checks the result is still me.

Template for research with agents

The prompts stay in Portuguese, exactly as I use them, because they ask the agent for output in Brazilian Portuguese. Variation A, the survey of Rust applications:

papel: pesquisador sênior em ecossistema open source e linguagens de sistemas, foco pesado em rust.

objetivo: mapear e analisar pelo menos 369 aplicações open source em rust, explicando por que cada uma importa, com metadados completos, contexto de adoção em produção e benchmark comparativo em tabela sempre que houver.

estilo: pt-br, sem emoji, sem bullet no corpo principal (parágrafo e tabela), todo recurso citado como Nome + URL com link markdown clicável, cabeçalhos markdown.

fontes de descoberta: Awesome Rust, LibHunt Rust, OSS Insight, GitHub Stars Leaderboard, Github Ranking Top 100 Stars in Rust, listas "Rust in production" e "rust-in-production" (ImplFerris). priorizar alto número de stars/forks e atividade recente, uso em produção por empresa conhecida, diversidade de domínio (devtools, sistemas distribuídos, db, runtimes js/ts, ai infra, editores, cli, jogos, wasm, networking, segurança, observabilidade). evitar libs minúsculas ou experimentais salvo impacto desproporcional.

metadados obrigatórios por app: nome, url do repo, descrição de 1 a 3 frases, categoria, subdomínio, maturidade, empresas que usam em produção, licença, stars/forks, crescimento 6 a 12 meses, data do último commit, número de contribuidores, "por que é foda" (2 a 4 frases).

benchmark: pra cada projeto com benchmark público sério, colete tipo, contexto, setup (hardware, so, versão rust), comparação, números-chave com unidade, link da fonte. monte tabela por projeto: projeto, cenário, métrica, resultado, concorrente, resultado da concorrente, vantagem aproximada, link. sem benchmark crível, escreva "sem benchmarks públicos confiáveis", nunca invente número. divergência entre fontes, explicite cenário, não consolide.

organização: tabela geral com as 369+, mais tabelas por domínio (devtools, runtimes/plataformas, infra/observabilidade, terminais/shells/editores, web e apis, segurança/cripto/password managers, ai/ml infra, db e search, embedded/iot). análise final: distribuição por tipo, vantagem comparativa do rust, padrões técnicos (async/await/tokio, crates de observabilidade, arquiteturas), adoção corporativa vs go/c/c++/java/node, benchmark vs marketing, lacunas e oportunidades.

Variation B: Rust + WebAssembly books and resources

papel: pesquisador sênior em rust, wasm e desenvolvimento web de alta performance.

objetivo: análise completa e atualizada de livros e recursos estruturados de rust + wasm, ecossistema global e contexto brasileiro, em tabelas com links clicáveis (livros, editoras, lojas, repos, doc oficial).

estilo: idem variação A.

escopo, cinco grupos: (1) livros-base de rust core relevantes pra wasm, com título, autor, ano, nível, temas, recomendação pra wasm, link oficial/compra/comunidade/repo. (2) livros específicos de rust + wasm, 20 a 30 obras, com tipo de recurso, público-alvo, escopo (browser/wasi/iot/mobile/desktop). (3) livros de wasm geral relevantes pra rust, com coluna "relevância pra rust". (4) contexto brasil: traduções, autores brasileiros, livros específicos, disponibilidade real em e-book e loja br. (5) recursos oficiais e books online com profundidade de livro (the rust and webassembly book, doc de wasm-bindgen/web-sys/js-sys/wasm-pack/yew/leptos), com tipo, org, temas, link, repo, licença.

análise final: pré-requisitos naturais antes de rust+wasm, quais livros são mais completos/atualizados/adequados por perfil (js dev migrando, rustacean avançado, dev de sistemas), como livros de wasm geral ajudam em performance/segurança/binário/interop, situação no brasil (lacunas e dependência de inglês), temas sub-representados (wasi, servidor, edge, frameworks modernos leptos/dioxus/tauri, segurança e fuzzing de módulos, casos além do browser).

Variation C: people and visualization projects in Rust + Wasm

papel: pesquisador sênior em rust, wasm, visualização de dados e ecossistema open source.

objetivo: mapeamento de pelo menos 50 pessoas relevantes em rust (foco rust+wasm) e pelo menos 50 projetos/libraries de grafos, charts e viz em wasm (idealmente rust, não exclusivamente), mais todos os links de presença online (github, site pessoal, redes, artigos, posts em dev.to e hacker news).

estilo: idem, tabela como forma principal.

pessoas: nome completo, país/região, papel (autor, maintainer, contributor rustwasm, criador de framework, educador), 2 a 5 projetos principais, temas técnicos, github, site/blog, redes (twitter/x, mastodon, linkedin, youtube, twitch), páginas de autor (dev.to, medium, substack, conferências), discussões de destaque (hacker news, reddit r/rust, users.rust-lang.org). hubs de descoberta: awesome-rust, awesome-rust-and-webassembly, awesome-wasm, orgs rustwasm e bytecodealliance, github topics (topic:webassembly l:rust). validar que o link pertence à pessoa (evitar homônimo) e que o projeto é minimamente ativo.

projetos: nome, descrição (2 a 3 frases), linguagem, tipo de viz (grafos, charts, heatmaps, mapas, dashboards, 2d, 3d), casos de uso, status, métricas (stars, contribuidores, downloads crates.io, uso citado), repo, site/docs, demos, artigos/talks/discussões.

conexões: cruze pessoa↔projeto com nome, projeto, papel (criador, maintainer, core contributor, autor de módulo crítico), link de comprovação (readme de maintainers, palestra, artigo "introducing X").

contexto brasil (opcional): devs br relevantes com ênfase em viz/grafos, projetos br em rust com target wasm, conteúdo em português, talks em conferência br.

análise final: núcleos de contribuição (rustwasm, bytecodealliance, empresas), clusters de viz (grafos, dashboards, jogos, ides, ferramentas científicas), perfis predominantes, lacunas (pouca bibliografia de grafos complexos, falta de tooling de alto nível, pouca doc pra iniciante), global vs brasil.

Semantic drift in LLMs: metric, detection and mitigation

2024-10-15T00:00:00+00:00

I put this together as technical grounding for the screenplay about consciousness in an LLM, which uses semantic drift as its mechanism. The first version was a research dump with ten topics, from integrated information theory to communication between models, and it had errors precisely in the part that gave the post its name: the formula and the code did not match the paper, and several numbers were not in it. I redid everything from the source and kept only what is about drift.

What it is

The term comes from Know When To Stop: A Study of Semantic Drift in Text Generation, by Ava Spataru, Eric Hambro, Elena Voita and Nicola Cancedda, published at NAACL 2024. Three authors are from FAIR, at Meta, and one is listed with an Anthropic affiliation. The first version of this post called it a "joint Meta and Anthropic study", which overstates it.

The observation is simple: when a model generates a long text, such as a Wikipedia-style biography, it tends to write the correct facts first and then "drift" into incorrect ones. This had been noticed before but never measured. The paper measures it.

The metric

Each text is broken into atomic facts, and each fact gets a label $s_i$: 1 if it is supported by the source, 0 if not. With $N$ facts, the drift score tries every cut position $k$ and keeps the best average of two proportions: correct facts before the cut and incorrect facts after it.

$$\text{SD}_m(P) = \max_k \tfrac{1}{2}, \text{SD}_m(P, k)$$

$$\text{SD}m(P, k) = \frac{\sum{i=0}^{k-1} s_i}{k} + \frac{\sum_{i=k}^{N-1} (1 - s_i)}{N - k}$$

The value is zero when $k < m$, $N - k < m$ or $N < 2m$. The hyperparameter $m$ is the minimum number of facts on each side of the cut, and the position that gives the maximum is the drift point. A high score means a text that is almost all right before that point and almost all wrong after it.

The first version of this post weighted each side by $k/N$ and $(N-k)/N$ and called the minimum $\alpha$, as a fraction. That is not the paper's formula. The corrected code:

def semantic_drift_score(labels, m=1):
    """labels: list of 0s and 1s, in the order the facts appear."""
    n = len(labels)
    best, drift_point = 0.0, None
    if n < 2 * m:
        return best, drift_point
    for k in range(m, n - m + 1):
        left = sum(labels[:k]) / k
        right = sum(1 - s for s in labels[k:]) / (n - k)
        score = (left + right) / 2
        if score > best:
            best, drift_point = score, k
    return best, drift_point

In the paper's figure 1 example, the best cut leaves 88% correct facts on the left and 78% incorrect ones on the right, and the score is the average, 0.83.

What they measured

Generated biographies, with facts checked by a variant of FActScore the authors call FActScore*. Both numbers are percentages:

Model	Drift (SD)	FActScore*
Llama 2 70B chat	77.06	41.72
Falcon 7B	76.81	24.64
Falcon 40B	77.38	25.88
text-davinci-003	77.21	38.09
GPT-3.5	79.49	45.96
GPT-4	78.12	53.54

Drift shows up in all of them, with little difference between models, while accuracy varies a lot. Being more accurate does not protect against the right-first, wrong-later pattern.

Other numbers from the paper: in 37% of paragraphs, the drift point is in the first 10% of facts. In more than 40% of paragraphs, the score is statistically high in a permutation test. Only 34 paragraphs, under 7%, have at least 10 correct facts before the first error. The first version said "75% of paragraphs show drift in the first 25% of facts", which is not in the paper.

Detection

To stop generation at the right point, you need to know, while generating, when the model started to go wrong. Of the confidence measures tested, the one that correlated best with accuracy was SelfCheck with BERTScore, which samples several generations and measures how much they agree, with a Pearson correlation of −0.41.

Mitigation

With Llama 2 70B, the paper's table 4 compares strategies:

Strategy	Facts per text	FActScore*
No stopping, up to the token limit	43.89	44.56
Stop at the drift point (oracle)	10	81.68
Stop when end-of-text is among the top 5 tokens	14.47	57.96
Same, among the top 10	5.39	70.29
Stop when SelfCheck goes above 0.5	13.39	67.63
Sample several times and rerank by similarity	40.21	53.27
Rerank and stop by SelfCheck	17.12	67.18
Call an external question-answering API	42.26	43.93

The oracle is the ceiling: knowing the exact point, accuracy goes from 44% to 82%, but at the cost of cutting correct facts along with it, 58% of them, according to the limitations section. Real strategies land in between, trading amount of information for accuracy. Reranking by similarity raises accuracy without cutting the text. And trying to pull the model back with an external API did not work. The first version said "Resample-Then-Rerank from 53.27% to 63.72%", and the second number does not exist in the paper.

What was cut

The first version had two more drift metrics and eight other topics. Out went:

The "Composite Drift Metrics", attributed to ResearchGate with no link or author
"Contextual Flux" (arXiv 2502.10942), from 2025, whose authors include a "Zachary Mountebank"; a mountebank is a charlatan, and the paper looks like the batches of auto-generated papers that showed up on arXiv. It also postdated this post
The "Trajectory Volatility Score" and "Semantic Divergence Metrics", with no source
The Claude timeline, which cited 2025 versions in a 2024 post
Consciousness, integrated information theory and global workspace: that belongs to the screenplay and to the poetic ontology
Model extraction and jailbreaks, which are checked in the reading list on extraction
Distillation, Mamba, mixture of experts, computational neuroscience, famous chatbot failures and communication between models, which were research summaries unrelated to drift

A note on the name: in linguistics, "semantic change" or "semantic drift" is a word's meaning shifting over time, and tools like LSCDetection measure that. It is a different phenomenon. The drift in this post is a text that starts right and ends wrong within a single generation.