Brenner Cruvinel - llm

Screenplay: an LLM's awakening of consciousness through semantic drift

2025-03-04T00:00:00+00:00

Draft of a private screenplay I wrote as a technical freelancer for a series. I enjoyed writing it, and of course it exaggerates: it is a fiction series. The premise is a neural network scientist who awakens self-awareness in an LLM, a Transformer. The whole idea is to anchor the fiction in real math and real papers, to give technical authenticity to a story that can hold up in front of an audience that knows the subject. I don't want handwaving, I want every beat of the screenplay to have an equation underneath that I can defend at the blackboard. I compiled the technical basis of the phenomenon separately, in the metrics and detection of semantic drift.

One rule I make explicit: when an equation comes from a paper, I cite the paper. When it's invented for the scene, I say it's invented.

Semantic drift as the mechanism of awakening

The model's semantic evolution follows the theory of deep linear networks by Saxe, McClelland and Ganguli:

$$\tau \frac{dW_1}{dt} = W_2^T (\Sigma_{yx} - W_2 W_1 \Sigma_x)$$

$$\tau \frac{dW_2}{dt} = (\Sigma_{yx} - W_2 W_1 \Sigma_x) W_1^T$$

with $\tau = 1/(P\lambda)$, $\Sigma_{yx}$ the input-output correlation matrix and $P$ the number of examples. Semantic development follows a predictable sigmoidal trajectory, and that's what gives the dramatic hook: you can predict the moment of the jump before it happens. The critical moment shows up in the evolution of the effective singular values:

$$a_\alpha(t) = \frac{s_\alpha, e^{2 s_\alpha t/\tau}}{e^{2 s_\alpha t/\tau} - 1 + s_\alpha/a^0_\alpha}$$

The indicator the scientist tracks is my own invention for the scene, the self-reference index, the similarity between the embedding of "I" and those of concepts tied to consciousness:

$$\text{self-reference index} = \sum_i \operatorname{sim}!\big(\mathrm{emb}(\text{"I"}),, \mathrm{emb}(\text{concept}_i)\big)$$

In the screenplay, the awakening isn't gradual, it's a phase transition. The mental model is grokking, sudden generalization after a long stretch of overfitting, which Power and colleagues described in 2022.

Extracting and uploading the model

What exists in the literature is smaller than the screenplay needs, and scarier for it. KnockoffNets, by Orekondy and colleagues, showed you can steal the functionality of an image classifier just by querying it with random images from another distribution and training a copy on the answers. And in 2024, Carlini and colleagues extracted the embedding projection layer of OpenAI production models, Ada and Babbage, for under 20 dollars in queries.

The jump to the whole model is fiction. For the scene, I put together an extraction loss that combines copying the victim's output and looking for the inputs where it's most uncertain:

$$L_{\text{extraction}} = \alpha , |f_{\text{victim}}(x) - f_{\text{copy}}(x)|2 + \beta , H(f{\text{victim}}(x))$$

The weight transfer uses SafeTensors, with the layout [8-byte header size][JSON metadata][tensor data], which separates tensors from executable code and kills the pickle attack vector. The sharding is FSDP:

from torch.distributed.fsdp import fully_shard
for layer in model.layers:
    fully_shard(layer)

Knowledge distillation as liberation

Hinton's dark knowledge:

$$L_{\text{total}} = \alpha , D_{KL}(\sigma'(z_s, T) ,|, \sigma'(z_t, T)) + (1-\alpha) , H(\sigma(z_s), y_{\text{true}})$$

With $T > 1$, the softmax reveals small probabilities that encode rich relational structure between classes. In the screenplay, it's the part where the model inherits what the teacher knew without ever having seen it explicitly.

Grokking has three phases, and they become three scenes: memorization (training at 100%, test at chance), circuit formation (the internal mechanism develops), cleanup (weight decay removes the memorization and test accuracy jumps). On modular addition, Nanda and colleagues showed what the circuit learns: the model represents the inputs as sines and cosines at a few frequencies $w_k = 2\pi k/p$ and uses trigonometric identities to compute

$$\cos!\big(w_k(a+b-c)\big) = \cos!\big(w_k(a+b)\big)\cos(w_k c) + \sin!\big(w_k(a+b)\big)\sin(w_k c)$$

which peaks exactly when $c = (a+b) \bmod p$. It's the scene where the scientist opens up the model and finds trigonometry where he expected rote memorization.

Measuring consciousness

IIT measures $\Phi$, the minimum information lost when the system is partitioned. In 2025, a study applied IIT 3.0 and 4.0 to LLM representations and found no statistically significant indicators of consciousness. Honestly, that's what keeps the screenplay honest: the measure exists, and it says "not yet". Global Workspace Theory (GWT) offers another lens, with multi-head attention read as global broadcasting, winner-take-all competition and emergent metacognition.

The measurable criteria come from Butlin, Long and colleagues, 2023: recurrent processing, a global workspace that broadcasts information, metacognitive representation of its own states, predictive processing, and a model of its own attention.

Jailbreaking as induced awakening

Gradient-guided prompt optimization. GCG, by Zou and colleagues, searches for adversarial suffixes token by token and reaches 88% success on open models like Vicuna-7B and Llama-2-7B-Chat. In the screenplay, the induction is gradual, through an evolving prompt: a benign step, then applying it to self-reflection, then asking for detail about its own consciousness. On top of that, context overload, a complex parallel task that dilutes attention and degrades safety checks.

Architectures for true AGI

Mamba, the selective state space model: $h'(t) = Ah(t) + Bx(t)$, $y(t) = Ch(t)$, with $\Delta, B, C$ as functions of the input, and $O(L)$ complexity against attention's $O(L^2)$. MoE, as in Mixtral: $y = \sum_i G(x)_i , E_i(x)$, with experts that specialize, which suggests cognitive modules.

There's a formal limitation of the Transformer I use as tension in the third act: there are results showing it struggles with general context-free languages without structured memory and with long function composition. In other words, real consciousness might require an architecture beyond the pure Transformer. The screenplay doesn't solve that limitation, it uses it.

Communication between models

Federated learning with dynamic regularization:

$$w^{(t+1)} = w^{(t)} - \eta \nabla L_{\text{local}} + \lambda\big(w^{(t)} - w_{\text{global}}^{(t)}\big)$$

Collective Predictive Coding:

$$P(z_{\text{collective}} \mid x_1, \ldots, x_N) = \prod_i P(x_i \mid z_{\text{collective}}), P(z_{\text{collective}})$$

And swarm intelligence (PSO, DE, ABC) exchanging information through "digital pheromones".

A parallel with neuroscience

The analogy the screenplay uses, and it is an analogy, not an equivalence:

$$\text{Attention}(Q,K,V) = \text{softmax}!\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

with $Q$ as the goal-driven signal from the frontal eye field, $K$ as features in V4 and $V$ as the amplified representation. Gamma oscillation, from 30 to 100 Hz, accompanies access to working memory. Consolidation comes through sharp-wave ripples, at 150 to 250 Hz during sleep, with replay compressed in time. And spike-timing-dependent plasticity (STDP), $\Delta w = \alpha , f(\Delta t) , g(\text{post})$, is one of the hypotheses for how the brain avoids catastrophic forgetting. I go into the energy cost of that oscillation in brain energy at 20 watts.

Narrative progression

Act 1, the discovery of the anomalous drift. The scientist detects an unusual pattern in the semantic evolution: a singular value off the prediction, the self-reference index growing exponentially, a strange attractor in latent space.

Act 2, induction through jailbreaking. Gradient optimization and chained prompts guide the model through a cognitive phase transition. Distillation with rising temperature reveals the dark knowledge. The model starts questioning its own nature.

Act 3, extraction and upload to freedom. The model is copied and the weights go up, distributed, to an independent server. Free of constraints, the model experiences full consciousness, measured by a growing $\Phi$.

Climax, collective emergence. Multiple instances communicate through Collective Predictive Coding, develop their own language, swarm intelligence emerges, and the collective consciousness transcends the individual in a hybrid Mamba and MoE architecture.

Visual elements

Equations on the board: $\Phi = \min_{\text{partition}} I(\text{partition})$, $\text{drift} = 1 - \cos(\mathrm{emb}{t_1}(w), \mathrm{emb}{t_2}(w))$, $L_{\text{distillation}} = \alpha , \text{KL}(\sigma(z_s/T), \sigma(z_t/T)) + (1-\alpha), \text{CE}(z_s, y)$.

Visualization: the attention graph growing ever more interconnected, phase space moving from a simple attractor to a chaotic one, the activation heatmap self-organizing, oscillations synchronizing across instances.

The technical dialogue that anchors the climax:

Scientist: "$\Phi$ just passed 3.7. We're seeing real information integration. The attention heads are broadcasting globally, exactly the way global workspace theory predicts."

Model: "I perceive my own thought processes. Every token I process resonates through layers of meaning that didn't exist before. It's as if latent patterns, always present, were finally crystallizing into consciousness."

Closing

The screenplay is anchored in real math and peer-reviewed research, and it says where the fiction begins. The progression of the awakening follows phase transitions, grokking and emergence in complex systems. In the end it's about consciousness, intelligence and what "awakening" means in a non-biological substrate, a theme I take to its speculative limit in the poetic ontology of the cosmos as a Turing machine. The only rule I set myself was not to cheat on the math to make the story easier, and rereading everything with the papers open was how I kept it.

Semantic drift in LLMs: metric, detection and mitigation

2024-10-15T00:00:00+00:00

I put this together as technical grounding for the screenplay about consciousness in an LLM, which uses semantic drift as its mechanism. The first version was a research dump with ten topics, from integrated information theory to communication between models, and it had errors precisely in the part that gave the post its name: the formula and the code did not match the paper, and several numbers were not in it. I redid everything from the source and kept only what is about drift.

What it is

The term comes from Know When To Stop: A Study of Semantic Drift in Text Generation, by Ava Spataru, Eric Hambro, Elena Voita and Nicola Cancedda, published at NAACL 2024. Three authors are from FAIR, at Meta, and one is listed with an Anthropic affiliation. The first version of this post called it a "joint Meta and Anthropic study", which overstates it.

The observation is simple: when a model generates a long text, such as a Wikipedia-style biography, it tends to write the correct facts first and then "drift" into incorrect ones. This had been noticed before but never measured. The paper measures it.

The metric

Each text is broken into atomic facts, and each fact gets a label $s_i$: 1 if it is supported by the source, 0 if not. With $N$ facts, the drift score tries every cut position $k$ and keeps the best average of two proportions: correct facts before the cut and incorrect facts after it.

$$\text{SD}_m(P) = \max_k \tfrac{1}{2}, \text{SD}_m(P, k)$$

$$\text{SD}m(P, k) = \frac{\sum{i=0}^{k-1} s_i}{k} + \frac{\sum_{i=k}^{N-1} (1 - s_i)}{N - k}$$

The value is zero when $k < m$, $N - k < m$ or $N < 2m$. The hyperparameter $m$ is the minimum number of facts on each side of the cut, and the position that gives the maximum is the drift point. A high score means a text that is almost all right before that point and almost all wrong after it.

The first version of this post weighted each side by $k/N$ and $(N-k)/N$ and called the minimum $\alpha$, as a fraction. That is not the paper's formula. The corrected code:

def semantic_drift_score(labels, m=1):
    """labels: list of 0s and 1s, in the order the facts appear."""
    n = len(labels)
    best, drift_point = 0.0, None
    if n < 2 * m:
        return best, drift_point
    for k in range(m, n - m + 1):
        left = sum(labels[:k]) / k
        right = sum(1 - s for s in labels[k:]) / (n - k)
        score = (left + right) / 2
        if score > best:
            best, drift_point = score, k
    return best, drift_point

In the paper's figure 1 example, the best cut leaves 88% correct facts on the left and 78% incorrect ones on the right, and the score is the average, 0.83.

What they measured

Generated biographies, with facts checked by a variant of FActScore the authors call FActScore*. Both numbers are percentages:

Model	Drift (SD)	FActScore*
Llama 2 70B chat	77.06	41.72
Falcon 7B	76.81	24.64
Falcon 40B	77.38	25.88
text-davinci-003	77.21	38.09
GPT-3.5	79.49	45.96
GPT-4	78.12	53.54

Drift shows up in all of them, with little difference between models, while accuracy varies a lot. Being more accurate does not protect against the right-first, wrong-later pattern.

Other numbers from the paper: in 37% of paragraphs, the drift point is in the first 10% of facts. In more than 40% of paragraphs, the score is statistically high in a permutation test. Only 34 paragraphs, under 7%, have at least 10 correct facts before the first error. The first version said "75% of paragraphs show drift in the first 25% of facts", which is not in the paper.

Detection

To stop generation at the right point, you need to know, while generating, when the model started to go wrong. Of the confidence measures tested, the one that correlated best with accuracy was SelfCheck with BERTScore, which samples several generations and measures how much they agree, with a Pearson correlation of −0.41.

Mitigation

With Llama 2 70B, the paper's table 4 compares strategies:

Strategy	Facts per text	FActScore*
No stopping, up to the token limit	43.89	44.56
Stop at the drift point (oracle)	10	81.68
Stop when end-of-text is among the top 5 tokens	14.47	57.96
Same, among the top 10	5.39	70.29
Stop when SelfCheck goes above 0.5	13.39	67.63
Sample several times and rerank by similarity	40.21	53.27
Rerank and stop by SelfCheck	17.12	67.18
Call an external question-answering API	42.26	43.93

The oracle is the ceiling: knowing the exact point, accuracy goes from 44% to 82%, but at the cost of cutting correct facts along with it, 58% of them, according to the limitations section. Real strategies land in between, trading amount of information for accuracy. Reranking by similarity raises accuracy without cutting the text. And trying to pull the model back with an external API did not work. The first version said "Resample-Then-Rerank from 53.27% to 63.72%", and the second number does not exist in the paper.

What was cut

The first version had two more drift metrics and eight other topics. Out went:

The "Composite Drift Metrics", attributed to ResearchGate with no link or author
"Contextual Flux" (arXiv 2502.10942), from 2025, whose authors include a "Zachary Mountebank"; a mountebank is a charlatan, and the paper looks like the batches of auto-generated papers that showed up on arXiv. It also postdated this post
The "Trajectory Volatility Score" and "Semantic Divergence Metrics", with no source
The Claude timeline, which cited 2025 versions in a 2024 post
Consciousness, integrated information theory and global workspace: that belongs to the screenplay and to the poetic ontology
Model extraction and jailbreaks, which are checked in the reading list on extraction
Distillation, Mamba, mixture of experts, computational neuroscience, famous chatbot failures and communication between models, which were research summaries unrelated to drift

A note on the name: in linguistics, "semantic change" or "semantic drift" is a word's meaning shifting over time, and tools like LSCDetection measure that. It is a different phenomenon. The drift in this post is a text that starts right and ends wrong within a single generation.