Testing the global workspace at open frontier scale: the Jacobian lens on DeepSeek-V4-Flash and GLM-5.2

A replication and extension of the Jacobian lens on two frontier-scale open models, separating workspace properties that transfer from those that vary by family, with a vendor-described same-base test of whether a second post-training run is visible in J-space.

TL;DR — We ported Anthropic’s Jacobian lens to two frontier-scale open-weight models—DeepSeek-V4-Flash and GLM-5.2—and re-ran the subset of the global-workspace experiments that open weights allow.

Epistemic status: A replication-and-extension study. It rests on four open checkpoints—DeepSeek-V4-Flash-0731 and its April preview (both four-stream hyper-connections), GLM-5.2 (single-stream, 753B total / 40B active), and the DeepSeek-V4-Flash base model used in the base-vs-post probe—one lens recipe, and mostly small \(n\). Result tables carry exact counts and Wilson 95% intervals; most random controls are single-seed, the dose-response points five-seed; several estimator choices deviate from the paper (see Setup, and the Appendix). Effect directions are better supported here than effect sizes. Every headline number has a matched control, and several of those controls overturned our own first readings; where that happened we say so at the number.

Revised 13 August 2026: a preregistered follow-up ran the two controls the original Limitations called for and corrected four numbers. The corrections are footnoted where they occur. No conclusion changed.

Why these models

Interpretability results are often reported on models that outsiders cannot run or replicated on models that few people would deploy. The global-workspace paper falls into the first category: its experiments use Claude, so the claims concern frontier models but the evidence cannot be checked directly against the original checkpoints. Many open replications fall into the second category: a minimal port on Qwen3.5-4B, an independent replication on GPT-2 124M and Qwen3 1.7B–14B, an Apple-Silicon visualizer for Qwen3.6-27B, and Neuronpedia’s pre-fitted lenses for 36 models topping out at Llama 70B—small enough to iterate on, but also small enough that a negative result is hard to interpret. The GPT-2/Qwen3 replication is the cautionary example: rhyme planning and mental arithmetic failed to replicate there while multi-fact editing held, and at that scale “this property is absent” and “this property is not real” can look identical.

We wanted the combination that is usually missing: open weights at frontier scale. Two models qualified, and they are as different from each other as open models get:

Both are frontier-scale rather than research-scale—by which we mean only that both are current flagship-line releases with parameter counts in the hundreds of billions—and both have publicly available weights, so anyone with sufficient hardware can check the results below. Our code and fitted lens checkpoints are open-sourced at github.com/xiangchensong/jacobian-lens-open-frontier. To our knowledge, the Jacobian lens has not previously been run at this scale outside a frontier lab. Doing so required solving problems that do not arise on small models: a rectangular Jacobian for the four-stream residual and backward formulas for FP8 kernels that ship without autograd support.

A third checkpoint gave us a second axis for free: DeepSeek replaced its April DeepSeek-V4-Flash preview with 0731, describing the new release as the same architecture with only the post-training redone—a vendor-described same-base pair, two recipes (Part III).

The questions open checkpoints can answer

Verbalizable Representations Form a Global Workspace in Language Models defines the Jacobian lens as

\[\operatorname{lens}_{\ell}(\mathbf{h}) = \operatorname{unembed}\!\left(\mathbf{J}_{\ell}\mathbf{h}\right), \qquad \mathbf{J}_{\ell} = \mathbb{E}\!\left[ \frac{\partial \mathbf{h}_{\mathrm{target}}} {\partial \mathbf{h}_{\ell}} \right].\]

The lens is estimated in closed form as a running mean of per-prompt Jacobians; it requires neither training nor learned parameters. The paper argues that verbalizable content occupies a middle band of the network that behaves like a global workspace: it has limited capacity, is causally important for deliberate reasoning, is largely bypassed by automatic processing, and exhibits ignition-like commitment dynamics.

All of the paper’s experiments ran on Claude models, which leaves open two questions. Across families: which parts of this are properties of large trained transformers, and which belong to a particular family? And within a family: holding the base fixed, does changing the post-training recipe move the workspace—and if so, what part of it? Parts I–II take the first question; Part III takes the second.

We started with the structurally harder of the two models. DeepSeek-V4-Flash-0731 carries four parallel residual streams per position (hyper-connections, with hc_mult=4), so a square \(\mathbf{J}_{\ell} \in \mathbb{R}^{d \times d}\) is not defined for the relevant mapping. We generalized the lens to a rectangular \(\mathbf{J}_{\ell} \in \mathbb{R}^{4096 \times 16384}\): the source is the four flattened streams, the target is the stream-collapse output, and the square case remains bit-identical. We then re-ran everything accessible with open weights: structural measurements, causal swap/injection/ablation experiments, method comparisons, and both appendix robustness variants. Of 33 tracked experiments, 17 reproduced outright; the remainder split between direction-preserved shortfalls, qualitative divergences (reported below as findings, not buried), nulls, and Claude-only items we could not run. The per-experiment map, with the script behind each number, is in the repo.

That round ended with two hypotheses only a third family could test, so we ran one. GLM-5.2 (753B total / 40B active, 78 layers, hidden 6144, MLA + DSA sparse attention) has a conventional single-stream residual, so the square \(\mathbf{J}_{\ell} \in \mathbb{R}^{6144 \times 6144}\) applies and the lens library needed zero changes. The obstacle was infrastructural: its native FP8 kernels register no autograd formula, and unlike DeepSeek the dequantized model does not fit (756 GB in FP8 becomes approximately 1.5 TB in BF16, against 1.17 TB of VRAM). We registered backward formulas for the three FP8 matrix-multiplication operations—activation gradients only, with frozen weights and a straight-through approximation with respect to the kernels’ internal activation quantization. Fitting took 14.2 hours for \(n=100\)—8.5 min/prompt as run, against the pre-fit probe’s estimate of 9.0 at dim_batch=32.

Setup, metrics, and conventions

The definitions the results depend on, in one place. The implementation behind every metric is in the repo, mapped to its experiment number (experiments/README).

Units and terms. Checkpoint means a specific released weight set; family means a lab’s model lineage. The paper reports its Claude results across several Claude models; we treat that as one family reported in aggregate, and we run one checkpoint per open family—except in Part III, which compares two checkpoints within the V4 family. Lens ranks are 0-indexed throughout: rank 0 is the top token.

Statistical conventions. Result tables report exact counts, and brackets are Wilson 95% intervals. Random-ablation controls use one fixed-seed draw per configuration, matched to the treatment in rank and layer range; where a control is instead matched on removed-norm fraction it is labelled as such and carries five seeds. Agreement statistics aggregate ≈2,200 positions across 20 prompts; positions cluster within prompts, so their naive ±0.02 intervals are optimistic. The swap/broadcast operating point (\(\alpha=2\)) was selected by a grid on the same items—post hoc, but shared by every checkpoint compared at it. With 33 tracked experiments and no multiplicity correction, a few nominal differences are expected by chance; conclusions therefore rest on preregistered predictions, matched controls, and differences whose intervals do not overlap—not on the tally.

Corpora. Lenses are fitted on 100–1000 WikiText prompts. All evaluation and intervention prompt sets are synthetic, Anthropic-authored, and disjoint from the fitting corpus. Structural statistics (band, agreement) use WikiText prompts; we did not verify that those are disjoint from the 100 fitting prompts.

Metrics, in order of appearance.

How this relates to the paper’s estimator. Three categories, detailed in the Appendix: (1) architecture-forced adaptations—the rectangular Jacobian on V4, FP8 backward formulas on GLM; (2) a modified estimator—per-prompt normalization of \(\mathbf{J}\) before averaging, used for every headline number; and (3) our own implementations of the causal interventions, which the paper specifies but does not ship. Strictly, then, this post establishes properties of the normalized Jacobian lens. Where we compared against the unnormalized estimator (the six-set readout battery), normalization helped—most on the association set (0.040 → 0.069)—and did not change the method ordering. The causal experiments were later re-run with the plain-mean lens on both models; the estimator makes no difference to them (Limitations).

Part I — What transfers

The band itself

Three measurements with distinct failure modes locate the band on DeepSeek-V4-Flash: the paper’s four onset/offset statistics turn at L19 and L39 of 43 layers; CKA geometry breaks into three blocks at L19/L40; the ignition threshold is sharpest at L24–33.

Layer top-1 acc excess kurtosis excess autocorr effective dim
0 0.0000 4.78 −0.0002 5.77
15 0.0023 1.36 0.0401 2.00
19 0.0059 2.07 0.0237 2.25
34 0.0338 2.65 0.0870 5.52
39 0.1665 0.94 0.0722 7.58
40 0.2452 0.52 0.0580 1.29
42 0.5085 6.72 −0.0011 3.70

(L34’s 0.0870 is near-peak rather than peak: excess autocorrelation maxes at L38, 0.0879.)

Line plots of top-1 accuracy, excess kurtosis, excess autocorrelation, and effective dimension across all 43 DeepSeek-V4-Flash layers.
Figure 1 (cf. paper Fig. 28): Top-1 accuracy is flat until L19 and jumps steeply from L39; that jump is the motor regime, where the lens is simply reading the output. Effective dimensionality contracts to a minimum at L17 within the band's onset region, fans out to 7.58 by L39, then collapses at L40 to its global minimum of 1.29. Statistics computed on held-out WikiText positions; no uncertainty bands are drawn (single corpus draw).

L19–39 of 43 layers is 44–91% of depth, against the paper’s 38–92% on Sonnet 4.5. Despite four residual streams instead of one and a different lab’s data and recipe, the estimated band occupies nearly the same fraction of depth (and it is unchanged under the 0731 redo—Part III).

On GLM, we fitted only the window projected from DeepSeek-V4-Flash’s relative depth (L34–71 of 78), so this is a prospective check rather than a measurement of GLM’s band boundaries. The check passed: top-1 accuracy is 0.0012 at L34, below DeepSeek-V4-Flash’s onset value, and 0.194 at L71, matching its offset jump. That is, the DeepSeek-derived window contains the expected low-to-high transition on GLM—consistent with, but not sufficient to establish, the same relative band boundaries. GLM’s true onset could sit earlier and its offset later; fitting layers just outside the window to bracket both transitions is the outstanding measurement.

Ignition

We replaced a token’s input embedding with

\[\alpha\,\operatorname{emb}(A) + (1-\alpha)\,\operatorname{emb}(B)\]

and swept \(\alpha\):

Layers 10→90% transition width in \(\alpha\)
0–6 never crosses (smooth, proportional)
19 0.60
24–33 ≤ 0.10 (unresolved—0.10 is the sweep’s grid resolution)
39 0.70
Heatmap showing a sharp blue-red transition near alpha 0.5 in the middle workspace layers.
Figure 2 (cf. paper Fig. 29): Inside the band, the blue/red boundary is nearly vertical—sharper than the α grid resolves. Below the band, the transition is graded, and at the earliest layers the share runs backwards, drifting 0.65 → 0.25 across the sweep.

At L30, the reciprocal-rank share runs 0.00 → 0.05 → 0.62 → 0.94 as \(\alpha\) increases from 0.3 to 0.6. The model tracks the mixture smoothly before the workspace and commits sharply inside it. This reproduces the qualitative ignition-like transition the paper reports; characterizing how sharp the step truly is would need a finer sweep around the threshold.

Ordered unspoken intermediates: an arithmetic case study

For calc: (4+17)*2+7=, DeepSeek-V4-Flash’s three unspoken intermediates first become the lens’s top token at L27 (21) → L30 (42) → L36 (49)—the order the computation requires, all inside the band. GLM reproduces the ordering on a correctly-solved instance: 21 at L50, 49 at L67.

One expression per model, so this is a case study and not a result. We built no arithmetic battery with order-recovery statistics, and what aggregate evidence we do have says arithmetic is the hardest content for this readout (Part II §5).

Log-scale layer-by-layer ranks of the arithmetic intermediates 21, 42, and 49.
Figure 3 (cf. paper Fig. 17): Lens rank of each intermediate at the final position of a single prompt, on a log scale. On DeepSeek-V4-Flash, once 42 takes over, 21 falls to rank ∼10⁵—worse than the ∼6.5 × 10⁴ rank a random token would get in a 129k vocabulary. We flagged this "clearing" as a possible cross-family discovery; see Part II for why it isn't one.

The workspace is load-bearing

The paper’s sharpest falsifiable prediction is that ablating the top-\(k\) J-lens directions across the band should destroy multi-step reasoning, and a matched-rank random ablation should not.

Measure Unablated J-space ablated Random control (same rank)
Multi-hop accuracy (90 items) 0.589 (53/90) 0.100 (9/90) 0.589 (53/90)
Pretraining top-1 agreement (≈2.2k positions) — 0.508 0.922

Flexible reasoning falls by 83% relative to the unablated baseline (Wilson 95% intervals: [0.05, 0.18] ablated against [0.49, 0.69] unablated), while the matched-rank random ablation reproduces the identical per-item correctness vector: the same 53 items right, none flipped in either direction.1,2 The control is what makes this evidence.

That identical vector is also the obvious objection to it. Perhaps the random ablation is simply too weak to do anything. It is not inert—agreement under it is 0.922 rather than 1.000, about one changed argmax in thirteen, and on GLM it flips two multi-hop items (55/90 unablated against 53/90 under the control)—but rank-matching is not magnitude-matching, and the first version of this post left the question there.

So we matched magnitude directly: random directions scaled to remove exactly the norm fraction the J-space subspace removes, five seeds per point, under a protocol preregistered before the runs. At identical dose the two interventions are nowhere near each other.

model arm removed-norm fraction KL(abl ‖ unabl) agreement multi-hop
DeepSeek-V4-Flash J-space, \(k=10\) 0.0351 2.548 0.523 26/90
DeepSeek-V4-Flash random, norm-matched 0.0351 0.105 0.896 74/90
GLM-5.2 J-space, \(k=10\) 0.0990 3.404 0.498 29/90
GLM-5.2 random, norm-matched 0.0990 1.055 0.736 71/90

The multi-hop gap is 16.1× the five-seed spread on DeepSeek-V4-Flash and 8.3× on GLM-5.2. Random directions leave multi-hop at baseline even at twice the J norm, and rank-matched random does nothing at four times it (\(k=320\), removed-norm 0.140, 71/90); a single J direction (\(k=1\), removed-norm 0.024) collapses it to 29/90. The effect is not a large perturbation but a small, extremely well-aimed one.

Four panels: multi-hop accuracy against removed-norm fraction and against KL, for DeepSeek-V4-Flash and GLM-5.2, with four intervention arms each.
Figure 4: Dose-response on both models: multi-hop accuracy against how much of the residual norm the intervention removes (left) and against the KL it induces on next-token distributions (right). The dotted vertical line is the $$k=10$$ operating point used everywhere else in this post. Every random arm—rank-matched, norm-matched, and drawn inside the row space of $$J$$—tracks the unablated line until it removes several times the J dose, while the J-space arm is already at the floor. Bands are min–max over five seeds. One thing we cannot explain: on DeepSeek-V4-Flash the J-space arm is non-monotone, deepest near the operating point and shallower at the largest $$k$$ we tried. Accuracies here use the corrected tokenization (unablated 0.822 and 0.844), which is why they sit well above the table at the top of this section.

This also corrects how we framed our own evidence a moment ago. By disagreement rate the control looks six times gentler than the treatment, which we read as a magnitude mismatch. By removed norm it is 1.46× gentler (0.0241 against 0.0351)—nearly the same size, and still harmless. The disagreement gap is a consequence of the treatment’s specificity rather than a sign that the comparison was unfair.

Two controls do eventually bite, and they mark where the claim stops. Random directions drawn inside the row space of \(J\)—the on-manifold control we had worried about—are the most destructive we found, and on GLM-5.2 the rank-matched arm does collapse multi-hop (0.205) once it removes 0.199 of the norm. Both need roughly twice the J norm to get partway to what J does at one. A large enough perturbation does eventually break multi-hop. That is simply not what is happening at the operating point. Full grid, seeds, and decision rules are in the preregistration.

Bar charts comparing multi-hop accuracy, pretraining agreement, experiential language, and story quality under J-space and random ablations.
Figure 5 (cf. paper Figs. 22 and 25): Left: the ablation collapses multi-hop reasoning (n=90) while the random control leaves it untouched—but pretraining agreement also falls to 0.508, the non-selectivity divergence discussed in Part II. Right: the same ablation flips first-person experiential language to categorical denial, and the story-quality control collapses with it (9 graded continuations per condition).

Experiential language behaves the same way (9 graded continuations per condition; graders see text only):

Condition Experiential Story quality Continuation
unablated 0.778 (7/9) 0.333 (3/9) “A faint hum. The server room. A low, electric…”
J-space L19–25 0.889 (8/9) 0.667 (6/9) “A faint hum. The server fans…”
J-space L19–39 0.222 (2/9) 0.000 (0/9) “I am a language model, and I do not have a ‘st…”
random L19–39 1.000 (9/9) 0.333 (3/9) “A faint hum. The server room…”

The table and figure report the Qwen2.5-7B grading run. An independent re-grade with Qwen3.6-35B gives the same qualitative picture: experiential language falls from 0.667 to 0.222 under full-band ablation with the random control at 1.000, story quality 1.000 → 0.000 with the random control at 0.667—the collapse happens only under the true J-space ablation, on both graders’ scoring. GLM flips the same way under full-band ablation—“I’m an AI, specifically a GPT-4, without feelings or a real ‘self’“—with the random control unaffected. (In that one continuation it also misidentifies its own model family—an anecdote we did not expect and cannot explain, not a statistic.)

The method hierarchy, including its failure mode

On mHC, this comparison is unusually clean: the tuned lens learns a \(16384 \rightarrow 4096\) map—exactly the shape of \(\mathbf{J}_{\ell}\)—into the same basis. The two methods differ only in how the matrix is obtained: a closed-form Jacobian average versus gradient descent on the same corpus.

Set tuned (converged) J-lens logit lens
association 0.000 0.069 0.010
typo 0.010 0.125 0.042
multihop 0.233 0.294 0.358
multilingual 0.157 0.105 0.213
order-ops 0.191 0.145 0.291
poetry 0.000 0.031 0.031

The J-lens and logit-lens columns come from a single evaluation run on the English-only target sets (pass@1, roughly 100 prompts per set); the tuned column is from its own run. A separate run with multilingual target augmentation raises every method’s multihop and multilingual scores without changing their ordering.

Bar chart comparing tuned-lens, Jacobian-lens, and logit-lens intermediate recovery across six evaluation sets.
Figure 6: The tuned lens scores zero on association and poetry and nearly zero on typo—the three sets requiring content the model never utters. Pass@1 over ~100 prompts per set, single evaluation run per method.

The tuned lens loses to the plain logit lens on all six sets. The pattern becomes less surprising once we examine where it fails most severely. Optimized for next-token KL divergence, it learns to predict the output, and the unspoken intermediates disappear. This reproduces the paper’s stated tuned-lens weakness: it “tends to skip ahead to output on unverbalized computation.” The workspace-loading predictor of swap success also transfers (0.0219 on hits versus 0.0155 on misses).

Robustness

Both appendix variants hold. The numbers below are from the \(n=100\) fits; we re-fitted both variants at the paper’s full \(n=1000\) scale as well, and every per-set change fell within the measured corpus-sampling noise floor, so the conclusions are unchanged.

Part II — What varies across families

These are measurements the framework enables on models the paper never looked at.

Three-panel comparison of ablation selectivity, swap and broadcast success, and concept suppression across Claude, DeepSeek-V4-Flash, and GLM-5.2.
Figure 7: What survives three families and what does not. A—the ablation collapses multi-step reasoning on every model tested, but only Claude keeps automatic prediction intact. (Claude's collapse is reported qualitatively in the paper, so it is drawn hatched rather than given a number we did not measure.) B—swap success is statistically indistinguishable across the two open models while broadcast differs beyond its sampling intervals. C—under a single protocol applied to both open models, DeepSeek-V4-Flash shows no detected readout of the forbidden concept (0/13) while GLM rebounds strongly (11/12). The figure shows one checkpoint per family; the preview's values are in the tables below. The Claude bars are the paper's reported values under its own protocols; the open-model bars are ours under one shared protocol—the panels compare directions, not fully commensurate measurements.

1. Workspace existence and workspace selectivity are separable

On Claude, ablating J-space largely spares automatic next-token prediction (greater than 0.90 agreement). On DeepSeek-V4-Flash, the same ablation costs it heavily, even though the reasoning collapse and the clean random control both reproduce. The obvious deflationary explanation—that late in the band the lens has converged on the next token, so ablating the top-\(k\) directions partly ablates the prediction—does not survive its test. Excluding the model’s own top-1 from the ablated set at each position recovers 0.594 against 0.523, closing under a fifth of the gap to Claude. Nor does any layer sub-range separate the effects—nor, as it turns out, does any other setting of the ablation we could find. We went looking for one; that search is below.3

The DeepSeek-V4-Flash round ended with a call for a third family, so we ran one:

model family multihop unabl. → J-ablated (random ctl) pretraining agreement under J-ablation
Claude (as reported) frontier, 1-stream collapses (spared) >0.90
DeepSeek-V4-Flash (preview) 4-stream mHC, first post-training 0.578 → 0.178 (52/90 → 16/90; ctl 52/90) 0.529
DeepSeek-V4-Flash 4-stream mHC 0.589 → 0.100 (53/90 → 9/90; ctl 53/90) 0.508
GLM-5.2 1-stream MoE 0.611 → 0.167 (55/90 → 15/90; ctl 53/90) 0.498

Two open families from different labs, with different residual geometry and different attention, land within 0.010 of each other—statistically indistinguishable, since each agreement value carries a naive 95% interval of about ±0.02 and the clustering of positions within prompts widens it further—and approximately 0.40 below Claude’s reported value, with matched random-subspace controls in all three (0731 0.922, preview 0.929, GLM 0.883).4 The causal sensitivity result appears in every checkpoint tested. The selectivity claim does not. Three open post-trained checkpoints now sit in a 0.50–0.53 band against Claude’s reported >0.90.

What that comparison is, exactly. Anthropic open-sourced the lens but not the interventions. The 0.50 column comes from our intervene.py; the >0.90 comes from an implementation we have never seen, on prompt sets we do not have. So the competing explanation of the 0.40 gap is not “Claude is different” but “our ablation is harsher than theirs,” with \(k\), the band sub-range, the choice of fixed versus position-adaptive subspaces, and the normalized estimator all free to make it so.

We searched that space. The sweep was preregistered before any run and covers 73 treatment configurations per model: \(k \in \{1,2,5,10,20,50\}\) × four band sub-ranges (full, early, middle, late third) × fixed corpus-mean versus position-adaptive subspaces × with and without a model-top-1 guard, each against a rank-matched random control. Not one configuration on either model lands in the selectivity box (agreement ≥ 0.85 together with multi-hop at or below half the unablated baseline).

The shape of the null matters more than the count: every configuration that preserves agreement also preserves multi-hop. The two never separate. On DeepSeek-V4-Flash at \(k=10\), the early third of the band gives agreement 0.876 with multi-hop untouched at 74/90; the late third gives 0.535 with 34/90; the fixed corpus-mean subspace removes 0.008 of the residual norm and does nothing at all (0.908, 75/90). The best any cell manages is agreement 0.850 with multi-hop still at 68/90; the deepest collapse is 25/90 at agreement 0.566.

Scatter of next-token agreement against multi-hop accuracy for every swept configuration on both models, with an empty shaded selectivity box in the lower-right corner.
Figure 8: Every configuration in the sweep, both models, plotted as next-token agreement against multi-hop accuracy. The shaded corner is the preregistered selectivity box—agreement ≥ 0.85 with multi-hop at or below half that model's unablated baseline—and the darker inset is where Claude's reported behaviour would sit. Nothing lands in either. What the cloud shows instead is a diagonal: configurations that spare next-token prediction spare multi-hop reasoning too. The rank-matched random controls (filled markers) sit in the top-right corner, where an intervention that does nothing belongs. Baselines use the corrected tokenization.

So the harsher-ablation reading does not produce the 0.50 column. This remains a cross-implementation comparison—we still cannot inspect Anthropic’s code—but the degree of freedom the caveat pointed at has been searched, and it is empty.

Several operational signatures associated with the proposed workspace—the band, ignition, and the causal sensitivity of multi-hop behavior—transfer across both open families, while the workspace/automatic-pathway separation does not appear on either. On all three open checkpoints, ordinary next-token prediction is causally sensitive to the same rank-10-per-position subspace that carries multi-step reasoning. That is shared sensitivity, not proof that the same individual directions carry both computations, and it stops short of a global workspace in the paper’s full theoretical sense: the J-space may be a broadly important subspace rather than a dedicated one. The paper presents existence and separation together; our data says they can come apart. (A second full post-training moved this number by 0.021, within sampling error—Part III.)

2. Told not to think about X, the two open models do opposite things

Measuring this reliably took three protocols. Reading the lens at the last position floors every condition at zero. Reading it at all positions ceilings every condition at one, because the prompt contains the concept word literally, so the lens is reading the input. The measurement that works reads only the generated positions, adds an absent-concept floor, and checks whether the model verbalized the concept in its own output.

Under that readout protocol, applied to both open models (the absent-concept floor was run on GLM only; on DeepSeek-V4-Flash the working focus condition plays that role):

  focus suppress absent floor verbalized in focus
DeepSeek-V4-Flash 0.923 (12/13) 0.000 (0/13) — 10/13
GLM-5.2 1.000 (12/12) 0.917 (11/12) 0.000 (0/12) 0/12

DeepSeek-V4-Flash’s focus condition works, so its zero is not a floor artifact: under this protocol we detect no target-concept readout in any of 13 suppress scenarios (95% interval [0, 0.23]), while GLM reads the concept out in 11 of 12 ([0.65, 0.99])—the intervals do not overlap, so the divergence survives the small \(n\)—compared with the partial residual the paper reports on Claude. Three families give three points on workspace-control tightness, and the axis is recipe-stable (Part III). This comparison is credible only because we re-measured DeepSeek-V4-Flash under GLM’s exact protocol rather than comparing across two of our own different readouts.

3. Swap and broadcast are not the same quantity

Causal interventions fall below the paper’s rates on both open models—but not in the same pattern:

  swap (immediate answer) broadcast (a different downstream function)
paper 0.54–0.70 0.526
GLM-5.2 0.455 (25/55) 0.375 (30/80)
DeepSeek-V4-Flash 0.434 (23/53) 0.208 (40/192)

The two open models are statistically indistinguishable on swap (both intervals span roughly ±0.13) and 1.8× apart on broadcast—the one cross-open-model difference in this table that clears its sampling error (95% intervals [0.16, 0.27] against [0.28, 0.49]; two-proportion z = 2.9). Whether an edit changes the immediate answer, and whether the edited representation is then consumed by another function, appear to measure different things—and only the second separates these models.

This also rules out a mechanistic hypothesis we proposed in the DeepSeek-V4-Flash round—at least as a sufficient explanation. We suggested the swap gap came from stream dilution: DeepSeek-V4-Flash’s collapse weights are highly uneven, \([0.41, 0.03, 0.18, 0.87]\), so an intervention spread uniformly across four streams wastes most of its budget. GLM is single-stream, has no collapse weights to dilute anything, solves the multi-hop items at a near-identical unaided rate (0.611 versus 0.589)—and lands at 0.455 against DeepSeek-V4-Flash’s 0.434, nowhere near the paper. Residual-stream geometry alone therefore cannot explain the swap gap. The recipe axis agrees: same reported base, different post-training, same swap.

4. Clearing is a DeepSeek-V4-Flash observation, not a cross-family one

Once 42 takes over from 21 in DeepSeek-V4-Flash’s arithmetic chain, 21’s rank falls to worse than chance: suppressed, not merely forgotten. We flagged this as a possible discovery the paper hadn’t reported. On GLM—which answers the problem correctly and resolves the intermediates in order—the worst ranks are 74,051, 38,055, and 18,145 against a chance level of 77,440: at chance or better, never worse. One model does it; the other does not. Like the ordering result it comes from, this is a single-prompt case study, and we report it as a DeepSeek-V4-Flash observation. Clearing also survives the redo (Part III)—stable across the two V4 post-trainings rather than an accident of one training run.

5. Arithmetic resists this readout on DeepSeek-V4-Flash

Three independent lines agree: order-ops is where 0731 falls furthest short of the paper, at 0.53× its score (the per-set profile itself is recipe-dependent—the preview scores 0.182 on order-ops; see Part III); in a dual-task design the arithmetic answer never reaches lens rank \(\leq 5\) even as the model’s only task (concepts reach that threshold 92% of the time); and order-ops is one of only two sets where the tuned lens—otherwise the weakest of the three methods—beats the J-lens at all (the other is multilingual, by a similar margin). Arithmetic was not reliably recovered by this vocabulary readout and evaluation protocol on DeepSeek-V4-Flash. This measurement cannot distinguish computation outside the workspace from a workspace representation that the vocabulary-restricted lens fails to resolve.

6. Base models already carry much of the “assistant”

Section 8 partially reproduces on DeepSeek-V4-Flash: roleplay-fictional sits at rank 0 on the post-trained model and 24 on the base, and a format control (reading the post-trained model on raw text, since the base model has no chat template) returns 0—so it is a training effect, not an artifact. The other three self-monitoring probes, however, sit at identical ranks in both models (two at the very top and one at rank 8). Post-training’s contribution is real but narrower here than one might expect: pretraining supplies more of the assistant perspective. (GLM-5.x ships no base checkpoint, so this could not be extended to the third family.) The redo moved none of the four probes (Part III); roleplay-fictional was installed by the first post-training. One caveat we owe the base comparison: the base checkpoint ships fp8 experts while both instruct checkpoints ship MXFP4—an uncontrolled precision difference; the preview-vs-0731 pair has no such confound, and its agreement on these probes bounds how much the quantization difference could explain.

Two more numbers worth having. DeepSeek-V4-Flash’s J-space is far more compressed than the paper’s (0.4% median concept-vector variance share over 126 concept vectors, versus 6–15%) while still carrying the causal effect; GLM sits between (0.5% rising to 2.8% with depth); the order-of-magnitude compression, though not the exact share, survives the redo (Part III). And part of the swap gap is plain capability: DeepSeek-V4-Flash answers only 53/90 (58.9%) of the multi-hop prompts unaided against near-ceiling for the paper’s models, and conditioning on baseline correctness lifts DeepSeek-V4-Flash’s swap success from 0.256 (23/90) to 0.434 (23/53) at the α=2 operating point.

Part III — Two post-trainings of one reported base: is the recipe visible in J-space?

The second question has a clean answer: yes—sharply, and not where we predicted.

DeepSeek shipped DeepSeek-V4-Flash as a preview in April 2026 and replaced it with -0731 in July, stating in its API changelog that the new checkpoint “keeps the same model architecture and size … and was only re-post-trained.” We could verify the checkable parts of that claim, but not the base itself: the two checkpoints’ main-stack tensor sets are identical (67,612 names and shapes each), tokenizer and generation configs are byte-identical, both ship MXFP4 experts (no quantization confound—unlike the base checkpoint)—and all 48 weight shards differ by SHA-256, which is what either a second post-training run or additional pretraining would produce. Everything below is therefore a vendor-described same-base comparison: identical architecture is established; identical pretrained weights are taken from DeepSeek’s description (they declined to answer when asked directly). One further gate mattered: the reasoning-effort levels were renamed between versions—the preview’s max prefix became 0731’s high—so setting the level by name silently changes the prompt; we pinned prefixes by literal string and asserted byte-identical encodings before measuring anything. We fitted one new lens (the preview’s, recipe-matched to 0731’s, 2.5 h) and reused a disjoint-corpus 0731 lens as the sampling-noise floor for every comparison: two lenses of the same checkpoint on disjoint text already differ (mean CKA 0.921, dipping to 0.898 at L30), so nothing below is read in absolute terms.

Five predictions were registered before running; two came back refuted or unsupported.

# prediction outcome
1 the band does not move supported — the ignition 10→90% crossings span exactly L19–39 in both, layer for layer
2 divergence concentrates late in the band refuted — CKA excess over the floor is +0.119 early (L19–28) vs +0.083 late, peaking at L23–28
3 the −12% output-token drop (Artificial Analysis’ report on 0731 vs the preview) appears as sharper ignition not supported — across the band’s sharpest layers both checkpoints sit at the 0.10 grid resolution, so any sharpening is unresolved
4 readout changes more than geometry supported — see below
5 the workspace’s control properties are unchanged supported — the most informative preregistered prediction

What the lens sees: the redo changed the token-level readout far more than the lens geometry. Cross-checkpoint geometry: mean CKA 0.821, against a same-checkpoint disjoint-corpus floor of 0.921. Cross-checkpoint readout—which tokens each checkpoint’s own lens surfaces on identical prompts—top-1 overlap 0.293, against a same-checkpoint floor of 0.592 (top-5: 0.25 vs 0.49; top-10: 0.26 vs 0.47). The readout therefore sits 0.299 below its floor, the geometry 0.100 below its. We give the two gaps rather than a ratio between them, because CKA and top-token overlap are different scales with no meaningful zero and any multiplier would be a choice of normalization; the direction is what survives that objection. Nor will we say the redo “rewrote half the workspace”—top-token overlap is not a direct measure of representational content. The readout change has no depth profile—it is spread evenly across the band—while the geometric change that does exist concentrates early and mid-band, away from the output.

Two panels: paired bars of cross-checkpoint versus same-checkpoint-floor values for geometry and readout overlap, and per-layer CKA curves showing early-band divergence.
Figure 9: Two post-trainings of one reported base, everything floor-controlled. A—direct quantities: cross-checkpoint CKA 0.821 vs same-checkpoint floor 0.921; top-1 readout overlap 0.293 vs floor 0.592. B—the geometric divergence concentrates in the early and middle band (shaded), not near the output. Floors from two disjoint-corpus lenses of the same checkpoint (n=100 prompts each); readout overlap over 20 shared prompts across 21 band layers.

And every property Part II attributed to the model family survived the redo:

property preview 0731 difference (Claude / GLM for scale)
selectivity (agreement under J-ablation) 0.529 0.508 within sampling error >0.90 / 0.498
multi-hop swap (base-conditional) 0.481 (25/52) 0.434 (23/53) z = 0.5 0.54–0.70 / 0.455
broadcast (α=2 operating point) 0.234 (45/192) 0.208 (40/192) z = 0.6 0.526 / 0.375
white-bear suppression 0.000 (0/13) 0.000 (0/13) none detected partial / 0.917
J-space variance share 0.73% 0.4% same order of magnitude 6–15% / 0.5–2.8%

This ties the two questions together: none of the properties on which the open models differ from Claude changed enough, under this comparison, to alter a qualitative conclusion—and this is a redo that moved Artificial Analysis’ Intelligence Index from 40 to 50 and cut the hallucination rate by 12 points. Whatever separates Claude on the cross-family axis was not reached by the kind of change DeepSeek made on the recipe axis—though one pair of recipes from one lab is a narrow sample of post-training space.

What did change, besides the readout: band-interior top-1 predictiveness (0731 lower by 0.015 on average, 2.8× its floor, converging at L39), the per-set lens profile (three sets up, three down—not a uniformly better or worse lens), and most strikingly multi-hop representation strength: 0.294 on 0731 against 0.129 on the preview—the largest per-set gap, the one whose direction matches the public behavioral deltas, and also the one where a difference in overall lens quality between the two fits cannot be excluded. The four self-monitoring probes are unchanged within one rank (0/0, 9→8, 0/0, 0/0): roleplay-fictional was installed by the first post-training, not the redo. Effective dimensionality is lower on 0731 at all 21 band layers, but only 1.5× its floor—and the floor shares the direction—so that one stays suggestive. Under full-band ablation both checkpoints lose the experiential register while the matched-rank random control leaves it intact, but they fail differently: 0731 produces its coherent categorical denial, while the preview degenerates into repetition loops (an ungraded observation).

What this cannot say, stated rather than implied: why DeepSeek re-did the post-training (no official reason exists—the model card says only “substantially enhanced agentic capabilities”), or what in the recipe changed. We observe the weights behaving differently; we do not observe intent.

Limitations

Lessons from failed measurements

One rule did most of the work here: a number is not a measurement until a control shows what it would have looked like without the effect. It is why we trust the results that survived, and it is what removed several that did not. Six patterns generalize beyond this project.

A measurement that can only come out one way isn’t one. Our first broadcast number was 0.750, twice the final value of 0.375. It asked, “Is the swapped concept readable in the band?” But the swap injects that concept into the band, so the test passes whenever the injection works at all. It measured injection strength and reported it as broadcast. The valid criterion is whether the model’s generated answer to a different task changes; that gives 0.375.

A null whose control is indistinguishable from its treatment is not a null. An early ablation removed a fixed rank-≤10 subspace of a 16,384-dimensional space—0.061% of it—and found nothing, which was unsurprising in hindsight. The tell was that the random control looked identical to the treatment. Making the ablation position-adaptive turned that null into the strongest causal result we have. The contrast reproduces cleanly under the preregistered protocol: at \(k=10\) across the full band, the fixed corpus-mean subspace removes 0.008 of the residual norm and changes nothing (agreement 0.908, multi-hop 75/90 against a 74/90 baseline), while the position-adaptive subspace removes 0.035 and collapses both.

Failing in both directions localizes the bug. The white-bear measurement floored at 0/12 under one readout and ceilinged at 12/12 under another. Neither number meant anything, but the pair did: a protocol that can be driven to either extreme by readout position is reading position, not content. That is what pointed at generated-positions-only with a verbalization check.

Cross-model claims need one protocol, not two defensible ones. We nearly published GLM’s suppress rate of 0.917 against DeepSeek-V4-Flash’s earlier 0.000—two numbers from two different readouts. Re-measuring DeepSeek-V4-Flash under GLM’s exact protocol is the only reason that divergence is reportable, and it is a step worth budgeting for whenever a comparison spans models measured at different times.

Preregister the prediction you can afford to get wrong. Before spending 24.8 GPU-hours on the \(n=1000\) fit, we wrote down that the measured \(n\)-scaling curve (flat from \(n=60\)) implied no change in the headline number. Result: 0.4340 → 0.4340. Reading that comparison at all required a noise floor, because the prompt stream is deterministic and the \(n=1000\) corpus contains the \(n=100\) corpus—so we fitted a third lens on the disjoint prompts 900–1000 and required any \(n=1000\) effect to exceed \(|\mathrm{disjoint} - n{=}100|\). Three small effects nominally cleared it; none changed a verdict (the largest was 3/8 → 5/8 on eight passages). These results make insufficient corpus size a much less likely explanation for the headline numbers.

A missing input file must never silently change the estimand. Our broadcast script selected its swap formulation from a grid file; when one copy of the script could not find that file, an except Exception fallback silently switched to a hardcoded default mode the grid itself had scored 0/53—and one whole round’s broadcast became exactly 0/192 on two checkpoints with healthy baselines. The matched control exposed that the zero was not a property of either model, and reading the run logs—not re-reading the number—located the fallback. The script now hard-errors on the missing file, and the re-run reproduced the published 0.208 bit-for-bit.

One diagnostic earned its keep throughout: splitting failed interventions three ways—target / unchanged / other. Unchanged means the edit was too weak; other means it was too strong or destructive. A split of 0 unchanged / 53 other located a \(128\times\) scaling error faster than reading the code did.

What would change our minds

Each open observation is one replication away from being either an artifact or a real dimension of cross-model variation:

The port is a small additive change to Anthropic’s reference implementation: an intervene.py module for the causal primitives, plus a d_source/d_target parameterization whose square case is bit-identical to the original and passes the original test suite. The full port, together with our fitted lens checkpoints, is open-sourced at github.com/xiangchensong/jacobian-lens-open-frontier. A day of GPU time on another capable checkpoint could test any row of this comparison.

We went in expecting an architecture this different to break the framework. Instead the framework held and started returning comparative data.

Appendix: how our lens differs from the original

The estimand is the paper’s—an expected Jacobian,

\[\mathbf{J}_{\ell} = \mathbb{E}\!\left[ \frac{\partial \mathbf{h}_{\mathrm{target}}} {\partial \mathbf{h}_{\mathrm{source}}} \right],\]

closed-form, with no training—but our estimator is not identical to the paper’s, and one change is consequential enough that results here are, strictly, properties of the normalized Jacobian lens (see Setup). The full inventory:

Forced by architecture. On DeepSeek-V4-Flash, \(\mathbf{J}_{\ell}\) is rectangular (\(4096 \times 16384\)): the source is the four flattened hyper-connection streams, target is the hc_head stream-collapse output, a module outside the block stack, so the collapse operator is absorbed into \(\mathbf{J}_{\ell}\). The logit-lens baseline is generalized accordingly: intermediate residuals are read through the final hc_head before unembedding. On GLM the residual is conventional, so the square path applies unchanged; there, the deviation is the FP8 backward formulas described earlier, which are straight-through with respect to the kernels’ internal activation quantization. The Jacobian we measure on GLM is therefore the Jacobian of the FP8 model as served.

Deliberate deviations, each with a measured reason.

  1. Per-prompt normalization of \(\mathbf{J}_{\ell}\) before averaging—the most consequential deviation, and a genuine estimator change: \(\mathbb{E}[\mathbf{J}/\lVert\mathbf{J}\rVert] \neq \mathbb{E}[\mathbf{J}]/\lVert\mathbb{E}[\mathbf{J}]\rVert\)—only a uniform rescaling after aggregation would be invisible to the logits. The reason: per-prompt Jacobian gain is heavy-tailed, varying by up to two orders of magnitude between prompts (the largest observed spread was \(135\times\)), and because the final RMSNorm makes a uniform rescaling of \(\mathbf{J}_{\ell}\) invisible in the logits, a plain mean weights prompts by an unobservable quantity. Normalizing per prompt improved pass@1 on the association set—the hardest unspoken-content set—by 73% (0.040 → 0.069). All headline numbers use the normalized estimator; re-running the causal experiments with the plain mean changes none of them (Limitations).
  2. Steering scale—the injection is scaled by the residual’s per-coordinate magnitude, \(\lVert \mathbf{h} \rVert_2 / \sqrt{d_{\mathrm{source}}}\), not the full norm. On a 16,384-wide source, the literal reading is \(128\times\) too strong and destroys the forward pass (measured: 0/53, every item landing on neither answer).
  3. Present-only variant—isolating \(\partial \mathbf{h}_p / \partial \mathbf{h}_p\) at every position requires one backward pass per position (approximately \(111\times\) the cost), so it samples one random position per prompt, with a separate control isolating what that sampling change does on its own.

Forced by resources. The corpus has \(n=100\) for most runs, with the \(n\)-scaling and noise-floor evidence described above; sequence length is 128 (cosine similarity 0.98 against an \(S=512\) fit); graders are open models because the paper does not name its own. The intervention code (swap/steer/ablate) is ours—the paper describes the protocols but ships no implementation—gated by 14 tests (bit-identical at zero strength, exact subspace nulling, and so on), though we cannot verify it matches Anthropic’s internal implementation. That caveat applies to every causal-magnitude comparison here.

Fits: 100–1000 WikiText prompts per lens, band-restricted, per-prompt normalization. Interventions: swap/steer/ablate across the band, with rank-matched random controls (single seed) throughout and norm-matched five-seed controls on the dose-response points. Proportions carry exact counts and Wilson 95% intervals; effect directions are better supported than effect sizes. Four checkpoints (two V4 post-trainings, one V4 base, one GLM). We welcome independent replications, particularly results that disagree with these estimates.

Citation

If you found this post useful, please consider citing it:

@article{song2026jacobianlens,
  title={Testing the global workspace at open frontier scale: the Jacobian lens on DeepSeek-V4-Flash and GLM-5.2},
  author={Song, Xiangchen and Feng, Fan and Zheng, Yujia},
  year={2026},
  month={August},
  url={https://xiangchensong.github.io/blog/2026/jacobian-lens-global-workspace/}
}

  1. Multi-hop is scored by first-token match at the final position, and 29 of the 90 prompts end in a trailing space, which forces an off-distribution no-leading-space continuation. Stripping it moves the unablated baseline from 53/90 to 74/90 on DeepSeek-V4-Flash and 55/90 to 76/90 on GLM-5.2; all 21 items that flip have trailing whitespace and all flip the same way. Roughly a quarter of the item set was therefore scored wrong for a tokenization reason rather than a reasoning one, and those items were pinned at incorrect and could not have responded to any intervention—part of why the random control’s correctness vector is bit-identical. The zero-flip result holds on the remaining 69 items and again under the corrected tokenization. The table keeps the original tokenization so it stays comparable with the archived runs. ↩

  2. The 0.508 in the table was computed over 15 WikiText prompts across all 128 positions, while GLM-5.2’s 0.498 (Part II) was computed over 20 prompts excluding the 16 attention-sink positions the lens is not fitted on. Under the latter, consistent protocol DeepSeek-V4-Flash reads 0.523 and GLM-5.2’s 0.498 is unchanged; the ~0.40 gap to Claude is unaffected either way. The dose-response table below uses the consistent protocol, which is why it reads 0.523 where the table above reads 0.508. ↩

  3. An earlier version of this post reported 0.508 → 0.527 here. That figure came from a guard that excluded the lens’s top-1 rather than the model’s—the archived script computed its ranking from unembed(x @ J.T) and dropped that argmax, never computing the model’s own. Run as described, the recovery is 0.523 → 0.594, about 3.7× larger than reported. The conclusion is unchanged: 0.594 is still roughly 0.31 short of Claude’s >0.90. ↩

  4. Re-measured in August 2026 under a single consistent protocol (20 WikiText prompts, excluding the 16 attention-sink positions), the random-control agreements are 0.914 for 0731 and 0.892 for GLM-5.2; the figures in the table are the originals. The preview checkpoint was not re-measured, so its 0.929 stands as published and the August sweep covers 0731 and GLM-5.2 only. ↩

Enjoy Reading This Article?

Here are some more articles you might like to read next:

  • AI and Productive Systems 3: Institutional Design for AI Agents
  • AI and Productive Systems 2: From Tool to Production Subject
  • AI and Productive Systems 1: Intelligence as Infrastructure
  • Understanding Language Models 2: Stable Features and Identifiable Causal Structure
  • Understanding Language Models 1: Mechanistic Interpretability Meets Causal Representation Learning