12 The pipeline as matrices

Figure 19 brings the stages together. Shapes are a useful final check because they force us to say what each row and column represents. They can detect incompatible operations, although matching shapes alone do not prove that row identities, units, weights, or meanings agree.

Here $N$ counts articles and $E$ is their $N\times384$ embedding matrix. This use of $E$ names embeddings, distinct from the local covariance-perturbation symbol in the stability section. Each row is an article; each column is an embedding coordinate. Let $\mathbf1$ be the $N\times1$ column of ones and $\mu$ the $384\times1$ embedding mean. The product $\mathbf1\mu^{\top}$ repeats that mean in every article row. Subtracting it centres all rows at once.

The retained basis $V$ has 384 rows and 60 columns. Define the raw coordinate table $C$ by

$C=(E-\mathbf1\mu^{\top})V.$

The multiplication has shape $(N\times384)(384\times60)$ , producing $N\times60$ . Each article receives 60 coordinates. To reconstruct the retained part in the original embedding coordinates, multiply by $V^{\top}$ , giving shape $(N\times60)(60\times384)=N\times384$ , then add the repeated mean. The difference from $E$ gives one residual row per article.

The term-space mean remains $\mu_T$ . The table $S$ denotes the standardized raw score rows used for lexical naming; it has the same article ordering as the term matrix. Its cross-product with centred terms has shape $(50{,}000\times N)(N\times60)=50{,}000\times60$ . The $60\times60$ rotation changes the coordinates within the retained space without increasing their number. The subscript “lsa” below identifies a basis fitted in term space.

The numerical decompositions are consequently small—a $384\times384$ covariance, a $384\times60$ basis, a $50{,}000\times60$ loading table, and a $60\times60$ rotation—even though some article and story tables remain large. The preceding chapter identifies the objects that still determine memory requirements; small final matrices do not imply that every intermediate calculation is small.

The pipeline, stage by stage, as matrices.

stage	object	shape	operation
tokenise	TF‑IDF matrix $T$	$N \times 50{,}000$ , sparse	randomized SVD → $V_{\text{lsa}}$ , $U$
embed	$E$	$N \times 384$	streaming $Z, m, M$
covariance	$\Sigma$	$384 \times 384$	`eigh` → $V, \Lambda$
name	centred term–score product $\beta$	$50{,}000 \times 60$	varimax → $R$
measure	$C = (E - \mathbf{1}\mu^{\top})V$	$N \times 60$	$T^2$ , $Q$ , $\nu$ per row
stories	centroids	$842{,}278 \times 60$	profiles, precedents
days	spectra	$10{,}102 \times 60$	era statistics, z‑scores

The numbers in the table are those of the first edition on 28 August 2026; each grows by a day’s worth every night. The second edition, fitted on 3 September 2026 over the New York Times archive as well, has $N = 16{,}500{,}781$ articles (455 million non-zeros in $T$ ), 13,421,090 stories and 63,270 day spectra; the small matrices— $\Sigma$ , $V$ , $\beta$ , $R$ —have exactly the same shapes, which is the point of the chapter Streaming the computation.