Abstract

News is mostly recurrence. A war, a scandal in the capital, another bad turn in the Middle East, a central bank surprising the market: the shape repeats while the names, places and pretexts change. Eigen Times is a newspaper that takes this literally. It treats the archive of news as a matrix, extracts its principal directions—the eigen-news, a few dozen recurring archetypes that explain most of the variance—and prints each day as a decomposition: how much of today projects onto each archetype, what is left over, and, for every story, what is the same as always and what is different this time.

This article describes how Eigen Times was built as a complete system: a 1.09-million-article corpus assembled from public archives (the Guardian, 1999→; Wikipedia’s Current events portal, 2002→; Wikinews, 2004–2026); two independent constructions of the eigenbasis, one in term space (the singular value decomposition of a centred TF-IDF matrix) and one in embedding space (the eigenvectors of the covariance of sentence embeddings), both rotated into named axes; a story layer that clusters each day’s articles and threads them into episodes; the statistics that turn coordinates into journalism—Hotelling’s $T^2$, the residual energy $Q$, a novelty ratio, per-archetype profiles, precedents; and the newspaper itself, a static site of 10,102 daily issues. We compare the two bases against the human-assigned event categories of the Wikipedia portal, which are never used in fitting, and find that both recover them at similar rates (42.4% and 40.0% section purity) while their matched axes correlate at 0.57–0.75. The entire backend is Rust; the pipeline runs on a laptop.

A second edition, added after the first was published, extends the archive to 16.5 million articles across 175 years by ingesting the New York Times archive (September 1851→) and refits the eigenbasis with day-balanced weights, so that every calendar day since 1851 counts the same. The two editions are served side by side with a switch on every page. On the same human-labelled events the 175-year basis recovers categories as well as the modern one (section purity 0.435 against 0.431) while spreading today’s stories across all sixty axes instead of piling most of them onto one. Fitting it forced the computation to stream: the orientation of axes, the term loadings and the story clustering were rewritten to run in flat memory over a corpus fifteen times larger than the first, and the paper reports how.

1 Introduction

Newspapers are organised by desk—World, Politics, Business—because that is how newsrooms are organised. Readers, however, experience the news as a small repertoire of recurring situations, and much of what a seasoned reader does when reading a headline is to recognise which situation it is and then look for what departs from the template. “Another regional war” is the recognition; “but this time two great powers are on opposite sides, and it is six weeks before an election in one of them” is the reading.

Eigen Times is an attempt to make that reading mechanical, honest and daily. Its thesis is that the space of news articles has low effective dimension: a few dozen directions carry most of the variance of a quarter-century of reporting, and those directions are the archetypes. Once the archetypes are known, a day’s news can be measured rather than merely categorised. Each story becomes a vector; its projection onto the basis says which archetype it instantiates and at what intensity; the part of the vector the basis cannot reproduce says how much of the story is genuinely new; and the archetype’s own history—its typical profile, its previous episodes—gives the reader the precedents against which “different this time” can be stated as a number before it is stated as a sentence.

The contributions of this article are:

1. a precise formulation of the eigen-news as the eigenvectors of the weighted covariance of story vectors, with the statistics that follow from it (§3);
2. a public-archive corpus of 1,093,166 dated articles and a story layer over it (§4);
3. two complete implementations of the basis—term-space LSA and an embedding eigenbasis—with a common naming and rotation layer, and a comparison against human labels (§5, §8);
4. the newspaper: a rendering of the decomposition as a daily issue with spectra, archetype badges, delta panels, precedents, archetype pages and a comparison page (§7);
5. an all-Rust implementation over Parquet with Sail as the query engine, deployed as a static site (§9).

Everything described here is running at eigentimes.com; the source is at github.com/alexy/eigentimes.

2 Previous work

2.1 Automation in the newsroom

Automated journalism has mostly meant automated writing: templated prose generated from structured data. The Associated Press began publishing computer-written earnings reports with Automated Insights in 2014; the Washington Post’s Heliograf covered the 2016 Olympics and elections; Reuters’ Lynx Insight (2018) and Bloomberg’s Cyborg turned market data into draft stories. Carlson [4] framed the phenomenon as “the robotic reporter” and examined how journalists narrated its arrival; Graefe’s Guide to Automated Journalism [10] surveyed the practice and its limits, chiefly that generation from structured data covers only the small fraction of news that arrives structured. Eigen Times sits on the other side of the newsroom: it does not write stories, it reads them, and what it automates is the editorial act of recognition and comparison.

2.2 Topic detection and tracking

The closest research lineage is Topic Detection and Tracking. The TDT pilot study [1] defined the tasks—segmenting a stream, detecting first stories, tracking known events—that any system reading a news stream must solve, and first-story detection in particular [22] is the ancestor of what this article calls the residual: the part of today that no existing direction explains. Industrial news aggregation, notably the clustering behind Google News [6], solved the daily grouping problem at scale. Eigen Times reuses the TDT decomposition of the stream into stories (a day’s cluster) and episodes (a chain of stories across days), but its unit of recognition is not the event; it is the archetype, a direction that recurs across many events.

2.3 Latent semantic structure of text

Latent Semantic Analysis [8] was the first demonstration that the singular vectors of a term–document matrix are meaningful directions, and it remains the most interpretable: a component is literally a weighted word list. Probabilistic successors—pLSA [13], LDA [2]—recast the directions as topics; dynamic topic models [3] let topics drift through time, which is the model-side counterpart of the refinement problem in §6. Topic models have been applied to news at scale [21], and meme-tracking [19] followed short phrases through the news cycle. Leetaru’s GDELT [18] took the opposite approach and coded the world’s news into a fixed event ontology; its themes and actors are a hand-built basis where Eigen Times wants a learned one.

Sentence embeddings changed what a document vector is. Word vectors [20] made semantic similarity linear; Sentence-BERT [23] produced document vectors in which cosine similarity tracks meaning rather than vocabulary; the BGE family [24] used here is a compact descendant. BERTopic [11] clusters such embeddings and names the clusters with class-based TF-IDF, which is close in spirit to how Eigen Times names its embedding axes (§5.2), except that Eigen Times keeps the axes orthogonal and continuous rather than discrete.

2.4 The matrix toolkit

The numerical ingredients are classical. Varimax rotation [16] makes a factor basis readable without changing the subspace. Hotelling’s $T^2$ [14] and the squared prediction error $Q$ [15] are the standard pair for monitoring a process against a principal-component model—here, monitoring the news against its own history. Randomized SVD [12] makes the term-space decomposition of a million-row sparse matrix a matter of minutes. The Davis–Kahan theorem [7] bounds how far an eigenvector can move when the covariance is perturbed, which is what makes a nightly refinement of the basis provably stable. Archetypal analysis [5] is the non-negative alternative to rotation for producing unipolar axes, and the Kuhn–Munkres assignment [17] matches axes across refinements and across bases.

2.5 The Wikipedia Current Events portal as data

Wikipedia’s Portal:Current events is a curated daily bulletin, one page per day since January 2002, with each event filed under a section such as Armed conflicts and attacks or Politics and elections. It has been used as a source for multi-document summarisation [9]. Eigen Times uses it twice: as a compact, consistent source of events, and—because the sections are human category labels that were never shown to the fitting procedure—as the held-out check that the eigenvectors recover categories people would choose.

3 The mathematics

3.1 Stories, vectors, weights

The unit of analysis is a story: the set of articles from one day that report one event. A story $s$ has a day $\tau(s)$, a vector $x_s \in \mathbb{R}^d$ (a sentence embedding of its headline and lede, $d = 384$ in the deployed system), a sparse term vector $t_s \in \mathbb{R}^p$ (TF-IDF over its text, $p = 50{,}000$, used for naming), and an energy $w_s = (\text{distinct sources}) \cdot \ln(1 + \text{articles})$, normalised so the mean weight per day is one. The archive to day $T$ is the weighted matrix $X_T \in \mathbb{R}^{N_T \times d}$ with $W_T = \mathrm{diag}(w_s)$.

3.2 The eigen-news

The eigen-news are the eigenvectors of the weighted covariance of stories:

$$Z = \sum_s w_s, \qquad \mu = Z^{-1}\sum_s w_s x_s,$$

$$\Sigma = Z^{-1}\sum_s w_s (x_s-\mu)(x_s-\mu)^{\top} = V \Lambda V^{\top}.$$

The top $k$ eigenvectors $V \in \mathbb{R}^{d\times k}$ span the news subspace; $\lambda_j$ is the variance of the archive along $v_j$. With $\tilde{X} = W^{1/2}(X - \mathbf{1}\mu^{\top})$ and thin SVD $\tilde{X} = U S V^{\top}$, the same $V$ appears with $\Lambda = S^2/Z$. Both factors are used: $V$ says what kind of news a direction is; the story scores $U$ say which historical stories are its purest instances, and the extreme entries of column $j$ are the canon of archetype $j$.

For choosing $k$ we retain eigenvalues above the Marchenko–Pastur noise edge $\lambda_{+} = \hat{\sigma}^{2} (1+(d/N)^{1/2})^{2}$ and check against the explained-variance curve; the deployed bases use $k=60$.

3.3 Orientation and named axes

Eigenvectors have no sign, and within nearly degenerate eigenvalue clusters no unique direction. Two deterministic fixes make them printable. Sign: each $v_j$ is oriented so that the skewness of its story scores is positive—news is sparse, and a war is a heavy positive tail on the war axis, so the heavy tail identifies the archetype pole. Named axes: the subspace carries the mathematics, but the printed axes are a varimax rotation $V' = VR$ inside it, chosen to make the term loadings sparse. Coordinates, residuals and statistics are computed in $V$; names and canons live in $V'$.

Term loadings need no language model. For component $j$, $\beta_j = T^{\top} u_j \in \mathbb{R}^p$—the TF-IDF matrix weighted by the component’s whitened story scores—ranks the words that characterise the direction. This is what makes a component print as troops · offensive · ceasefire · shelling before anything is named, and it is the same procedure for both bases, so their axes can be compared word for word.

3.4 Extraction: a story in the basis

For a story $x$ on day $t$, measured against the basis of day $t-1$:

$$c = V^{\top}(x-\mu), \qquad \hat{x} = \mu + V c, \qquad r = x - \hat{x},$$

$$T^{2} = c^{\top}\Lambda^{-1}c, \qquad Q = \Vert r\Vert ^{2}, \qquad \nu = \Vert r\Vert ^{2} / \Vert x-\mu\Vert ^{2}.$$

$T^2$ and $Q$ are the classical monitoring pair, and they are exactly the two senses of “different this time”: a high $T^2$ is a familiar archetype at unusual intensity or in an unusual mixture; a high $Q$ is something the archive has no direction for. The decomposition the paper prints is

$$x - \mu = c_j v_j + \sum_{i \neq j} c_i v_i + r, \qquad j = \arg\max_i c_i \lambda_i^{-1/2},$$

the archetype part, the mixture part and the residual respectively, with the dominant component chosen in whitened coordinates so that a small-variance archetype can dominate when it is strongly present.

3.5 Per day, per archetype

The day vector $\bar{x}_t = \sum_s w_s x_s / \sum_s w_s$ has spectrum $e_t = V^{\top}(\bar{x}_t - \mu)$. Against the running mean and variance of $e_{\cdot,j}$ over past days—the era statistics—each axis gets a $z$-score: loud, usual, or silent. For each archetype $j$ the stories it dominates define a profile: the mean and standard deviation of every coordinate over those stories. “Same as always” is the set of strong profile axes a story also carries; “different this time” is the set of axes on which the story departs from the profile by more than $1.5\sigma$. Precedents are the nearest earlier stories with the same dominant axis, one per episode, by cosine in named-axis coordinates.

4 The dataset

4.1 Sources

The first three sources were the fitted corpus of the first edition (1.09 million articles). The New York Times archive—headlines and abstracts for every item the paper indexed since 18 September 1851—was added for the second edition (§Two editions): 2,100 monthly calls, embedded as they arrived at about 550 articles per second on a laptop, for a corpus of 16,500,781 articles over 63,270 days. Its text changes register along the way: until the 1980s the “abstract” is the index’s terse summary (“Commerce Dept repts mfrs sales at record; illus”), with placeholder headlines for untitled items (“Article 8 – No Title”); afterwards it is the lede paragraph, for a decade prefixed with a literal “LEAD:”. Those conventions are cleaned before the text is tokenised for naming (§Two editions), and only for that source. Every source is read by an adapter split into a pure parser, unit-tested on fixtures, and a rate-limited, retrying fetcher; the whole backfill took one afternoon.

Two properties matter more than volume. Length: archetypes recur on scales of months to decades, so the basis needs decades. Consistency: a change in source mix shows up as a fake change in the world, so backfill and live stream must be embedded with the same pinned model and clustered by the same procedure.

4.2 Storage

Raw articles are append-only Parquet in a Hive layout, one file per (source, month), written atomically so that a finished partition is the resume marker for a backfill. Sail—a Spark-compatible engine in Rust—is the query engine over the layout (et sql registers raw/articles as a view with the partition columns), while the numeric pipeline reads the same files directly with arrow-rs. Every derived table (embeddings, coordinates, statistics, stories) is keyed by the model or basis version that produced it, so any day can be re-materialised against any basis.

4.3 Stories and episodes

Per day, articles from all sources are clustered by single-linkage on cosine similarity (0.72 in embedding space; 0.6 in the diffuser LSA space). A story’s representative is the member with the most text from a full-text source, so headlines come from newspapers rather than bulletin lines; a story’s section is the human section of a bulletin member, if any—newspaper desk names are deliberately not used, since only the bulletin sections are event categories. A story on day $t$ continues yesterday’s episode when its centroid agrees with a story of day $t-1$ at cosine $\ge 0.9$; episodes are chains, so a 600-day war is one episode, and “episode day 7” on a card means the seventh day of that thread. The deployed story layer has 842,278 stories over 10,102 days; 24 February 2022 yields a 62-article, three-source lead story, “Russia launches attack on Ukraine”, on episode day 7—the thread begins with the build-up.

5 Two algorithms

Both constructions produce the same objects—an orthogonal basis, a rotation to named axes, per-article coordinates, term loadings—and everything downstream (stories, spectra, profiles, precedents, the site) is computed identically for both. They differ in what a document is.

5.1 v0: term-space LSA

Each article is tokenised (lower-cased, stop-listed, Snowball-stemmed), counted into a 50,000-term vocabulary (document frequency between 20 and 30% of the corpus), and encoded as an $\ell_2$-normalised $(1+\ln \mathrm{tf})\cdot\mathrm{idf}$ row. The corpus matrix has 1,093,166 rows and 149.4 million non-zeros. Its column means are subtracted implicitly—$(X-\mathbf{1}\mu^{\top})\Omega = X\Omega - \mathbf{1}(\mu^{\top}\Omega)$—so sparsity is preserved, and the top 60 singular triplets are computed by randomized SVD with 20 oversampling columns and two rounds of subspace iteration, deterministic for a seed. The whole fit—two tokenisation passes over 1.6 GB of text, the SVD, the varimax rotation of the $50{,}000 \times 60$ loading matrix—runs in about five minutes on a ten-core laptop.

The rotated axes are word lists by construction, and they are strikingly clean. Among the sixty: bank · financial · credit · rate · loan · Barclays; party · Labour · Tory · conservative · leader · election; police · arrest · officer · protest · investigate · suspect · murder; price · rise · market · oil · inflation · rate · FTSE; Trump · Donald · president · Biden · Republican · immigrant; climate · emissions · carbon · global · warming · temperature; Israel · Palestinian · Gaza · Hamas · Netanyahu; flight · airline · passenger · airport · plane · crash; European · Europe · France · Germany · Brussels · euro; court · judge · rule · case · trial · legal; Russian · Russia · Ukraine · Putin · Moscow · Kremlin; covid · case · vaccine · pandemic · lockdown; house · property · home · mortgage · buyer; protest · demonstrator · activist · arrest; Korea · north · south · nuclear · Kim · Pyongyang; Ireland · northern · Irish · IRA · Sinn · Belfast; Brexit · deal · Johnson · Boris · negotiate · referendum; asylum · immigrant · refugee · seeker · migrant · border; prison · sentence · charge · jail · trial · convict; energy · gas · electricity · nuclear · carbon · renewable; attack · bomb · terrorist · terror · security · Islam; Syrian · Syria · Assad · rebel · Turkey · Damascus. The Gaza axis, the Ukraine axis and the Brexit axis are each one archetype with many episodes.

5.2 v1: the embedding eigenbasis

Each article’s headline and first 300 characters are embedded with a pinned local model (bge-small-en-v1.5, 96 tokens, CLS pooling, 384 dimensions, run through ONNX Runtime). Embedding is the only expensive step: about 45 documents per second per process, so two shards took roughly four hours for the corpus.

The basis is then exact. A first pass accumulates the sufficient statistics $(Z, m, M)$—$M$ is $384\times384$—in parallel over partitions and merges them; $\Sigma = M/Z - \mu\mu^{\mathsf T}$ is eigendecomposed with a dense symmetric solver in milliseconds; the top 60 eigenvectors carry 55% of the variance. A second pass projects every article, orients each component by skewness, and writes coordinates and $(T^2, Q, \nu)$ per article. A third pass re-tokenises the corpus with the LSA vocabulary and computes the term loadings $\beta = T^{\top}U$ for the embedding components, and a varimax rotation of that $50{,}000\times 60$ matrix gives the named axes.

The rotated embedding axes are mostly topical—Trump · president · Republican · Senate · Obama; covid · vaccine · coronavirus · pandemic · hospital; climate · energy · emissions · carbon · fuel; flight · airline · passenger · airport · aircraft; court · judge · sentence · trial · legal; Russian · Russia · Ukraine · Putin · Kremlin · missile; China · Chinese · Beijing · India · Hong Kong; car · vehicle · driver · road · manufacturer; church · pope · Catholic · bishop · Vatican; Ireland · northern · IRA · Belfast · Sinn—but a few are axes of register that no term matrix can see: think · thing · know · seem · want · question · feel (opinion and interview), resign · leave · down · fall · cut · suspend · cancel (the vocabulary of ending), claim · say · statement · deny · reject · ask (contested assertion). Whether these are archetypes or artefacts is an editorial question the comparison page leaves to the reader.

5.3 What the two constructions share and where they differ

Both are eigendecompositions of a centred second-moment matrix; the difference is whether the document is a bag of words or a point in a semantic space. The term basis is interpretable by construction and has no model to pin; the embedding basis gives genuinely semantic residuals—$Q$ measures what the archive cannot mean, not what it cannot spell—and clusters stories far better (in LSA space, same-event articles rarely exceed cosine 0.7; in embedding space the 9/11 day collapses into a 34-article lead story at 0.72). Their eigenvalue spectra differ in shape: LSA’s leading singular values decay slowly, and its top axes are broad (markets, party politics), while the embedding covariance concentrates variance in fewer directions. Section 8 quantifies the agreement.

6 Refinement and identity

The basis refines every night from sufficient statistics: $Z_t = Z_{t-1} + \sum_{s\in C_t} w_s$, $m_t = m_{t-1} + \sum w_s x_s$, $M_t = M_{t-1} + \sum w_s x_s x_s^{\top}$, then a fresh eigendecomposition—exact, and cheap at $d = 384$. What is not free is identity: eigh returns eigenvectors in eigenvalue order with arbitrary sign, components swap places when eigenvalues cross, and within a near-degenerate cluster they rotate freely. “Archetype 18” must therefore be tracked, not assumed. Each refinement matches new components to old by the Kuhn–Munkres assignment on $|\cos|$, fixes signs so matched inner products are positive, aligns near-degenerate groups by orthogonal Procrustes, and warm-starts the varimax rotation from the previous one so the printed axes move continuously. The Davis–Kahan bound, $\sin\theta_j \le \Vert \Sigma_t - \Sigma_{t-1}\Vert / \mathrm{gap}_j$ in the spectral norm, with $\Vert \Sigma_t-\Sigma_{t-1}\Vert \lesssim (\sum_{s\in C_t} w_s/Z_t)\max\Vert x_s-\mu\Vert ^2$, shows the daily drift is $O(1/t)$: the basis provably settles. A component with no match above 0.8 is born and announced; one whose eigenvalue falls below the noise edge is retired but kept in the registry so old issues still resolve.

Two ordering rules keep the measurement honest. The day’s issue is measured against the basis of day $t-1$ and only afterwards folded into $(m_t, M_t)$; otherwise today’s stories would partly explain themselves and $Q$ would be biased toward zero. And the basis uses the full archive with uniform weights—the canon of everything that has happened—while recency enters only through the era statistics; a recency-weighted basis would forget a pandemic or a financial crisis before it recurs.

The matching, alignment, statistics and rotation are implemented and tested in the et-basis crate; the nightly schedule that drives them is the next piece of work (§10).

7 Two editions

7.1 Day-balanced weights

The first edition’s basis is the basis of 1999–2026. With the Times archive in place the question is what weight to give 175 years: by article count the modern era dominates (the Guardian alone contributes about as many articles per day as the Times ever did), and by wall-clock time it is fifteen per cent of the record. The second edition takes the second view and weights each article by the inverse of the number of articles published on its day,

$$w_i = 1 / n(\tau(i)), \qquad \sum_{\tau(i) = \tau} w_i = 1 \text{ for every day } \tau,$$

so that a day in 1887 and a day in 2024 contribute equally to $\mu$ and $\Sigma$ (§The mathematics; the covariance formulae are unchanged, with $W = \mathrm{diag}(w_i)$). The LSA of the second edition is the SVD of $W^{1/2}(T - \mathbf{1}\mu^{\top})$ with the same weights, so both bases are balanced the same way. The total weight is the number of days with news, 63,270.

7.2 Cleaning the index register

Fitting on the Times exposed a register problem that the modern sources never had. The strongest terms of several rotated axes were titl · articl · page · front · editori · correct · repts: the vocabulary of the newspaper’s own index rather than of events. Three rules, applied only to Times documents before tokenising, remove it: placeholder headlines lose their boilerplate (“Article 8 – No Title” is dropped, “Letter to the Editor 3 – No Title” keeps its kind), the “LEAD:” marker is stripped, and a source-specific stop list drops the index’s annotation words (illus, por, repts, mfrs, page, correction, editorial and the like). The rule lives in one place, et-cli’s terms module, and is shared by the LSA fit, the eigen term loadings and the nightly projection; the embedding axes themselves are untouched by it, since embeddings are computed from the raw text. After the fix the sixty axes of the 175-year basis read as news: quarterly earnings; the presidency; deaths and funerals; Israel and Palestine; sports; the boroughs of New York; the courts; Congress and taxes; banking; corporate appointments; police and crime; war and the military; elections; the performing arts.

7.3 What 175 years changed

We measured the two eigenbases on the same 20,471 Wikipedia events of 2023–2026, whose human sections were never used in fitting.

The old news did not overwhelm the new. The 175-year basis explains today’s events exactly as well as the modern one and distributes them better: the first edition sends 58 per cent of labelled events to a single armed conflict axis, the second keeps its largest axis to 27 per cent and separates conflict, disasters, law and crime, and elections. The last three rows say how: the spans of the two bases nearly coincide (42 of 60 principal directions agree to within 26°), but the individual axes within that span have rotated, as the Davis–Kahan bound predicts when eigenvalue gaps (here of order $10^{-3}$) are smaller than the perturbation a new weighting introduces. Since the newspaper prints axes, not spans, the two editions are different newspapers of the same news, and the deployed paper carries both: the root edition is the first, /v2 the second, and every page has a masthead switch that shows both editions with the current one marked.

8 The newspaper

A day page (Figure 1) opens with the spectrum: for each basis, the axes ranked by their share of the day’s energy, each bar coloured when its $z$-score against the era statistics is beyond $\pm 2$, followed by a Silence line listing archetypes that are usually present and are not. On 24 February 2022 both bases put russian · russia · ukrain · putin at $z \approx +8$; the LSA also flags covid and nation · million · union · strike, the eigenbasis unit · injur · forc · target.

Then the stories, ranked by energy times $(1+\nu)$ so that big and unexplained rises. Each card carries the dominant archetype of each basis as a badge, the human section if a bulletin member had one, the counts of articles and sources, the episode day, $T^2$ and novelty, and then one panel per basis: the top four coordinates as bars; Same as always—the strongest axes of the archetype’s profile that this story also carries; Different this time—the axes on which the story departs from the profile by more than $1.5\sigma$, with direction and magnitude; and the five closest precedents from distinct episodes, each a link to its day. For the invasion story both panels honestly report nothing beyond 1.5σ of this archetype’s profile—it is a textbook instance of its archetype—while the precedents differ instructively: the LSA finds the same week’s build-up briefings and 26 February 2014 (Crimea, 0.97), the eigenbasis reaches for “Ukraine: what will happen now?” (2 March 2014) and “Ukrainian president: Russian troops have crossed border” (28 August 2014).

The day after 11 September 2001 (Figure 2) shows the delta panel doing its job. The spectrum lights attack · bomb · terrorist · terror at $+2.9$, european at $+2.6$ and market at $+2.5$; the lead story, 34 articles under the terrorism archetype, is different from that archetype’s profile in one direction—flight · airline · passenger · airport, $+3.2\sigma$ more than usual—and its precedents are “Airports put on alert for bombs” (December 1999), “US bases in Italy on terror alert” (January 2001) and “Bin Laden is suspect in rocket raids” (November 1999).

Below the stories, The Residual lists the day’s stories with the highest $\nu$: the ones the basis explains least, which is where the genuinely new hides.

An archetype page (Figure 3) shows the axis’s full term list, the number of stories it dominates, its energy by year as a bar per year—the Ukraine axis has spikes in 2014 and from 2022—the distribution of human sections among its stories (1,084 Armed conflicts and attacks, 529 International relations), and the canon: the purest instances over the years, one per episode, by coordinate.

The comparison page (Figure 4) is where the two algorithms meet: the number of axes, the number of articles with coordinates, the section purity of each basis, the matched-axis table, and every axis of each basis with its terms, story count and majority section. An all-issues page lists every day since 1 January 1999.

9 Evaluation

9.1 Recovery of human categories

Wikipedia’s bulletins carry a human section per event. The sections were never used to fit either basis. For each basis we define an axis’s majority section as the most common section among the labelled stories it dominates, and section purity as the share of labelled stories whose dominant axis’s majority section is their own. The LSA basis scores 42.4%, the embedding eigenbasis 40.0%. Both are far above chance (the largest section is under a quarter of the labelled stories), and both are limited by the same thing: the sections are eight coarse categories while the axes are sixty fine ones, so an Armed conflicts bulletin about Gaza and one about Ukraine sit on different axes that both have Armed conflicts as majority—counted as hits—while Politics and elections is spread across a dozen political axes whose majority sections differ.

9.2 Agreement between the bases

Matching the sixty LSA axes to the sixty embedding axes by Kuhn–Munkres on the absolute correlation of story coordinates (a deterministic 100,000-story sample) gives pairs whose correlations run from 0.75 down to about 0.5. The clearest pairs are the ones a reader would draw by hand:

The negative correlations in the full table are also informative: the LSA’s police · arrest · investigate · murder axis pairs at $-0.69$ with the embedding’s world · company · market · develop axis—the two bases agree that crime reporting and business reporting are opposite ends of one direction, and disagree only about which end to name.

9.3 What each basis sees that the other does not

The term basis cannot see register; the embedding basis cannot help seeing it. The embedding basis clusters events far better and gives a meaningful residual; the term basis produces crisper names and needs no model. On the evidence of §8.1 neither dominates, and the deployed paper prints both.

10 Engineering: computing in flat memory

The first edition was fitted on a laptop in an afternoon, and its code held what it needed in memory: every article’s embedding and coordinates for orienting the axes, and the whole document–term matrix, twice, for the term loadings. At 16.5 million articles that design was killed by the operating system in the third pass of et eigen, at about forty gigabytes. The fix was not a bigger machine but a change of shape: every quantity the fit needs is a sum over documents, and sums can be accumulated over partitions and thrown away.

Covariance already was. The weighted mean and covariance are functions of $Z = \sum w_i$, $m = \sum w_i x_i$ and $M = \sum w_i x_i x_i^{\top}$, accumulated per partition and merged.

Orientation used to sort every axis’s column of scores to decide its sign; it now accumulates, per axis $j$, the running power sums $n$, $S_1 = \sum c_{ij}$, $S_2 = \sum c_{ij}^2$, $S_3 = \sum c_{ij}^3$. The third central moment, which decides the sign, is

$$m_3 = S_3 / n - 3 \bar{c} \, S_2 / n + 2 \bar{c}^{\,3}, \qquad \bar{c} = S_1 / n,$$

and an axis is flipped when $m_3 < 0$, exactly the rule the batch code applied to the full column (a test checks the two agree on every axis, including the merge of two partial accumulators).

Term loadings are $\beta = (T - \mathbf{1}\mu^{\top})^{\top}{} S$ for the $N \times k$ score matrix $S$; expanded, $\beta = T^{\top} S - \mu \,(\mathbf{1}^{\top} S)$, and both terms decompose over row blocks $b$:

$$\beta = \sum_b T_b^{\top} S_b - \mu \sum_b \mathbf{1}^{\top} S_b, \qquad \mu = N^{-1} \sum_b T_b^{\top} \mathbf{1}.$$

Each partition therefore tokenises its own articles into a small sparse block $T_b$, projects its own embeddings into $S_b$, adds $T_b^{\top} S_b$ (a $50{,}000 \times 60$ dense matrix, 24 MB), the column sums of $T_b$ and the column sums of $S_b$ into an accumulator, and is dropped. Blocks may arrive in any order and from any thread; accumulators merge by addition. A property test checks that the block form equals the whole-matrix operator to $10^{-12}$. The same pass writes the partition’s coordinates and $T^2/Q/\nu$ rows, so the whole of et eigen is now three streaming passes—covariance, orientation, measurement—and runs on the 16.5-million-article corpus in ten minutes at 0.3 GB resident.

Stories clustered the archive after loading every vector; a day’s clustering depends only on that day and, for episode threading, on the previous day’s stories, so the pass now runs month by month, carrying yesterday’s stories across the boundary and holding one month of articles at a time.

Two things still do not stream. The randomized SVD of the term matrix needs $T$ in memory—455 million non-zeros for this corpus, a peak of 31 GB—which a 64 GB laptop affords; the block form of the mean and of $T^{\top} \Omega$ would remove that too and is the obvious next step. And the site’s in-memory index, which serves any day from memory and finds precedents across the whole archive, holds all 13.4 million stories of the second edition (about 20 GB). That last fact decides where things run: the GitHub runner (7 GB, 14 GB of disk) builds the root edition nightly and never pulls the second edition’s tables; the second edition is maintained from the laptop by a script that pulls the runner’s fresh articles and embeddings, measures them against the fitted bases without refitting, extends the stories, exports the recent issues and deploys them under /v2.

Two failure-mode decisions were made along the way. Partition reads that used to be swallowed into empty results now fail the fit with the partition’s name, and every partition’s articles are checked id-by-id against its embeddings before they are measured, so a stale or partial embedding can no longer silently shift a basis.

11 Implementation

The system is a Rust workspace of ten crates behind one binary, et:

The code follows a normative style guide inherited from the author’s QueryGraph work: algebraic data types for domain alternatives, a functional core behind an imperative shell, typed errors at library boundaries, tests in sibling files, and fmt/clippy -D warnings/test as gates. Two implementation notes are worth recording. The varimax rotation is done by Kaiser’s pairwise plane rotations rather than the popular SVD-polar fixed-point iteration, because the latter oscillates on symmetric configurations (it bounced between two mirror-image 30° states on a test case) while the pairwise method increases the criterion monotonically. And the sparse operator computes $X^{\top}Y$ by per-thread partial accumulation over row chunks, which is what makes randomized SVD on a 149-million-non-zero matrix a matter of seconds per product.

The site loads the whole index of the first edition in about three minutes—1.09 million article records, two 60-dimensional coordinate tables, 842,278 stories, the era statistics and profiles—and serves any day from memory; the second edition’s index (16.5 million records, 13,421,090 stories) loads in about one minute from local disk and holds 20 GB. For deployment it is exported as 10,227 static objects (about 2 GB) to a private S3 bucket behind CloudFront, with an AWS WAF rate-based rule (600 requests per IP per five minutes) as the edge rate limit and a Route 53 zone for the domain.

12 Limitations and next steps

The delta is quantitative only. The design calls for a per-archetype slot schema—belligerents, trigger, external powers, weapons class for a war; office holder, allegation, evidence, stage for a scandal—filled by a language model for today’s story and its precedents and diffed. That layer, and the location of the residual in the story’s own sentences (embedding sentences and taking those most aligned with $r$), are designed but not built; the deployed paper says how far a story is from its archetype’s profile, not yet what is different in words. Its honesty is a feature—the invasion of Ukraine reads as a textbook instance of its archetype, which is true of its coordinates—but the reader wants the actors.

Names are term lists. Without a language model the axes are named by their loadings; russian · russia · ukrain · putin is unambiguous, unit · injur · forc · target less so.

The basis is the basis of the sources read. Three Anglophone sources make “a bad turn in the Middle East” an archetype and “a bad turn in the Sahel” a residual. Per-source contributions to each axis should be tracked and printed.

The nightly loop measures but does not yet refit. Both editions are rebuilt every night—the root edition by a GitHub Actions runner, the second from a laptop—by refreshing the sources, embedding and measuring the new articles against the fitted bases, extending the stories and exporting the recent issues. The refinement mathematics is implemented and tested but not scheduled: the colophon reporting drift against its bound, the RSS and Google News adapters, and the Time Machine (any date against any basis version) are the next build.

Register axes. Whether the embedding basis’s opinion, ending and assertion axes are archetypes of news or artefacts of the embedding is unresolved; the comparison page is the instrument for deciding. The 175-year basis adds a variant of the question: obituaries, marriage announcements and letters to the editor are recurring newspaper content and form axes of their own, which a reader of today’s page may or may not want counted as news.

13 Conclusion

Eigen Times reads a quarter-century of news as a matrix and prints each day as coordinates. The archetypes it finds—by two unrelated constructions—are the ones a reader would name, and its statistics give “how is it different this time” a number before a sentence. The system is small: one binary, a laptop, an afternoon of backfill, five minutes for one basis and a few hours for the other. What it lacks is the language layer that turns a $3.2\sigma$ excess on flight · airline · passenger · airport into the sentence a newspaper would print. That is the next issue.

Acknowledgements

The system and this article were built in a single day’s pair-programming session with Claude (Fable 5) in Claude Code, from a one-paragraph brief to the deployed site. The Guardian Open Platform, the Wikimedia Foundation and the Wikinews community made the corpus possible.

References

1. Allan, J., Carbonell, J., Doddington, G., Yamron, J., & Yang, Y. (1998). Topic Detection and Tracking pilot study: Final report. Proceedings of the DARPA Broadcast News Transcription and Understanding Workshop.
2. Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022.
3. Blei, D. M., & Lafferty, J. D. (2006). Dynamic topic models. Proceedings of the 23rd International Conference on Machine Learning, 113–120.
4. Carlson, M. (2015). The robotic reporter: Automated journalism and the redefinition of labor, compositional forms, and journalistic authority. Digital Journalism, 3(3), 416–431.
5. Cutler, A., & Breiman, L. (1994). Archetypal analysis. Technometrics, 36(4), 338–347.
6. Das, A. S., Datar, M., Garg, A., & Rajaram, S. (2007). Google News personalization: Scalable online collaborative filtering. Proceedings of the 16th International Conference on World Wide Web, 271–280.
7. Davis, C., & Kahan, W. M. (1970). The rotation of eigenvectors by a perturbation. III. SIAM Journal on Numerical Analysis, 7(1), 1–46.
8. Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K., & Harshman, R. (1990). Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6), 391–407.
9. Gholipour Ghalandari, D., Hokamp, C., Pham, N. T., Glover, J., & Ifrim, G. (2020). A large-scale multi-document summarization dataset from the Wikipedia Current Events Portal. Proceedings of ACL 2020, 1302–1308.
10. Graefe, A. (2016). Guide to Automated Journalism. Tow Center for Digital Journalism, Columbia University.
11. Grootendorst, M. (2022). BERTopic: Neural topic modeling with a class-based TF-IDF procedure. arXiv:2203.05794.
12. Halko, N., Martinsson, P.-G., & Tropp, J. A. (2011). Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM Review, 53(2), 217–288.
13. Hofmann, T. (1999). Probabilistic latent semantic indexing. Proceedings of the 22nd ACM SIGIR Conference, 50–57.
14. Hotelling, H. (1931). The generalization of Student’s ratio. Annals of Mathematical Statistics, 2(3), 360–378.
15. Jackson, J. E., & Mudholkar, G. S. (1979). Control procedures for residuals associated with principal component analysis. Technometrics, 21(3), 341–349.
16. Kaiser, H. F. (1958). The varimax criterion for analytic rotation in factor analysis. Psychometrika, 23(3), 187–200.
17. Kuhn, H. W. (1955). The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1–2), 83–97.
18. Leetaru, K., & Schrodt, P. A. (2013). GDELT: Global data on events, location, and tone, 1979–2012. ISA Annual Convention.
19. Leskovec, J., Backstrom, L., & Kleinberg, J. (2009). Meme-tracking and the dynamics of the news cycle. Proceedings of KDD 2009, 497–506.
20. Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems, 26.
21. Newman, D., Chemudugunta, C., Smyth, P., & Steyvers, M. (2006). Analyzing entities and topics in news articles using statistical topic models. Intelligence and Security Informatics (ISI 2006), 93–104.
22. Petrović, S., Osborne, M., & Lavrenko, V. (2010). Streaming first story detection with application to Twitter. Proceedings of NAACL-HLT 2010, 181–189.
23. Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. Proceedings of EMNLP-IJCNLP 2019, 3982–3992.
24. Xiao, S., Liu, Z., Zhang, P., & Muennighoff, N. (2023). C-Pack: Packaged resources to advance general Chinese embedding. arXiv:2309.07597.