9 The statistical thresholds

9.1 ±2σ on the spectrum

The symbol $\sigma$ in this heading means a reference standard deviation. For each named axis, collect one spectrum value per loaded day. Subtract their mean to describe departures from the usual level, then divide by their positive standard deviation to put those departures on a common scale. The unit is now “reference standard deviations”, rather than the original coordinate unit. An axis is loud above $+2$ and silent below $-2$.

9.1.1 Calculate a reference spread before a z-score

Use the synthetic daily values $-1,0,1,2,3$. Their mean is $1$. Their deviations are $-2,-1,0,1,2$, whose squares add to $4+1+0+1+4=10$. The notebook uses the sample variance: divide by the count minus one, giving $10/(5-1)=2.5$. Its standard deviation is $\sqrt{2.5}\approx1.581$. This count-minus-one convention differs from the total-weight normalization of the fitted covariance earlier; name the convention instead of silently switching denominators.

Against this reference, a value $-3$ first gives deviation $-3-1=-4$, then z-score $-4/\sqrt{2.5}\approx-2.530$. It belongs to Silence. Neither $-3$ nor $-4$ alone could be compared directly with the standardized cutoff $-2$; dividing by the reference spread is essential.

To obtain the same summaries one value at a time, let $n$ be the count after a new scalar value $a$ arrives. Let $h_{\mathrm{old}}$ denote the previous running mean and $A_{\mathrm{old}}$ the previous sum of squared deviations from that mean. Define $\delta=a-h_{\mathrm{old}}$, the new value’s departure from the old mean. The updated mean $h_{\mathrm{new}}$ moves by that departure divided among all $n$ values:

$$h_{\mathrm{new}}=h_{\mathrm{old}}+\delta/n.$$

The updated squared-deviation sum is

$$A_{\mathrm{new}}=A_{\mathrm{old}}+\delta(a-h_{\mathrm{new}}).$$

These are Welford’s updates. Initialize the first mean at the first observation and its deviation sum at zero. Each later update uses both the old and new mean; this compensates for the movement of the reference point. With at least two values, divide the final sum by $n-1$ for the sample variance. With only one value, that sample variance is undefined. The notebooks verify these updates against the direct five-value calculation.

9.1.2 A standardized cutoff is not automatically a probability

A probability model specifies how frequently values are expected to fall in different ranges. A Gaussian, or normal bell-shaped distribution, is one such model. Under that model, about 4.55% of observations lie outside two standard deviations, often rounded to “one in twenty”. News series can be heavy-tailed, meaning that large departures occur more often than this model predicts. Standardization alone does not make them Gaussian.

There is also a many-comparisons issue. If 60 axes each really had the Gaussian 4.55% tail frequency, the expected number flagged would be about $60\times0.0455=2.73$. An expected number is a long-run average count, not the probability that any particular day has a flag. This average-count calculation does not require independent axes; independence would be an additional assumption when calculating the probability of at least one flag. The notebooks compute the Gaussian benchmark, not a calibrated newspaper alarm probability.

The empirical display intent is that a typical day lights one to three axes, not ten. On 24 February 2022 the Ukraine axis was at $+8$; on 12 September 2001 terrorism was at $+2.9$, Europe at $+2.6$, markets at $+2.5$. These recorded numbers are spectrum z-scores against the loaded reference, not probabilities.

9.2 1.5σ for “different this time”

Now change the reference set. We are comparing one story with the stories usually assigned to its archetype, rather than comparing a day with other days. On each axis, subtract the archetype profile mean from the story coordinate and divide by the positive profile standard deviation. Keep departures whose absolute values exceed 1.5, sort them by that magnitude, and display at most three.

For example, the notebook has profile mean $2$, profile standard deviation $0.5$, and story value $3.6$ on one synthetic axis. The departure is $(3.6-2)/0.5=3.2$ profile standard deviations. Another has mean $0$, standard deviation $2$, and value $-4$, giving $-2$. Both exceed 1.5 in magnitude, but their signs say whether the story lies above or below its profile. A story exactly at every profile mean produces zero departures and an empty panel.

The threshold is lower than the spectrum’s because the display asks a different question: whether this instance differs from its class. A specific secondary signal, such as airline · airport on a terrorism story at $+3.2\sigma$, can be useful to inspect. The recorded invasion example has nothing beyond 1.5σ of this archetype’s profile; an empty panel is a possible result, not an error.

This display rule is not a formal proof that a difference is statistically significant. Under a Gaussian reference, a two-sided 1.5-standard-deviation cutoff is crossed about 13.36% of the time. Across 59 such secondary axes, the expected count would be about $7.88$. Correlation between axes changes how flags arrive together; it does not by itself invalidate this expectation calculation if the individual Gaussian tail assumptions held. Here those marginal assumptions are not established either. The empirical panel depends on the observed profile distributions, their correlations, and the three-item display cap.

9.3 Energy × (1 + ν) for ranking

Let $\nu_s$ be story $s$’s mean member-article novelty ratio, distinguishing it from a ratio calculated on the story centroid. Recall its attention weight $\varepsilon_s$. The ranking score is

$$\varepsilon_s(1+\nu_s).$$

The first factor rewards the recorded source and article attention. The second raises that score when more of the member articles’ centred variation lies outside the retained model. Since $0\le\nu_s\le1$, the multiplier runs from one to two: an equally energetic story at novelty one gets twice the score of one at novelty zero. This is an explicit ranking design, not a learned probability of importance. The residual section orders by $\nu_s$ alone. A large residual remains a property of this representation, not independent evidence of newsworthiness.

9.4 0.8 for matching axes across nights

A nightly refit supplies new eigenvectors. Their order can change, and multiplying an eigenvector by $-1$ leaves the same geometric line. Comparing column numbers or signed cosines alone would confuse these bookkeeping changes with new patterns.

An assignment first pairs new and old eigenvectors to maximize total absolute cosine, written $|\cos|$ and explained in the next chapter. Absolute value removes the sign ambiguity: cosine $-1$ and cosine $+1$ both identify the same line. After pairing, signs can be oriented to agree with the old directions.

A matched cosine below 0.8 is rejected: the component is treated as a new archetype, while retirement uses the separate noise-edge rule. The notebook reverses one axis’s sign during a small rotation and verifies that this does not turn a strong match into a rejection. Reported well-separated axes match at 0.99+ between nights. The 0.8 value is an operational continuity rule, not proof of semantic identity; poorly separated directions can move substantially while their common subspace stays similar.