## 1. Precise statement and hidden assumptions The construction is mathematically valid under the following interpretation: - There are three fixed inputs, with feature matrix \[ \Phi= \begin{pmatrix} 1&2&0\\ 0&1&2\\ 2&0&1 \end{pmatrix}, \qquad \Phi_{ji}=\phi_i(x_j). \] - Learner \(i\) has one unrestricted real coefficient: \[ f_i(x)=a_i\phi_i(x). \] - Learner \(i\) trains only at \(x_i\), where its own feature value is \(1\). - Updates are **synchronous**: each learner uses its teacher’s coefficient from the preceding round. - “Exact distillation” means **exact minimization of the ridge objective**, not exact matching of the teacher’s prediction. The teacher’s prediction is held fixed during that minimization. - The regularization parameter is fixed and satisfies \(0<\lambda<1\). - The reference target function is identically zero. It is used to measure error, not supplied as a training target. - A two-learner subsystem retains each learner’s original training input and feature function, but makes the two surviving learners teach each other. For definiteness, define the average prediction MSE by averaging over both learners and the uniform input distribution: \[ E(a)=\frac13\sum_{i=1}^3\left[\frac13\sum_{j=1}^3 f_i(x_j)^2\right]. \] The precise conclusion is: > For every initial-error tolerance \(\delta>0\), there is a nonzero initialization with \(E(a^{(0)})<\delta\) whose three-learner trajectory satisfies \(E(a^{(t)})\to\infty\). Nevertheless, every two-learner subsystem sends every initial coefficient vector to zero after at most two synchronous rounds. The pairwise statement concerns error relative to the **zero target**, not arbitrary ground truth. ## 2. Checking the argument and independently deriving the result ### Checking the update The feature table gives \[ f_{i+1}(x_i)=2a_{i+1}. \] Because \(\phi_i(x_i)=1\), learner \(i\)’s objective is indeed \[ L_i(a)=(a-2a_{i+1}^{(t)})^2+\lambda a^2. \] Differentiating, \[ L_i'(a)=2(a-2a_{i+1}^{(t)})+2\lambda a. \] Thus the unique minimizer is \[ a_i^{(t+1)}=\frac{2}{1+\lambda}a_{i+1}^{(t)}. \] Uniqueness follows from \(L_i''(a)=2(1+\lambda)>0\). Set \[ r=\frac{2}{1+\lambda}. \] The stipulated range \(0<\lambda<1\) gives \(r>1\). Starting from \(a^{(0)}=\varepsilon(1,1,1)^\top\), \[ a^{(t)}=r^t\varepsilon(1,1,1)^\top. \] So the coefficient-growth assertion is correct. ### Independent linear-algebra derivation The full update matrix is \[ A=rP,\qquad P= \begin{pmatrix} 0&1&0\\ 0&0&1\\ 1&0&0 \end{pmatrix}. \] Since \(P\) is a permutation matrix, \[ P^3=I,\qquad \|Pa\|_2=\|a\|_2. \] Consequently, \[ A^3=r^3I,\qquad \|a^{(t)}\|_2=r^t\|a^{(0)}\|_2. \] Each feature column has squared norm \(5\), so \[ E(a)=\frac{5}{9}\sum_i a_i^2 =\frac59\|a\|_2^2. \] Therefore \[ E(a^{(t)})=r^{2t}E(a^{(0)}). \] This proves something slightly stronger than the text: **the average MSE diverges for every nonzero initialization**, not just equal positive coefficients. For equal initialization, \[ E(a^{(0)})=\frac53\varepsilon^2. \] Choosing \[ 0<\varepsilon<\sqrt{\frac{3\delta}{5}} \] makes the initial MSE less than any prescribed \(\delta>0\), while its subsequent value diverges. Notice that coefficients grow by \(r\) per round; **MSE grows by \(r^2\)**. ### Checking all pairs For learners \(1,2\), mutual teaching gives \[ \begin{pmatrix}a_1'\\a_2'\end{pmatrix} = \underbrace{\begin{pmatrix}0&r\\0&0\end{pmatrix}}_{B_{12}} \begin{pmatrix}a_1\\a_2\end{pmatrix}. \] The reverse direction vanishes because \(\phi_1(x_2)=0\). The pair \(2,3\) has the same matrix form. For the ordering \(1,3\), \[ B_{13}= \begin{pmatrix}0&0\\r&0\end{pmatrix}. \] Every one of these matrices satisfies \[ B_{ij}^2=0. \] Thus every pair reaches zero coefficients—and hence zero predictions on all three inputs—after at most two rounds, for every starting state. All substantive algebraic steps in the supporting argument check out. ## 3. Attempts to break it: examples, boundaries, and restrictions ### A concrete numerical example Take \(\lambda=\tfrac12\), so \(r=\tfrac43\), and initialize every coefficient at \(10^{-3}\). The full system has \[ a_i^{(t)}=10^{-3}(4/3)^t, \qquad E_t=\frac53\,10^{-6}(16/9)^t. \] The first two MSE values are approximately \[ E_0=1.667\times10^{-6},\qquad E_1=2.963\times10^{-6}, \] and the formula proves unbounded growth. For pair \(1,2\), \[ (0.001,0.001) \longmapsto (0.001333\ldots,0) \longmapsto (0,0). \] ### Important boundaries - **Exactly zero initialization:** it remains zero. “Arbitrarily small” must mean arbitrarily small **nonzero** error. - **\(\lambda=1\):** \(r=1\), so the full-system MSE remains constant. - **\(\lambda>1\):** \(r<1\), so the full system converges to zero. - **Nonzero ground truth:** pairs still collapse to zero, but that generally does not eliminate prediction error. The zero-target assumption is essential to the wording “eliminates all errors.” - **Joint optimization:** this is not a joint minimization over all teacher and student coefficients simultaneously. Teacher predictions must be frozen during each learner’s update. - **Bounded-output modifications:** clipping, projection, or other constraints could prevent unbounded growth. They are absent here. ### Exact pairwise extinction is fragile, but pairwise stability is not The two-round result depends on exact feature zeros. Replace the three off-cycle zeros by a small positive \(\eta\), leaving the other feature values unchanged. The full cyclic system still has multiplier \(r\). Each pair now has forward and reverse gains \[ r=\frac{2}{1+\lambda}, \qquad s=\frac{\eta}{1+\lambda}, \] so its two-step matrix is \[ B^2=rsI=\frac{2\eta}{(1+\lambda)^2}I. \] For \[ 0<\eta<\frac{(1+\lambda)^2}{2}, \] every pair still converges to zero, while the three-way system still diverges. Thus: - **Finite-time pairwise extinction** relies on exact zeros. - **Pairwise convergence versus three-way divergence** persists under small feature perturbations. ### Does this mechanism require different training inputs? Within the stated scalar-ridge setting, a common single training input would prevent this instability. If all learners trained at the same input \(x\), write \(c_i=\phi_i(x)\). Ordinary ridge fitting would give \[ a_i'=\frac{c_i c_{i+1}}{c_i^2+\lambda}a_{i+1}. \] The product of the three cycle gains would be \[ \prod_{i=1}^3\frac{c_i^2}{c_i^2+\lambda}<1. \] Hence the common-input cycle would be stable. This supports the text’s explanation **for this class of scalar, single-input ridge updates**. It is not a general theorem that every possible distillation instability requires different training inputs. ## 4. Novelty The underlying mathematics is a straightforward instance of established linear-systems facts: 1. **Discrete-time linear stability:** \(a^{(t+1)}=Aa^{(t)}\) is asymptotically stable exactly when the spectral radius satisfies \(\rho(A)<1\). 2. **Weighted cyclic permutation matrices:** a directed three-cycle with edge gains \(g_1,g_2,g_3\) satisfies \[ A^3=(g_1g_2g_3)I, \qquad \rho(A)=|g_1g_2g_3|^{1/3}. \] 3. **Nilpotent restrictions of a directed cycle:** deleting any vertex breaks the cycle, leaving an acyclic graph whose adjacency matrix is nilpotent. Here the full matrix has spectral radius \(r>1\), while every two-dimensional principal submatrix has spectral radius zero. So this is a clean **ridge-distillation realization of a familiar directed-cycle feedback phenomenon**, rather than a new mathematical instability principle. I cannot establish whether this exact feature-table presentation has appeared before, but its mechanism is a special case of known results. ## 5. Fairness of the one-sentence claim The claim is **fair as an existential statement about the supplied construction**. It does not say that arbitrary triples behave this way. The qualifications worth making explicit are: - “Exact” refers to exact optimization of a **regularized** objective, not exact teacher matching. - Error is measured against an identically zero target. - The pairwise result concerns the specifically defined mutual-teaching interventions. - The dynamics use frozen previous-round teachers and synchronous updates. The construction rigorously shows that testing every such pair need not reveal a three-way feedback instability. It does not by itself establish that this is common in practical distillation systems. {"verdict":"correct with caveats","novelty":"special case or variant of known results","fair_summary":"yes","confidence":5,"one_line":"The construction is valid under synchronous exact ridge minimization with zero ground truth, and realizes a classical amplifying directed cycle whose every two-learner restriction is nilpotent."}