# Review: cyclic ridge distillation with three learners ## 1. Precise restatement **Setup.** - There are three inputs x₁, x₂, x₃ and three models fᵢ(x) = aᵢφᵢ(x). Feature values are φⱼ(xᵢ) = Mᵢⱼ, where M = [[1,2,0],[0,1,2],[2,0,1]]. - Learner i fits only at xᵢ, with teacher i+1 (indices mod 3). - Each synchronous round, learner i sets aᵢ to the exact minimizer of (a − f_{i+1}(xᵢ))² + λa², with 0 < λ < 1. - The target function is identically zero. - Error is the mean squared prediction error of the ensemble predictions under the uniform distribution on {x₁, x₂, x₃}. **Claims.** - (A) Starting from an arbitrarily small nonzero coefficient vector, the error diverges. - (B) For each pair of learners, mutual distillation drives both coefficients to exactly 0 within two rounds. **Hidden assumptions.** - The ridge penalty is applied to a, and the fit term is written as a rather than aφᵢ(xᵢ). This is legitimate only because the diagonal φᵢ(xᵢ) = 1. - λ < 1 is essential. - "Eliminates all errors" in the pairs means only "collapses to the zero function". That equals zero error only because the target is zero. - In the pair subsystems, one learner's teacher is changed. The objectives are therefore not literally "exactly the same" as in the cycle. ## 2. Checking the steps **Ridge update.** Setting the derivative to zero gives 2(a − t) + 2λa = 0, so a = t/(1+λ). **Teacher signal.** t = a_{i+1}φ_{i+1}(xᵢ). The relevant entries are φ₂(x₁) = 2, φ₃(x₂) = 2, φ₁(x₃) = 2. Hence aᵢ ← c·a_{i+1}, with c = 2/(1+λ) > 1 when λ < 1. ✓ **Three-learner dynamics.** The update is a(t+1) = cPa(t), where P is a cyclic permutation matrix. Because P preserves the norm, ‖a(t)‖ = cᵗ‖a(0)‖. So divergence holds for any nonzero start, not just equal positive ε. **Predictions diverge too.** Predictions are Ma. M is circulant with eigenvalues 1 + 2ωᵏ: - 3 (for k = 0), - 1 + 2ω and 1 + 2ω², each of modulus √3. So M is invertible, the predictions grow like cᵗ, and the MSE grows like c^{2t}. ✓ For the equal start ε, the predictions are 3εcᵗ at each input. **Pairs.** In pair {i, i+1}: - Learner i is taught by i+1 with factor c. - Learner i+1 is taught by i using φᵢ(x_{i+1}), which is 0 in all three cases: φ₁(x₂), φ₂(x₃), φ₃(x₁). The update matrix is [[0,c],[0,0]], which is nilpotent with square zero. - After round 1: a_{i+1} = 0. - After round 2: aᵢ = 0. This holds for all three pairs. ✓ ## 3. Attempts to break it - **λ ≥ 1.** Then c ≤ 1 and there is no blow-up. The claim says "can", so an existential λ suffices. - **Nonzero target.** The pairs collapse to the zero predictor. They do not "eliminate error"; the residual error equals the target's norm. This is a phrasing caveat. - **Alternative pair tests.** One could keep only the cyclic edge, with the other learner frozen or untaught. Those subsystems are also bounded, so the conclusion survives. - **"Mechanism requires different training inputs."** This necessity is asserted in the text but not proved. The one-sentence claim does not rely on it. No counterexample to the stated claim was found. ## 4. Novelty The result reduces to a classical linear-systems fact. Stability, or even nilpotency, of every pairwise subsystem does not imply stability of the full coupled system. The whole system here is a scaled cyclic permutation whose loop gain is c³ > 1, while each 2×2 restriction is strictly triangular. This is a standard "cycle gain" / feedback-loop phenomenon. It is close in spirit to divergence examples for bootstrapped learning with function approximation, such as Baird-type counterexamples. It is a clean toy instance in a distillation framing, but not a new phenomenon. ## 5. Fairness The construction is correct but highly special: - It uses hand-chosen features with zeros exactly where needed. - It uses a single training point per learner. - It requires λ < 1. - It assumes a zero target. - In the pair tests, a learner's teacher is changed, so the setups differ from the cycle. "Eliminates all errors" really means "collapses to zero". The claim is accurate as an existence statement, but it overstates generality somewhat by not mentioning these restrictions. ```json {"verdict": "correct with caveats", "novelty": "special case or variant of known results", "fair_summary": "partly", "confidence": 4, "one_line": "The arithmetic checks out (cycle gain (2/(1+λ))³>1, each pair's update nilpotent), but it is a hand-built instance of the known fact that pairwise-stable subsystems need not make a stable cyclic system, and 'eliminates errors' relies on the zero target."} ```