Research · distributed optimization
The subspace was right. The geometry inside it wasn’t. A QR step turns correlated unit columns into the orthonormal frame the spectral update intended.
Spectral optimizers transform matrix-shaped gradients, but communicating a full matrix across sharded workers is expensive. Dion makes that practical by tracking a rank-r subspace and sending two skinny factors instead.
The communication order falls from matrix scale to O((m+n)r). That compression is not the problem Orth-Dion diagnoses.
dense update → U · Vᵀ
Dion’s power iteration can identify the right low-rank span. Its final column normalization then rescales each right-factor column to length one—but leaves the angles between columns untouched.
The basis vectors remain correlated, visible as nonzero off-diagonal entries in their Gram matrix. The update is therefore mis-scaled inside the correct subspace.
‖vᵢ‖₂ = 1 does not imply VᵀV = I
Orth-Dion replaces ColNorm(W) with orth(W) through QR. QR changes the coordinates inside the tracked subspace without changing which subspace was selected.
The resulting partial isometry has dual-norm factor ν = 1. Dion’s correlated basis can have ν > 1, bounded in the analysis by √r, which is where the rank-dependent geometric penalty enters.
same span · V̄ orthonormal · ν = 1
The wire format does not grow: Orth-Dion retains Dion’s low-rank communication order. QR adds local O(nr²) work, small relative to the leading matrix multiplication when r is low—but not literally free.
On Llama 3 320M/C4, matched-rank Orth-Dion reached lower validation loss at every tested rank and reached Dion’s best loss in 12.3–13.1% fewer steps. Adaptive rank is the paper’s route for offsetting QR overhead in wider models.
correct geometry + same communication order
The complete mechanism
Power iteration finds a low-rank subspace that is shared by Dion and Orth-Dion.
Column normalization fixes lengths separately but does not remove cross-column correlation.
QR orthogonalization preserves the span and makes the update’s dual-norm factor exactly one.
Low-rank communication remains. Adaptive rank can trade a small amount of compression for the local QR work.
Scope. The convergence theorem assumes deterministic gradients, smoothness, a spectral gap, a small spectral tail, bounded gradients, and contraction conditions. Adaptive rank is heuristic. The 17.1B result is a short, wide-and-shallow wall-clock study—not a full 17.1B convergence run—and fixed-rank QR adds measurable step time.
Read the paper
Google Scholar currently shows separate arXiv and ICML’26 CoLoRAI workshop records for Orth-Dion. The workshop link above points to the venue record; the arXiv record remains the stable paper source.
