In June, researchers in New York City reached across 5,000 miles and ran live training experiments on BUZZ HPC's GPU cluster in Asunción, Paraguay.
The team, from Columbia University's Department of Industrial Engineering and Operations Research, has now submitted the resulting paper to NeurIPS, one of the three top conferences in machine learning.
The researchers study a newer class of optimizers, including Muon, that scale each update to match the shape of a layer's weight matrix instead of treating every parameter the same way. That approach already shows up in large-scale model training, but its behavior under noisy, real-world gradients was not well understood.
The paper proves a hard limit on how efficient these optimizers can be once that kind of noise is accounted for:
It then shows a method called Scion reaches that limit:
It also introduces a modified version, called a transported Scion method, that trains faster when the loss surface is smooth enough to support it:
Both methods were tested against standard optimizers across CNN and language model training runs, including runs on BUZZ HPC's A40 nodes in Paraguay.
Once hardware differences were normalized, A40 performance matched H100-class GPUs in Columbia's benchmarks.
Full paper: Scale-Invariant Neural Network Optimization: Norm Geometry and Heavy-Tailed Noise (Zhang & Lin, Columbia University)
