Skip to content

dew.newton_schulz

Muon’s Newton-Schulz orthogonalization (dew.optim’s Muon and M3 iterate through it): optax’s own iteration (orthogonalized_leaf) over each update’s matrices (as_matrices), a head’s alone where a run names its width (headed), the matrices of a shape stacked and spread whole across the mesh’s devices (orthogonalized), in the run’s numerics or a stated dot algorithm (ns_operands), at Jordan’s coefficients or Polar Express’s (polar_express).

NameSummary
NEWTON_SCHULZMuon’s quintic Newton-Schulz coefficients, Jordan’s (modded-nanogpt), optax’s default.
POLAR_EXPRESS_TABLEPolar Express’s quintic per iteration (arXiv 2505.16932, Appendix A’s coeffs_list), designed for singular values in [1e-3, 1], before its safety factor.
PreconditioningHow optax’s Newton-Schulz scales a matrix before it iterates (optax.contrib.muon’s preconditioning).
ns_operandsThe dtype Muon’s Newton-Schulz takes its operands in: an algorithm’s own, where one states the iteration’s products (BF16_BF16_F32’s bfloat16, TF32_TF32_F32’s and F32_F32_F32’s float32, which a platform that has not the algorithm refuses in JAX’s own words, and a complex update, whose products no real algorithm holds, refuses by name); else the run’s numerics’ (dew.nn.kernels.generation.numerics), each product at its precision (Numerics.matmul): bfloat16 in its fast forms, as Jordan’s Muon (modded-nanogpt) and Moonlight (arXiv 2502.16982) iterate; float32 in its accurate ones, TF32 products on a GPU at the default precision and IEEE float32 under the accurate preset (precision="accurate"); complex64 for a complex update; and float32 for a spectral preconditioning, whose norm is an SVD of float32 operands.
orthogonalizedEach update numbers makes a stack of matrices of (as_matrices) orthogonalized by optax’s Newton-Schulz (orthogonalized_leaf) in the run’s numerics or algorithm (ns_operands), as Moonlight distributes it (arXiv 2502.16982, 3): the matrices of one shape gathered into one stack, spread across the mesh’s devices whole, each device iterating its own share, and handed back as the updates lie.
orthogonalized_leafupdate orthogonalized by Muon’s steps quintic Newton-Schulz iterations at coeffs, one triple or a schedule of them, after preconditioning, optax’s own: its Muon transform with no momentum (scale_by_muon, beta 0 and no Nesterov step) hands back the update it is given orthogonalized, in its own dtype, bit for bit its orthogonalize_via_newton_schulz, which optax 0.2.8 does not export.
polar_expressPolar Express’s coefficients for steps iterations as its code applies them (arXiv 2505.16932, Appendix A): each but the last triple’s polynomial taken at x / safety, (a / s, b / s^3, c / s^5), the first’s again for its input’s own scaling, X / (safety * ||X||_F), which optax’s iteration, over X / ||X||_F, then reproduces; the table’s first steps, its last repeated past it.

attribute source

NEWTON_SCHULZ = (3.4445, -4.775, 2.0315)

Muon’s quintic Newton-Schulz coefficients, Jordan’s (modded-nanogpt), optax’s default.

attribute source

POLAR_EXPRESS_TABLE

Polar Express’s quintic per iteration (arXiv 2505.16932, Appendix A’s coeffs_list), designed for singular values in [1e-3, 1], before its safety factor.

attribute source

Preconditioning = Literal['frobenius', 'spectral', 'aol', 'schatten']

How optax’s Newton-Schulz scales a matrix before it iterates (optax.contrib.muon’s preconditioning).

function source

def ns_operands(
dtype,
preconditioning: Preconditioning = 'frobenius',
algorithm: jax.lax.DotAlgorithmPreset | None = None,
) -> jnp.dtype

The dtype Muon’s Newton-Schulz takes its operands in: an algorithm’s own, where one states the iteration’s products (BF16_BF16_F32’s bfloat16, TF32_TF32_F32’s and F32_F32_F32’s float32, which a platform that has not the algorithm refuses in JAX’s own words, and a complex update, whose products no real algorithm holds, refuses by name); else the run’s numerics’ (dew.nn.kernels.generation.numerics), each product at its precision (Numerics.matmul): bfloat16 in its fast forms, as Jordan’s Muon (modded-nanogpt) and Moonlight (arXiv 2502.16982) iterate; float32 in its accurate ones, TF32 products on a GPU at the default precision and IEEE float32 under the accurate preset (precision="accurate"); complex64 for a complex update; and float32 for a spectral preconditioning, whose norm is an SVD of float32 operands.

The fast forms’ bfloat16, where optax iterates in its momentum’s own float32: the quintic brings the singular values only near one, roughly within 0.7 to 1.2 (optax orthogonalize_via_newton_schulz), far coarser than bfloat16’s rounding. A Qwen3.5-4B decision fine-tune on one H100 (node job LR-h100-muon-2752c at 699d038ca, 8192 tokens a step) stepped in 844 ms with its iteration in bfloat16 (BF16_BF16_F32), 1287 ms at TF32 (TF32_TF32_F32) and 4641 ms at IEEE float32 (F32_F32_F32), against AdamW’s 495 ms, its loss at each of 30 steps within 0.07 of the float32 iteration’s (0.02 on average). Accumulated, its update’s microbatch takes 860 ms against 516 and the others about AdamW’s: an update of 65,536 tokens 9.4% over AdamW’s and one of 262,144 2.5% (dew.optim.muon; ledger 2685, 2752).

function source

def orthogonalized(
updates: optax.Updates,
numbers,
coeffs: jax.Array,
steps: int,
eps: float,
preconditioning: Preconditioning = 'frobenius',
algorithm: jax.lax.DotAlgorithmPreset | None = None,
widths=None,
) -> optax.Updates

Each update numbers makes a stack of matrices of (as_matrices) orthogonalized by optax’s Newton-Schulz (orthogonalized_leaf) in the run’s numerics or algorithm (ns_operands), as Moonlight distributes it (arXiv 2502.16982, 3): the matrices of one shape gathered into one stack, spread across the mesh’s devices whole, each device iterating its own share, and handed back as the updates lie. A matrix split across devices would exchange partial products at each of the iteration’s products; spread whole, a step moves each matrix twice, there and back. On a mesh with Explicit axes the spreading runs with them Auto (jax.sharding.auto_axes), where a constraint may name them and a split stack slices, each update leaving where it came; on one device, or none in context, each stack iterates whole. A tree of head widths (head_widths) splits an update’s output side into heads, each one matrix.

function source

def orthogonalized_leaf(
update: jax.Array,
steps: int,
dimension_numbers: optax.contrib.MuonDimensionNumbers | None,
*,
coeffs=NEWTON_SCHULZ,
eps: float = 1e-08,
preconditioning: Preconditioning = 'frobenius',
) -> jax.Array

update orthogonalized by Muon’s steps quintic Newton-Schulz iterations at coeffs, one triple or a schedule of them, after preconditioning, optax’s own: its Muon transform with no momentum (scale_by_muon, beta 0 and no Nesterov step) hands back the update it is given orthogonalized, in its own dtype, bit for bit its orthogonalize_via_newton_schulz, which optax 0.2.8 does not export. A leaf with no dimension_numbers is one matrix, its rows the reduction axis (a Linen kernel’s), its other axes a stack of such matrices.

function source

def polar_express(
steps: int,
safety: float = 1.01,
) -> tuple[tuple[float, float, float], ...]

Polar Express’s coefficients for steps iterations as its code applies them (arXiv 2505.16932, Appendix A): each but the last triple’s polynomial taken at x / safety, (a / s, b / s^3, c / s^5), the first’s again for its input’s own scaling, X / (safety * ||X||_F), which optax’s iteration, over X / ||X||_F, then reproduces; the table’s first steps, its last repeated past it.