dew.nn.precision
The precision a bf16 model’s fp32 boundaries multiply at.
A model’s last matmul before a loss (the patch output head, the vocabulary
head) keeps an fp32 result, since the loss is fp32; head_product says why
the vocabulary head still rounds to bf16 values at the default precision.
preferred_element_type=jnp.float32 alone makes that dot’s cotangent fp32,
so the mixed f32 x bf16 backward products run at the fp32 rate. XLA’s
BF16_BF16_F32 dot algorithm names the arithmetic instead (bf16 operands,
fp32 accumulation, whatever the storage), and jax carries precision into
the transpose rules, so the backward follows; jax’s own xla attention sets it
for this reason (jax-ml/jax#24047). Dew’s products at that algorithm type
their operands bf16 instead (bf16_dot_general, which the dot family runs
for it): an A100 computes the algorithm wrong where a dot has a vector side
or sits below XLA’s GEMM rewrite threshold.
Dew’s test processes run wide CPU arithmetic (WIDE_CPU_ARITHMETIC, which tests/lane_environment.py
sets): a CPU then sums every float32 product of the dot family, dot_general, einsum and matmul, in
float64 and rounds once (accumulated), since XLA:CPU chooses its fp32 dot accumulation by shape,
instruction set and release, and jax 0.12’s summed twice numpy’s error from float64 where 0.11’s
summed 0.6 of it. A library process’s products are the plain products on every platform.
Convolutions, depthwise ones included, keep XLA’s fp32 sum on a CPU: no convolution’s check 1
failed there, and XLA:CPU’s float64 convolution cost a 3x3 one’s training step 44 times (15.5 to
675 ms on a 4x32x32x128 map), its best rewrite 2 to 4. Should a convolution’s check 1 resolve above
its bar, the recorded fix is that rewrite: the kernel’s shifted windows taken inside the float64
scope and contracted with the taps as one 2D product (stride 1 train 3.9 times, stride 2 1.6).
| Name | Summary |
|---|---|
ACCURATE | Accurate, scientific work at its cost: float32 throughout, every product that states no precision in IEEE float32 and each op’s accurate form. |
DTYPE_PRESETS | The usual dtypes by Lightning’s names: float32 parameters computing in bf16 by default, float32 throughout, bf16 throughout, and float16 compute over float32 parameters. |
FP32_SUMS | JAX’s dot algorithm of fp32 operands summed in fp32: as a process’s or a scope’s default matmul precision, the statement that its fp32 products sum as an accelerator sums them, which accumulated honors on a CPU. |
L2NORM_EPS | The epsilon l2norm adds under its square root: FLA’s, and Dory’s justnorm’s. |
MATMUL_PRECISIONS | JAX’s matmul precision levels (jax_default_matmul_precision): “default” TF32 for a float32 product on a GPU and one bf16 pass on a TPU, “highest” IEEE float32. |
NUMERICS_PRESETS | The named run numerics beside the default (Numerics()). |
ROUNDING_STREAM | The stream stochastic rounding draws from: what a training step folds its key with (dew.objectives.base.training_rngs), and what a served row’s rounding key starts from (dew.sampling.text.rounding_keys), apart from dropout’s and sampling’s. |
WIDE_CPU_ARITHMETIC | The environment variable that, set to 1, has a process’s CPU programs take the arithmetic Dew’s tests hold a CPU to: float32 products summed in float64 and rounded once (accumulated, spread), a loop’s running sum of them compensated so it rounds once too (compensates_loops), and a float32 reduction taken in float64 and rounded once (widened_reduction). |
Dtypes | A run’s dtypes, three as Flax casts them: the parameters’ (its layers’ param_dtype, and the optimizer’s moments’), the compute’s (their dtype) and the outputs’ (the logits and losses a model returns). |
Numerics | A run’s numerics, set once and recorded with it, which its model, optimizer and trainer read: its dtypes (Dtypes, which a model takes as Flax’s param_dtype and dtype and its optimizer as its moments’; DTYPE_PRESETS names the usual ones by Lightning’s names), the precision every product that states none multiplies at (JAX’s own jax_default_matmul_precision level, or a jax.lax.DotAlgorithmPreset, which a record holds by its name), and each op’s form, its kernels’ fast or accurate arithmetic (dew.nn.kernels.generation.run_precision). |
accumulated | product(*operands, preferred_element_type=...), a sum of products of operands (a dot, an einsum, a matmul): the plain product, but in a process running wide CPU arithmetic (WIDE_CPU_ARITHMETIC, Dew’s tests), where they multiply in float32 (products_dtype) a CPU sums the products in float64 and rounds the sum once to the dtype the product returns. |
activation_and_slope | An fp32 gate’s act(gate) and its derivative, for ‘silu’ or ‘sigmoid’. |
added_to | running (running_sum) with term added: through compensated_add where it is a pair. |
asks_default_precision | Whether precision asks for no more than the default. |
at_default_precision | fn traced, forward and backward, under the default matmul precision. |
at_least_fp32 | The dtype arithmetic Dew keeps from rounding below float32 runs in: float32 for float32, every narrower float and a module’s unset dtype (None), and dtype itself where it is wider. |
bf16_dot_general | jax.lax.dot_general(lhs, rhs, dimension_numbers) multiplied as XLA’s BF16_BF16_F32 dot algorithm states the arithmetic: the operands rounded to bf16 (to nearest even), each product exact and summed in fp32 to an fp32 result. |
bf16_einsum | jnp.einsum(subscripts, *operands) in fp32, its every product bf16_dot_general’s: einsum plans the contraction (its path, broadcasts, and each operand’s own sums, diagonals and transposes, which form no product and run in fp32) and hands each pairwise product to bf16_dot_general as its _dot_general. |
bf16_logits | fp32 logits rounded to bf16 and held in fp32, as rounded_to rounds them, written as bf16 under CUDA. |
bf16_product | jnp.einsum(subscripts, hidden, head) multiplied as bf16 and its result rounded to bf16, held in fp32: the one product a bf16 head computes (head_product) and a tiled head computes over each tile, so the two round each logit alike. |
biased | product plus bias broadcast over its leading axes, in the dtype their addition promotes to, as flax’s layer adds them. |
capped_logits | Softcapped logits over a sampling temperature, which an engine applies to what the model’s head returns. |
compensated_add | Add two double-float values, (hi, lo) pairs whose sum is the value: hi is the sum rounded and lo what the rounding dropped (the his’ TwoSum, the los added to its remainder, then the pair’s own TwoSum), so a running sum keeps fp32’s precision squared and its hi is the running value rounded once, where a pair left unrenormalised keeps the plain running sum in hi, every addition’s rounding in it. |
compensated_difference | later - earlier of two double-float values, rounded once: the halves’ differences summed, so the difference of two near values (a range of a running sum) keeps its own precision, where the his’ difference alone carries their whole magnitude’s rounding: a one-token range of a decay’s running sum near -78 off by 4e-6. |
compensated_sum | The sum of values over axes, rounded once: one XLA reduction whose accumulator is a double-float (hi, lo) pair combined by compensated_add, so it partitions as any reduction does, and what the roundings dropped is added back at the end. |
compensated_total | A compensated sum’s value, its (hi, lo) pair rounded once, or the plain sum of the same values where that reaches an infinity: +-inf or NaN as jnp.sum’s is, where every residual past the infinity is inf - inf. |
compensates_loops | Whether a running sum of float32 products carried across a loop keeps what each addition drops (compensated_add), so the loop’s sum rounds once as one product’s does: where a CPU sums each product in float64 (accumulated), in a process running Dew’s wide CPU arithmetic (WIDE_CPU_ARITHMETIC). |
configurable_precision | Whether jax.default_matmul_precision can name precision: one precision or dot algorithm preset, unset or default included; not a pair of precisions or a dot algorithm. |
configured_precision | A context in which a product that takes no precision of its own (jax.nn’s xla attention) multiplies at precision, through jax.default_matmul_precision; an unset or default precision changes nothing. |
correctly_rounded_divide | The correctly rounded fp32 quotient, IEEE division’s, on XLA’s approximate dividers too: XLA:CPU rewrites a division by a broadcast into a multiplication by its reciprocal (openxla/xla#51148), and XLA:GPU’s fp32 a / b is off IEEE on 30 to 50% of quotients (DIVIDE-a100, DIVIDE-h100 receipts), where a quantizer’s scales and a diffusion schedule’s states must round as the reference’s host or CUDA division does (compressed-tensors’ NVFP4, the FP8 block cast, PyTorch’s schedulers). |
divided | numerator / rate as a source written in PyTorch divides a state by a rate, or an array by a constant: IEEE division, whatever the backend’s own division does (XLA takes a division by a constant as a product with its rounded reciprocal, on a CPU and an A100 alike, row 1402). |
exact_float32 | The precision a float32 product whose rounding decides what follows takes, a numerically critical one, a vocabulary head’s (the loss’s logits), a router’s or a recurrence’s state’s, each stating its reason where it calls this: the one the call states, else IEEE float32 (Precision.HIGHEST), stated over JAX’s configured precision, whose default runs TF32 on a GPU and one bf16 pass on a TPU, where the references multiply IEEE float32 (torch’s float32 matmul). |
float32_dot_general | jax.lax.dot_general at precision, accumulated as Dew accumulates a CPU’s float32 products (accumulated): the product a flax layer forms (nn.Dense(dot_general=...)). |
float32_einsum | jnp.einsum at precision, JAX’s configured one where unstated, as float32_dot_general is a layer’s, accumulated alike, and bf16_einsum’s product at the BF16_BF16_F32 algorithm. |
float32_matmul | lhs @ rhs at precision, JAX’s configured one where unstated, accumulated as float32_dot_general is. |
float64_sum | product(*operands), a sum of products, taken in float64 from its operands widened exactly and rounded once to dtype, under jax.enable_x64 for its own trace: the sum accumulated takes on a CPU under Dew’s wide arithmetic (WIDE_CPU_ARITHMETIC), with no platform of its own to choose. |
fp32_result_dot_general | A flax layer’s dot_general for a head whose result stays at least fp32. |
head_dot_general | A flax layer’s dot_general for a vocabulary head: a result at least fp32 whose operands are the compute dtype’s values, in both directions. |
head_product | jnp.einsum(subscripts, hidden, head) as a vocabulary head computes it, with accumulation and a result at least fp32. |
l2norm | x over its L2 length along the last axis, x * rsqrt(sum(x**2) + eps) with eps added under the root, computed in at least fp32 (at_least_fp32, a float64 input’s in float64) and rounded once to x’s dtype: the normalization of FLA’s l2norm (the linear attentions’ queries and keys, modeling_qwen3_next.py:368-371) and of nGPT’s justnorm (its residual stream and weights). |
numerics_of | A run’s numerics from what a flag, a record or a load names: a Numerics as it is, a preset’s name (NUMERICS_PRESETS, “default” for Numerics()), or the usual dtypes’ Lightning name (DTYPE_PRESETS) at the default precision and forms. |
operand_precision | The precision a product of dtype operands takes: the one stated, JAX’s configured one where none is, and for bf16 operands at the default precision the algorithm that keeps a bf16 dot bf16 in both directions. |
paired_sum | An empty running sum of shape that is a double-float pair on every backend, the plain sum beside it (added_to, running_total): for partials of a product Dew splits where its reference takes the product whole, so the sum’s error is the partials’ own, whether XLA takes the addition into a GEMM’s epilogue or cuBLAS splits a product’s sum by its shape (row 1725). |
precision_names | The canonical names a jax.lax.PrecisionLike spells, unordered. |
products_dtype | The dtype operands multiply in, by jax’s implicit promotion; None where an 8-bit float takes part, which jax promotes with nothing and whose product has no precision to resolve. |
rms_statistics | Mean square and inverse RMS in at least fp32. |
rms_weighted | An RMS norm’s arithmetic past its statistics: x times its rows’ inverse RMS (rms_statistics’s, or a kernel’s own), times weight, times activation(gate) (activation_and_slope: Qwen3-Next’s gated norm), in that order and in x’s dtype, which the caller widens (at least fp32) and rounds once after. |
rounded_operand | The value x takes in dtype, held in x’s own dtype (_rounded), with a straight-through derivative (dew.nn.derivatives.surrogate): its tangent is x’s, never rounded a second time. |
rounded_to | x rounded to dtype’s precision and held in x’s own dtype: where a torch computation stores a tensor of dtype. |
rounds_to_bf16 | Whether an fp32-result product under this compute dtype and precision multiplies bf16 operands (operand_precision). |
running_sum | An empty running sum of shape (added_to, running_total): [3, *shape], a double-float pair and the plain sum beside it, where loops compensate (compensates_loops), else [1, *shape], the plain sum alone. |
running_total | A running sum’s value (running_sum, or its leading axis’s parts): a pair’s rounded once (compensated_total), else the sum. |
scaled | x * factor with the factor kept in fp32 and only the product rounded to x’s dtype, which is what torch’s bf16_tensor * python_float does (fp32 opmath) and what Gemma’s embed_scale here does. |
scaled_logits | A vocabulary head’s per-column terms after its product’s rounding, in fp32 or wider: each column times its scale (nGPT’s per-token logit gain), then plus its bias. |
span_means | The mean of values [B, L, D] over each [..., 2] (start, end) span of its row. |
spread | value broadcast to shape, whose transpose, the sum of a cotangent over every position the value spread over (a bias’s gradient), a CPU running wide arithmetic (WIDE_CPU_ARITHMETIC, Dew’s tests) takes in float64 and rounds once, as it sums a float32 product (accumulated): there the broadcast is the outer product of ones and the value, whose transpose is a product. |
sqrt_or_zero | sqrt(square), and zero with a zero gradient where square is zero: sqrt’s infinite derivative there would turn a masked row’s zero upstream gradient into NaN for every parameter it touches. |
stochastic_bf16 | fp32 x rounded to bf16 up in magnitude with the probability of its dropped low 16 bits: noise’s low 16 bits (uint32, uniform) added below the kept bits and the sum truncated, which carries into them exactly with that probability. |
two_product | Dekker’s rounded product and its residual, with a Veltkamp significand split. |
two_sum | left + right rounded, and exactly what the rounding dropped (Knuth’s TwoSum). |
weighted_total | Weighted total and mass along axis (an int, a tuple, or None for every axis), in x’s dtype, each summed once rounded (compensated_sum), so a loss over many tokens rounds as its exact sum does. |
wide_cpu_arithmetic | Whether this process runs wide CPU arithmetic (WIDE_CPU_ARITHMETIC). |
widened_reduction | reduction(*operands), a reduction returning arrays: the plain one, but in a process running wide CPU arithmetic (WIDE_CPU_ARITHMETIC, Dew’s tests) a CPU takes it over the float32 operands widened exactly to float64, each float64 result rounded once back to float32, as accumulated takes a product’s sums: XLA’s CPU compiler runs a reduction with a combine of its own as one loop over the row, rounding at each step. |
ACCURATE
Section titled “ACCURATE”ACCURATE = Numerics(DTYPE_PRESETS['32-true'], matmul='highest', forms='accurate')Accurate, scientific work at its cost: float32 throughout, every product that states no precision in IEEE float32 and each op’s accurate form.
DTYPE_PRESETS
Section titled “DTYPE_PRESETS”DTYPE_PRESETS = {'bf16-mixed': Dtypes(), '32-true': Dtypes(compute='float32'), 'bf16-true': Dtypes(params='bfloat16', output='bfloat16'), '16-mixed': Dtypes(compute='float16')}The usual dtypes by Lightning’s names: float32 parameters computing in bf16 by default, float32 throughout, bf16 throughout, and float16 compute over float32 parameters.
FP32_SUMS
Section titled “FP32_SUMS”FP32_SUMS = 'F32_F32_F32'JAX’s dot algorithm of fp32 operands summed in fp32: as a process’s or a scope’s default matmul
precision, the statement that its fp32 products sum as an accelerator sums them, which accumulated
honors on a CPU.
L2NORM_EPS
Section titled “L2NORM_EPS”L2NORM_EPS = 1e-06The epsilon l2norm adds under its square root: FLA’s, and Dory’s justnorm’s.
MATMUL_PRECISIONS
Section titled “MATMUL_PRECISIONS”MATMUL_PRECISIONS = ('default', 'high', 'highest')JAX’s matmul precision levels (jax_default_matmul_precision): “default” TF32 for a float32 product
on a GPU and one bf16 pass on a TPU, “highest” IEEE float32.
NUMERICS_PRESETS
Section titled “NUMERICS_PRESETS”NUMERICS_PRESETS = {'accurate': ACCURATE}The named run numerics beside the default (Numerics()).
ROUNDING_STREAM
Section titled “ROUNDING_STREAM”ROUNDING_STREAM = 1919907182The stream stochastic rounding draws from: what a training step folds its key with
(dew.objectives.base.training_rngs), and what a served row’s rounding key starts from
(dew.sampling.text.rounding_keys), apart from dropout’s and sampling’s.
WIDE_CPU_ARITHMETIC
Section titled “WIDE_CPU_ARITHMETIC”WIDE_CPU_ARITHMETIC = 'DEW_WIDE_CPU_ARITHMETIC'The environment variable that, set to 1, has a process’s CPU programs take the arithmetic Dew’s
tests hold a CPU to: float32 products summed in float64 and rounded once (accumulated, spread), a
loop’s running sum of them compensated so it rounds once too (compensates_loops), and a float32
reduction taken in float64 and rounded once (widened_reduction). Read as each program is traced;
set by the tests’ environment (tests/lane_environment.py), never by the library, which without it
takes none of these forms on any platform. Each such site chooses the wide form by the platform the
program is lowered for (platform_dependent, or for a loop’s carry the platform its code is chosen
for), so a test that lowers for an accelerator keeps the target’s.
Dtypes
Section titled “Dtypes”class Dtypes( params: str = 'float32', compute: str = 'bfloat16', output: str = 'float32',)A run’s dtypes, three as Flax casts them: the parameters’ (its layers’ param_dtype, and the
optimizer’s moments’), the compute’s (their dtype) and the outputs’ (the logits and losses a
model returns). A source the run keeps frozen is stored as its checkpoint stores it, unless the
parameters’ dtype is narrower (dew.config.RunConfig.source_keywords).
compute: str-
bf16 by default, Lightning’s bf16-mixed, where Flax computes in its parameters’ float32: a run’s matmuls run on the tensor cores’ bf16 rate, its float32 masters and moments keep the update exact, and the float32 products a run needs say so (
exact_float32); “32-true” computes in float32.
Numerics
Section titled “Numerics”class Numerics( dtypes: Dtypes = Dtypes(), matmul: str | jax.lax.DotAlgorithmPreset = 'default', forms: Literal['fast', 'accurate'] = 'fast',)A run’s numerics, set once and recorded with it, which its model, optimizer and trainer read: its
dtypes (Dtypes, which a model takes as Flax’s param_dtype and dtype and its optimizer as its
moments’; DTYPE_PRESETS names the usual ones by Lightning’s names), the precision every product
that states none multiplies at (JAX’s own jax_default_matmul_precision level, or a
jax.lax.DotAlgorithmPreset, which a record holds by its name), and each op’s form, its kernels’
fast or accurate arithmetic (dew.nn.kernels.generation.run_precision). A precision a call or a
model states still wins, and determinism is a setting of its own. The default is bf16-mixed at
JAX’s default precision in the fast forms; ACCURATE names the scientific one
(NUMERICS_PRESETS).
accumulated
Section titled “accumulated”def accumulated( product: Callable[..., jax.Array], *operands=(), preferred_element_type: DTypeLike | None = None,) -> jax.Arrayproduct(*operands, preferred_element_type=...), a sum of products of operands (a dot, an
einsum, a matmul): the plain product, but in a process running wide CPU arithmetic
(WIDE_CPU_ARITHMETIC, Dew’s tests), where they multiply in float32 (products_dtype) a CPU
sums the products in float64 and rounds the sum once to the dtype the product returns.
Each fp32 product is exact in float64, so the result is within half an fp32 ulp of the float64
sum at every shape, where XLA’s CPU dot chooses its fp32 accumulation by shape, instruction set
and release (jax 0.12’s sums twice numpy’s error from float64 at widths where jax 0.11’s summed
0.6 of it). It costs a CPU product about twice the time (2048-cubed 56 to 126 ms on armada’s
EPYC); a GPU or TPU product, and any whose products are 16-bit, are the product itself. float64
runs on these ops alone, under jax.enable_x64 for their own trace, which nothing outside
reads, and their gradients and tangents are float64 sums too. An operand that is not an array (a
quantized one, Qwix’s QArray) leaves the product to its own library, whose dequantized sum
keeps XLA’s accumulation.
A library that swaps JAX’s products for its own while a model runs (Qwix, quantizing) owns them:
the product is its, as flax’s own layers’ would be. One whose trace calls a backward of its own
(a jax.custom_vjp, Qwix’s when it quantizes a model directly) is left to it whole, its forward
and its backward, read from the trace that gives the product’s dtype (_own_backward).
The wide sum is Dew’s test setting, not a user option: a process without WIDE_CPU_ARITHMETIC
traces the plain product alone. Under it, it is real CPU execution’s policy alone, chosen by the
platform the program is lowered for
(platform_dependent), which keys every compiled program: a step a CPU lowers for a GPU target
(dew.training.trainer.Trainer.lower_for, lowering_platforms=('cuda',)) takes the target’s
fp32 sum, and a Pallas kernel’s body, interpreted on a CPU to stand in for a GPU, never forms its
products here. A computation that stands for such a target, a reference for a kernel interpreted
on a CPU, states its fp32 sums with JAX’s dot algorithm for them, with jax.default_matmul_precision(FP32_SUMS), which jit’s cache keys as it keys every precision.
activation_and_slope
Section titled “activation_and_slope”def activation_and_slope(gate, activation: str) -> tuple[jax.Array, jax.Array]An fp32 gate’s act(gate) and its derivative, for ‘silu’ or ‘sigmoid’.
silu’s derivative s + g s (1 - s), with s = sigmoid(g), cancels to
zero near g = -1.28; formed as the product rule forms it, s + g (s (1 - s)),
its worst element there is half (g s)(1 - s) + s’s, as XLA’s is.
added_to
Section titled “added_to”def added_to(running: Running, term: jax.Array) -> Runningrunning (running_sum) with term added: through compensated_add where it is a pair. A running
sum held as its leading axis’s parts (a tuple) stays one, as a kernel whose layouts hold no unit axis
carries it.
asks_default_precision
Section titled “asks_default_precision”def asks_default_precision( precision: jax.lax.PrecisionLike, *, configured: bool = False,) -> boolWhether precision asks for no more than the default. With
configured, an unset precision reads jax_default_matmul_precision,
the default a product without one gets.
at_default_precision
Section titled “at_default_precision”def at_default_precision(fn: Callable[..., Output]) -> Callable[..., Output]fn traced, forward and backward, under the default matmul precision.
Mosaic’s TPU kernels refuse a 16-bit matmul at HIGHEST (“Bad lhs type”), and Pallas dots read the precision from the configuration at trace time. 16-bit dots are exact at any precision; an fp32 dot takes one bf16 pass on a TPU, so splash’s P·V rounds P to bf16 as every default-precision run does. The kernels take no per-dot precision, so there is no other setting. The backward is traced after the forward returns, so it is held under the same setting by hand.
at_least_fp32
Section titled “at_least_fp32”def at_least_fp32(dtype: DTypeLike | None) -> jnp.dtypeThe dtype arithmetic Dew keeps from rounding below float32 runs in:
float32 for float32, every narrower float and a module’s unset dtype
(None), and dtype itself where it is wider. Norm statistics, softmaxes,
rotary and time-embedding angles, the vocabulary head’s product and loss
reductions read it, so a bfloat16 model reduces in float32 and a float64
model computes in float64 throughout, not with its float32 twin’s
roundings. An 8-bit float, which promotes implicitly to nothing, reads
float32 as every narrower float does.
bf16_dot_general
Section titled “bf16_dot_general”def bf16_dot_general(lhs: jax.Array, rhs: jax.Array, dimension_numbers) -> jax.Arrayjax.lax.dot_general(lhs, rhs, dimension_numbers) multiplied as XLA’s BF16_BF16_F32 dot
algorithm states the arithmetic: the operands rounded to bf16 (to nearest even), each product
exact and summed in fp32 to an fp32 result. Backward, the cotangent rounds to bf16 once and each
operand’s gradient is the same product of it and the other operand, held in the operand’s dtype;
forward mode takes the product’s own tangent (linear_tangent). The one product Dew runs at
that algorithm: the dot family’s (float32_dot_general, float32_einsum) and a bf16 head’s.
Its operands are bf16 arrays multiplied at the default precision, whatever a scope configures (an F16 preset would narrow a bf16 operand to fp16), not wider ones under the algorithm; a CPU, which runs no bf16 dot at some layouts, multiplies their values in fp32, exactly. On an A100, XLA’s dot under the algorithm with a vector side (one row or one column), or below its GEMM rewrite threshold (6 x 3 x 8), did not round fp32 operands to nearest even, and products of exact bf16 values came out off round-to-nearest-even at their ties; at a vector side bf16 operands under the algorithm missed too, at the default precision and at the highest. With bf16 operands and no algorithm every shape was exact (node receipts SF-a100-tie-probe3 and 4). A one-token tile of the chunked head and a one-row decode are such dots.
bf16_einsum
Section titled “bf16_einsum”def bf16_einsum(subscripts: str, *operands=(), **kwargs={}) -> jax.Arrayjnp.einsum(subscripts, *operands) in fp32, its every product bf16_dot_general’s: einsum plans
the contraction (its path, broadcasts, and each operand’s own sums, diagonals and transposes, which
form no product and run in fp32) and hands each pairwise product to bf16_dot_general as its
_dot_general. kwargs are einsum’s own (optimize, out_sharding).
bf16_logits
Section titled “bf16_logits”def bf16_logits(logits: jax.Array) -> jax.Arrayfp32 logits rounded to bf16 and held in fp32, as rounded_to rounds
them, written as bf16 under CUDA.
There the cast joins the product’s epilogue and each reader widens in its
own fusion, behind a barrier xla_allow_excess_precision cannot see
through. reduce_precision instead stayed a kernel of its own after the
Triton head, writing and reading the fp32 logits in full when a serving
step’s draw read them three times: 0.16 ms of a step at 128 rows of
Qwen3-0.6B on an RTX 4080, the same bits (docs/performance.md).
bf16_product
Section titled “bf16_product”def bf16_product(subscripts: str, hidden: jax.Array, head: jax.Array) -> jax.Arrayjnp.einsum(subscripts, hidden, head) multiplied as bf16 and its result rounded to bf16, held
in fp32: the one product a bf16 head computes (head_product) and a tiled head computes over
each tile, so the two round each logit alike.
biased
Section titled “biased”def biased(product: jax.Array, bias: jax.Array) -> jax.Arrayproduct plus bias broadcast over its leading axes, in the dtype their addition promotes
to, as flax’s layer adds them.
Rounding the product and the bias once together, as torch’s addmm does, wrote an fp32 product
and a pass of its own on every biased layer (a DiT-B step 17.5% slower on an A100), for a bf16
output 1.37 times torch’s rounding, which the parity rule holds already.
JAX transposes a broadcast into a sum in the broadcast operand’s dtype (jax-ml/jax#30310), so
a bfloat16 bias added to bfloat16 products sums its gradient over every position in bfloat16,
where torch’s addmm sums its in float32: Qwen 2’s projection biases, summed over 8192 tokens,
came out up to 3.75 times further from float64 than torch’s own bfloat16 gradients. Broadcast
at fp32, the gradient sums at fp32 and rounds once into the bias. A CPU running Dew’s wide
arithmetic (WIDE_CPU_ARITHMETIC) sums it in float64, as it sums a float32 product (spread).
capped_logits
Section titled “capped_logits”def capped_logits(logits: jax.Array, softcap, temperature: float = 1.0) -> jax.ArraySoftcapped logits over a sampling temperature, which an engine applies to what the model’s head returns.
compensated_add
Section titled “compensated_add”def compensated_add(left, right)Add two double-float values, (hi, lo) pairs whose sum is the value:
hi is the sum rounded and lo what the rounding dropped (the his’
TwoSum, the los added to its remainder, then the pair’s own TwoSum), so a
running sum keeps fp32’s precision squared and its hi is the running
value rounded once, where a pair left unrenormalised keeps the plain
running sum in hi, every addition’s rounding in it.
XLA reassociates no float arithmetic unless fast math is on; with it,
lo cancels to zero and the sum is the plain one.
compensated_difference
Section titled “compensated_difference”def compensated_difference(later, earlier)later - earlier of two double-float values, rounded once: the halves’
differences summed, so the difference of two near values (a range of a
running sum) keeps its own precision, where the his’ difference alone
carries their whole magnitude’s rounding: a one-token range of a decay’s
running sum near -78 off by 4e-6. Broadcasts as the halves do.
compensated_sum
Section titled “compensated_sum”def compensated_sum(values: jax.Array, axes: tuple[int, ...]) -> jax.ArrayThe sum of values over axes, rounded once: one XLA reduction whose accumulator is a
double-float (hi, lo) pair combined by compensated_add, so it partitions as any reduction
does, and what the roundings dropped is added back at the end.
One plain reduction over a loss’s every axis rounds as it goes; on 6144 JEPA squares it left the
sum one float32 step above the correctly rounded one, and the loss a float32 ulp from float64,
where numpy’s float32 mean of the same squares was correctly rounded. A narrower float adds in
float32 and returns in its own dtype, as jnp.sum does; an integer sum is jnp.sum’s. The
derivative is jnp.sum’s.
compensated_total
Section titled “compensated_total”def compensated_total(pair: tuple[jax.Array, jax.Array], plain: jax.Array) -> jax.ArrayA compensated sum’s value, its (hi, lo) pair rounded once, or the plain sum of the same
values where that reaches an infinity: +-inf or NaN as jnp.sum’s is, where every residual past
the infinity is inf - inf. compensated_add itself stays arithmetic alone, so a tangent’s scan
through it stays linear; a running sum that can overflow keeps its plain sum beside the pair
and reads its value here.
compensates_loops
Section titled “compensates_loops”def compensates_loops() -> boolWhether a running sum of float32 products carried across a loop keeps what each addition
drops (compensated_add), so the loop’s sum rounds once as one product’s does: where a CPU sums
each product in float64 (accumulated), in a process running Dew’s wide CPU arithmetic
(WIDE_CPU_ARITHMETIC). A carry’s shape is fixed when its loop traces, so this
reads the platform a call’s code is chosen for (dew.nn.kernels.generation.backend), as a CPU’s
attention routing does; an accelerator’s running sum is the plain one, as its products’ sums are
float32’s.
configurable_precision
Section titled “configurable_precision”def configurable_precision(precision: jax.lax.PrecisionLike) -> boolWhether jax.default_matmul_precision can name precision: one precision or dot algorithm
preset, unset or default included; not a pair of precisions or a dot algorithm.
configured_precision
Section titled “configured_precision”def configured_precision(precision: jax.lax.PrecisionLike)A context in which a product that takes no precision of its own (jax.nn’s xla attention)
multiplies at precision, through jax.default_matmul_precision; an unset or default
precision changes nothing. A precision no config names (configurable_precision) is
refused.
correctly_rounded_divide
Section titled “correctly_rounded_divide”def correctly_rounded_divide( numerator: jax.typing.ArrayLike, denominator: jax.typing.ArrayLike,) -> jax.ArrayThe correctly rounded fp32 quotient, IEEE division’s, on XLA’s approximate dividers too: XLA:CPU rewrites a division by a broadcast into a multiplication by its reciprocal (openxla/xla#51148), and XLA:GPU’s fp32 a / b is off IEEE on 30 to 50% of quotients (DIVIDE-a100, DIVIDE-h100 receipts), where a quantizer’s scales and a diffusion schedule’s states must round as the reference’s host or CUDA division does (compressed-tensors’ NVFP4, the FP8 block cast, PyTorch’s schedulers).
Each operand’s significand, in [0.5, 1), divides apart from its exponent
(_rounded_significand), so every intermediate stays normal whatever the
operands’ range; the exponents’ difference then scales the rounded
significand by its exponent bits, exact for a normal result. A quotient
below the normal range rounds once to the subnormal grid: the significand
scaled onto that grid rounds to its nearest integer, ties to even, the exact
residual settling a scaled significand that lands on a half, and the
integer is the result’s bits, which no flush to zero reaches. Past the
largest float the quotient is infinite. A zero, infinite or NaN operand,
or a subnormal one, which XLA:CPU reads as zero, is the division’s. Its
derivative is the division’s, as JAX differentiates a / b.
divided
Section titled “divided”def divided(numerator, rate)numerator / rate as a source written in PyTorch divides a state by a rate, or an array by a
constant: IEEE division, whatever the backend’s own division does (XLA takes a division by a
constant as a product with its rounded reciprocal, on a CPU and an A100 alike, row 1402). A
float32, bfloat16 or float16 quotient is correctly_rounded_divide’s of the operands widened to
float32, rounded to its dtype, as PyTorch’s float32 opmath gives it, under any compiler flags. A
float64 one is the plain division.
exact_float32
Section titled “exact_float32”def exact_float32(precision: jax.lax.PrecisionLike) -> jax.lax.PrecisionLikeThe precision a float32 product whose rounding decides what follows takes, a numerically critical
one, a vocabulary head’s (the loss’s logits), a router’s or a recurrence’s state’s, each stating its
reason where it calls this: the one the call states, else IEEE float32 (Precision.HIGHEST), stated
over JAX’s configured precision, whose default runs TF32 on a GPU and one bf16 pass on a TPU, where
the references multiply IEEE float32 (torch’s float32 matmul). Every other product follows JAX’s
configured precision and the run’s numerics (dew.nn.kernels.generation.run_precision).
float32_dot_general
Section titled “float32_dot_general”def float32_dot_general( lhs, rhs, dimension_numbers, precision: jax.lax.PrecisionLike = None, preferred_element_type: DTypeLike | None = None, **kwargs={},) -> jax.Arrayjax.lax.dot_general at precision, accumulated as Dew accumulates a CPU’s float32 products
(accumulated): the product a flax layer forms (nn.Dense(dot_general=...)). Unstated, JAX’s
configured precision decides (jax_default_matmul_precision), as it does a flax layer’s. At the
BF16_BF16_F32 algorithm it is bf16_dot_general’s product.
float32_einsum
Section titled “float32_einsum”def float32_einsum( subscripts: str, *operands=(), precision: jax.lax.PrecisionLike = None, preferred_element_type: DTypeLike | None = None, **kwargs={},) -> jax.Arrayjnp.einsum at precision, JAX’s configured one where unstated, as float32_dot_general is a
layer’s, accumulated alike, and bf16_einsum’s product at the BF16_BF16_F32 algorithm.
float32_matmul
Section titled “float32_matmul”def float32_matmul(lhs, rhs, precision: jax.lax.PrecisionLike = None) -> jax.Arraylhs @ rhs at precision, JAX’s configured one where unstated, accumulated as
float32_dot_general is.
float64_sum
Section titled “float64_sum”def float64_sum( product: Callable[..., jax.Array], *operands=(), dtype: DTypeLike,) -> jax.Arrayproduct(*operands), a sum of products, taken in float64 from its operands widened exactly and
rounded once to dtype, under jax.enable_x64 for its own trace: the sum accumulated takes on a
CPU under Dew’s wide arithmetic (WIDE_CPU_ARITHMETIC), with no platform of its own to choose.
Its products run at HIGHEST precision, whatever fp32 algorithm the caller bound, which would
narrow float64 operands back to float32.
fp32_result_dot_general
Section titled “fp32_result_dot_general”def fp32_result_dot_general(precision: PrecisionLike = None)A flax layer’s dot_general for a head whose result stays at least
fp32.
The layer has already promoted both operands when this runs - to its own
dtype, or to the activations’ where it carries none - so the compute
dtype is the operands’ own. Only the accumulation and the result widen,
to at_least_fp32 of it.
head_dot_general
Section titled “head_dot_general”def head_dot_general(dtype: Dtype | None, precision: PrecisionLike = None)A flax layer’s dot_general for a vocabulary head: a result at
least fp32 whose operands are the compute dtype’s values, in both
directions.
dtype is the model’s compute dtype, not the layer’s: the layer
promotes the states to at least fp32. Under bf16 compute at the default
precision both operands multiply as bf16 and the logits are bf16 values
(head_product); otherwise the product is the layer’s own, in the
operands’ dtype, at the precision stated, IEEE float32 where none is (exact_float32): its
logits feed the loss’s logsumexp and the sampler, logits tens wide that TF32’s part in a thousand
moves.
head_product
Section titled “head_product”def head_product( subscripts: str, hidden: jax.Array, head: jax.Array, precision: PrecisionLike = None, *, compute: DTypeLike | None = None,) -> jax.Arrayjnp.einsum(subscripts, hidden, head) as a vocabulary head computes
it, with accumulation and a result at least fp32.
The product follows the compute dtype, as torch autocast and MaxText
(logits_dot_in_fp32=False) run it: bf16 states at the default precision
multiply the head as bf16 and the logits round to bf16, held in fp32, with
the cotangent rounded once alike. Anything else multiplies fp32 states by
the head as stored. The rounding cost nothing measurable in 2000-step
wikitext-103 runs (validation loss within 4.5e-4 and 1.6e-3 of fp32 logits,
below seed noise) and saves 13% of a 3-layer decoder’s step. It does move
layout parity (a 4 x RTX 3090 bf16 MoE read 5758 of its bound against 0.41
in fp32), so a run comparing layouts or resuming onto another mesh sets
matmul_precision “highest”, whose head is fp32. compute, the model’s
compute dtype where the states arrive in another, decides in their place.
l2norm
Section titled “l2norm”def l2norm(x: jax.Array, eps: float = L2NORM_EPS) -> jax.Arrayx over its L2 length along the last axis, x * rsqrt(sum(x**2) + eps) with eps added under
the root, computed in at least fp32 (at_least_fp32, a float64 input’s in float64) and rounded
once to x’s dtype: the normalization of FLA’s l2norm (the linear attentions’ queries and
keys, modeling_qwen3_next.py:368-371) and of nGPT’s justnorm (its residual stream and
weights).
numerics_of
Section titled “numerics_of”def numerics_of(given: Numerics | str) -> NumericsA run’s numerics from what a flag, a record or a load names: a Numerics as it is, a preset’s
name (NUMERICS_PRESETS, “default” for Numerics()), or the usual dtypes’ Lightning name
(DTYPE_PRESETS) at the default precision and forms.
operand_precision
Section titled “operand_precision”def operand_precision( dtype: Dtype | None, precision: jax.lax.PrecisionLike = None,) -> jax.lax.PrecisionLikeThe precision a product of dtype operands takes: the one stated, JAX’s configured one where
none is, and for bf16 operands at the default precision the algorithm that keeps a bf16 dot bf16
in both directions. A caller that asked for more than the default keeps what it asked for, and so
does a run whose configured level asks for more where the call states none (a run’s
Numerics.matmul, generation.run_precision), and a GPU older than sm80, which rejects the
algorithm at run time (dew.nn.kernels.generation.BF16_GPU). A configured dot algorithm preset is
no level a run picks but how dots that state none sum (check 1’s arms state F32_F32_F32), which the
algorithm’s fp32 sums already are, so it keeps the algorithm.
paired_sum
Section titled “paired_sum”def paired_sum(shape: tuple[int, ...], dtype: DTypeLike) -> jax.ArrayAn empty running sum of shape that is a double-float pair on every backend, the plain sum
beside it (added_to, running_total): for partials of a product Dew splits where its reference
takes the product whole, so the sum’s error is the partials’ own, whether XLA takes the addition
into a GEMM’s epilogue or cuBLAS splits a product’s sum by its shape (row 1725).
precision_names
Section titled “precision_names”def precision_names(precision: jax.lax.PrecisionLike) -> frozenset[str]The canonical names a jax.lax.PrecisionLike spells, unordered.
JAX’s alias is None, a string, a jax.lax.Precision, a dot algorithm
preset or a dot algorithm, or a pair of precisions; flax’s is the same
without the algorithms. A string is read as JAX reads
jax_default_matmul_precision: a dot algorithm preset’s name is that
preset (‘F32_F32_F32’), any other goes through jax.lax.Precision, so
‘tensorfloat32’ is ‘HIGH’ and ‘bfloat16’ is ‘DEFAULT’; a preset keeps its
own name, and a dot algorithm, which has none, is named by its fields.
products_dtype
Section titled “products_dtype”def products_dtype(*operands=()) -> jnp.dtype | NoneThe dtype operands multiply in, by jax’s implicit promotion; None where an 8-bit float
takes part, which jax promotes with nothing and whose product has no precision to resolve.
rms_statistics
Section titled “rms_statistics”def rms_statistics( x: jax.Array, epsilon: float, *, axis=-1, keepdims: bool = True, count: int | None = None,) -> tuple[jax.Array, jax.Array]Mean square and inverse RMS in at least fp32. A padded kernel tile supplies its logical
count after zeroing padding; an ordinary array uses its reduction dimensions. Both means multiply
by the reciprocal count, as XLA’s mean does. Callers own when the inverse, normalized values and
learned scale round. The count’s reciprocal is a Python float, where jnp.reciprocal of the count’s
array is an integer power, which a Pallas Triton kernel cannot lower on a folded constant.
rms_weighted
Section titled “rms_weighted”def rms_weighted( x: jax.Array, inverse: jax.Array, *, weight: jax.Array | None = None, gate: jax.Array | None = None, activation: str = 'silu', rounded: Callable[[jax.Array], jax.Array] | None = None,) -> jax.ArrayAn RMS norm’s arithmetic past its statistics: x times its rows’ inverse RMS (rms_statistics’s,
or a kernel’s own), times weight, times activation(gate) (activation_and_slope: Qwen3-Next’s gated
norm), in that order and in x’s dtype, which the caller widens (at least fp32) and rounds once after.
rounded rounds the normalized rows, before any weight multiplies them (Llama’s and Qwen3’s cast first,
RMSNorm.scale_after_cast; a norm computing in its input’s dtype). The fused norm kernel, the
recurrence’s gated output norm, the portable norm and the portable gated norm compute it alike, each
keeping its own statistics, memory schedule and derivative.
rounded_operand
Section titled “rounded_operand”rounded_operand = surrogate(_rounded, lambda dtype, primals, tangents, value: tangents[0], nondiff_argnums=(1,))The value x takes in dtype, held in x’s own dtype (_rounded), with a straight-through
derivative (dew.nn.derivatives.surrogate): its tangent is x’s, never rounded a second time.
rounded_to
Section titled “rounded_to”def rounded_to(x: jax.Array, dtype: Dtype) -> jax.Arrayx rounded to dtype’s precision and held in x’s own dtype: where a
torch computation stores a tensor of dtype. Under jit XLA’s GPU default
xla_allow_excess_precision deletes a narrowing cast a widening one
follows; reduce_precision keeps the rounding but flushes float16
subnormals, so float16 rounds by the cast behind an optimization barrier
(rounded_operand). The derivative rounds where the value does, as a torch
tensor of dtype’s gradient does.
rounds_to_bf16
Section titled “rounds_to_bf16”def rounds_to_bf16(dtype: Dtype | None, precision: PrecisionLike = None) -> boolWhether an fp32-result product under this compute dtype and precision
multiplies bf16 operands (operand_precision).
running_sum
Section titled “running_sum”def running_sum(shape: tuple[int, ...], dtype: DTypeLike) -> jax.ArrayAn empty running sum of shape (added_to, running_total): [3, *shape], a double-float
pair and the plain sum beside it, where loops compensate (compensates_loops), else [1, *shape],
the plain sum alone. Its leading axis is the form’s, so a loop slices and updates the sum’s own
axes one past theirs.
running_total
Section titled “running_total”def running_total(running: Running) -> jax.ArrayA running sum’s value (running_sum, or its leading axis’s parts): a pair’s rounded once
(compensated_total), else the sum.
scaled
Section titled “scaled”def scaled(x: jax.Array, factor: float) -> jax.Arrayx * factor with the factor kept in fp32 and only the product rounded
to x’s dtype, which is what torch’s bf16_tensor * python_float does
(fp32 opmath) and what Gemma’s embed_scale here does. Rounding 0.22 to
bf16 first would shrink every product by a systematic 0.12%.
scaled_logits
Section titled “scaled_logits”def scaled_logits( logits: jax.Array, scale: jax.Array | None, bias: jax.Array | None,) -> jax.ArrayA vocabulary head’s per-column terms after its product’s rounding, in fp32 or wider: each
column times its scale (nGPT’s per-token logit gain), then plus its bias.
span_means
Section titled “span_means”def span_means(values: jax.Array, spans: jax.Array) -> jax.ArrayThe mean of values [B, L, D] over each [..., 2] (start, end) span of its row.
spans is [B, ...]; an empty span, a padding slot’s, gives zeros. A token no span of its row
reads is selected away before the product, so padding’s NaN or infinity never reaches a mean (the
rule of weighted_total), and the sum is the product’s accumulated one.
spread
Section titled “spread”def spread(value: jax.Array, shape: tuple[int, ...]) -> jax.Arrayvalue broadcast to shape, whose transpose, the sum of a cotangent over every position the
value spread over (a bias’s gradient), a CPU running wide arithmetic (WIDE_CPU_ARITHMETIC, Dew’s
tests) takes in float64 and rounds once, as it sums a float32 product (accumulated): there the
broadcast is the outer product of ones and the value, whose transpose is a product. Elsewhere,
or where accumulated leaves a product its own sum, a CPU spreads a floating value by doubling
(_doubled), so its transpose sums pairwise in the value’s dtype, and an accelerator broadcasts
it, whose transpose JAX sums in the value’s dtype. A value an Explicit axis places along an axis it
spreads over (never a bias, whose leading axes biased adds) is broadcast: neither form keeps that
placement on a dropped or doubled size-one axis.
sqrt_or_zero
Section titled “sqrt_or_zero”def sqrt_or_zero(square)sqrt(square), and zero with a zero gradient where square is zero:
sqrt’s infinite derivative there would turn a masked row’s zero upstream
gradient into NaN for every parameter it touches.
stochastic_bf16
Section titled “stochastic_bf16”def stochastic_bf16(x, noise)fp32 x rounded to bf16 up in magnitude with the probability of its dropped low 16 bits: noise’s
low 16 bits (uint32, uniform) added below the kept bits and the sum truncated, which carries into them
exactly with that probability. It does not saturate: a value past bf16’s largest finite rounds to
infinity with its dropped bits’ probability, as rounding to nearest would. NaN stays NaN. The noise is
its caller’s policy: a hash of the state’s own bits and the token (recurrence.round_to_bf16), of the
step’s seed and the element’s index (dew._moments.stochastic_round_bf16).
two_product
Section titled “two_product”def two_product( left, right, *, bit_split: bool = False, subtract: bool = False, rounding: Callable[[jax.Array], jax.Array] | None = None,) -> tuple[jax.Array, jax.Array]Dekker’s rounded product and its residual, with a Veltkamp significand split.
Guidance keeps the multiply split, which also supports float64. bit_split
truncates fp32 significands instead, avoiding splitter overflow in norm
interpretation and quantizer division; near underflow the splits differ.
subtract keeps division’s subtractive residual order and its signed zeros.
Norm interpretation passes optimization_barrier as rounding to keep each
product and residual step separate; the other callers allow fusion.
With each arithmetic step rounded, the residual is exact where intermediates
neither overflow nor underflow.
two_sum
Section titled “two_sum”def two_sum(left, right)left + right rounded, and exactly what the rounding dropped (Knuth’s TwoSum).
weighted_total
Section titled “weighted_total”def weighted_total( x: jax.Array, weights: jax.Array, axis=None,) -> tuple[jax.Array, jax.Array]Weighted total and mass along axis (an int, a tuple, or None for every axis), in x’s
dtype, each summed once rounded (compensated_sum), so a loss over many tokens rounds as its
exact sum does. Broadcastable weights mark excluded values with zero; those values are selected
away before multiplication, so their NaNs and infinities cannot reach the sum. The caller owns
its denominator policy.
wide_cpu_arithmetic
Section titled “wide_cpu_arithmetic”def wide_cpu_arithmetic() -> boolWhether this process runs wide CPU arithmetic (WIDE_CPU_ARITHMETIC).
widened_reduction
Section titled “widened_reduction”def widened_reduction( reduction: Callable[..., Sequence[jax.Array]], *operands: jax.Array = (),) -> tuple[jax.Array, ...]reduction(*operands), a reduction returning arrays: the plain one, but in a process running wide
CPU arithmetic (WIDE_CPU_ARITHMETIC, Dew’s tests) a CPU takes it over the float32 operands widened
exactly to float64, each float64 result rounded once back to float32, as accumulated takes a
product’s sums: XLA’s CPU compiler runs a reduction with a combine of its own as one loop over the
row, rounding at each step. A process without the setting, or operands not float32, take the plain
reduction on every platform, chosen by the platform a program is lowered for (platform_dependent).