dew.distributed.placement
Where a run’s arrays live: the device mesh (MeshSpec), the parameter and state layout over it
(Layout, from the logical axes every module declares, dew.nn.sharding, and the rule table,
dew.mesh), and what a placed tree holds (placed_bytes, placed_shapes). A run’s batches on the
devices are dew.distributed.batches’. It sits below the objectives and the trainer, which place
their state by it, and imports nothing of theirs.
| Name | Summary |
|---|---|
Layout | Decides how a train state is placed on a mesh. |
LeafPlacement | Where one leaf of a placed state lives (Validation.leaves). |
MeshSpec | Device counts for the mesh’s sharding axes; the data axis takes the devices left over. |
RefinesLayout | A model that places its own tree’s leaves: a tier-3 model’s torch-named tree, which no module declares axes for, placed by its tensor-parallel plan (dew.interop.torchax_fallback.TorchLayout). |
Validation | What a placed state holds on a mesh before the first step (Layout.validate). |
Layout
Section titled “Layout”class Layout( rules: LogicalAxisRules | Mapping[str, MeshAxes] = DEFAULT_RULES, min_shard: int = 2 ** 16, tolerance: float = 0.02, host: tuple[str, ...] = (), host_parameters: tuple[str, ...] = (), frozen_whole: bool = False, stage_local: bool = True, data_axes: tuple[str, ...] = (),)Decides how a train state is placed on a mesh.
rules map the logical axis names that modules declare (dew.nn.sharding)
onto mesh axes, in precedence order. They place the parameters, and the
trainer sets them in context so they also place the activations a compiled
step constrains. An axis of size 1 splits nothing, so the same table serves
every topology.
What the rules place on fsdp also splits over the sequence axis, whose
devices compute their own positions with the same weights (STATE_AXES).
shardings refuses rules that place a parameter on the data or stage
axis. The data axis holds the batch’s replicas, each with the whole
state. The stage axis holds a pipeline’s stages: the trainer keeps each
run a pipeline repeats stacked by stage on it, each row placed as these
rules place its layer (dew.training.stages.StageLocal.placement).
Below min_shard elements a parameter costs more in collectives than it
saves in memory, so it stays replicated. tolerance is the fraction of
shardable parameter elements a layout may leave replicated before check
refuses it.
host names the fields of HOST_RESIDENT to keep in pinned host memory
between steps. For opt_state and ema, the step fetches the field to the
device, updates it as usual and writes it back, so the values are the same
and only where they are stored changes. Naming variables instead moves
the whole TrainState to the CPU, including the optimizer state, the EMA and
gradient accumulation. The optimizer update then runs on a CPU companion of
this mesh, and the parameter copies on the accelerators are read-only
snapshots for running the step, never a second master copy. Every process
needs at least as many CPU devices as accelerators, set before JAX
initializes; neither this layout nor the trainer changes the count, and a
process’s surplus CPUs, and the processes the mesh leaves out, stay out of
the companion (companion_mesh).
host_parameters lists the variables an inference placement keeps in
pinned host memory, as globs over their logical paths (params/layers_*).
A selected leaf keeps the spec the rules give it, and only its memory kind
changes. Only offloaded reads the patterns, because only the layer stack
fetches each layer’s parameters as it reaches that layer. When
host_parameters is set, check refuses a placement that keeps every
parameter on the device, so the patterns are never silently ignored.
frozen_whole keeps the frozen collection (FROZEN, a LoRA run’s base)
whole over fsdp and sequence (STATE_AXES), split only over the other
axes the rules give it (a tensor split stays). A frozen leaf has no
gradient and no optimizer state, so a split over them saves only its
memory, and the step gathers it whole forward and again backward.
check counts no leaf it keeps whole, and a leaf host_parameters also
names is kept so in pinned host memory.
stage_local: bool-
Whether a pipelined run’s layers, with their optimizer state and EMA, are kept stacked by stage on their own stage’s devices (
dew.training.stages), about 1/S of the run on each; False keeps every layer whole on every stage, which an optax transformation that may read a stacked leaf whole needs. Either trains the same arithmetic, bit for bit. data_axes: tuple[str, ...]-
The axes a batch’s rows split over, ahead of the rules’ own
activation_batchentries: a native mesh’s names for its data-parallel axes, which the run’s data shares, its batch placement and its row counts read (dew.mesh.batch_axes). Empty leaves the rules as they are. axis_rules: LogicalAxisRules-
The rules as the (name, axes) pairs flax reads, in precedence order,
data_axesfirst.
Layout.shardings
Section titled “Layout.shardings”def shardings(mesh: Mesh, tree: TreeT) -> Placement[TreeT]Return a NamedSharding for each leaf of tree, from the axes its module declared.
A parameter its module boxed (nn.with_logical_partitioning) takes the
axes it carries, in tree or in the trainer’s boxed table, when the
rules place one of them (ruled_boxes). A leaf
whose path no module declares falls back to a shape heuristic,
which splits its largest dimension that fsdp divides evenly over fsdp.
So the axes of one model family can be declared at a time. Either
way, what lands on fsdp splits over the sequence axis too where the
two divide it (state_spec). A leaf below min_shard elements is
replicated either way.
Flax metadata that a caller’s own module attached is removed, because
the state the trainer creates from these shardings holds plain arrays.
Raises ValueError when the rules place a leaf on the data or stage
axis (UNSPLIT_AXES) or on an Explicit axis, or name an axis
the mesh lacks.
Layout.offloaded
Section titled “Layout.offloaded”def offloaded(mesh: Mesh, tree: TreeT) -> Placement[TreeT]Return shardings, with the memory kind each leaf’s path asks for.
Leaves that host_parameters selects go to pinned host memory. Each
leaf keeps the spec the rules give it, so a selected parameter is the
same shard in another memory space, and its layer issues the same
collectives as when the parameter stays on the device.
Every pattern must match at least one variable, or this raises ValueError. A pattern that matches nothing is a typo or a stale path. Without the error the run would keep those weights on the device, and the only sign would be the memory it did not save. So each pattern is checked on its own.
Layout.check
Section titled “Layout.check”def check(params: Variables, shardings: Placement[Variables], mesh: Mesh) -> NoneRefuse a placement that replicates too much of the model or ignores host_parameters.
This is MaxText’s guardrail (base.yml sharding_tolerance) against a mesh whose parameter axes divide none of the model’s dimensions. On such a mesh the shape heuristic replicates every parameter, and nothing else reports it.
MaxText measures the excess per-chip memory over perfect sharding
across every parameter. This check takes the same ratio over the
parameters of at least min_shard elements whose rules name a
parameter axis this mesh splits. Smaller ones are replicated on
purpose, so counting them would fail models that are merely small;
one no rule splits over this mesh’s axes is replicated by the mesh’s
choice, as a data axis replicates every parameter, so an expert-only
mesh keeps a mixture’s dense weights whole as data parallelism does.
The sequence axis splits only where it divides what the rules place
on fsdp (state_spec), so a leaf it cannot split is the rules’ to
count, and a mesh of sequence alone refuses none.
Raises LayoutRefused when the replicated fraction is above
tolerance, naming the five largest replicated parameters, and
ValueError when host_parameters is set but every parameter in
shardings stays on the device. A mesh whose parameter axes all have
size 1 passes the replication check.
Layout.validate
Section titled “Layout.validate”def validate(mesh: Mesh, state, *, capacity: int | None = None) -> ValidationReport what state holds on mesh before the first step, and refuse a state a
device cannot hold.
MaxText prints each parameter’s logical and physical sharding before it trains
(print_shardings_params), and Megatron reports the weight and optimizer bytes a GPU
holds from a formula over the model’s config (report_theoretical_memory). Here both
are read off the placement itself, so they hold for any rules, mesh and optimizer.
state is placed, or is about to be: each leaf an array, or a jax.ShapeDtypeStruct
carrying the NamedSharding it is placed with (shardings, Trainer.shardings), so
nothing is allocated. The report (Validation) gives each leaf’s split and the part each
device holds, the bytes each device of mesh keeps between steps, the axes a batch’s rows
split over, the collectives a step runs over each axis (Validation.collectives), and the
logical axes the variables declare that no rule names (Validation.unbound). A leaf in
host memory (host), or on devices outside mesh (a CPU-owned state’s companion), takes
none of a device’s memory. Rules naming an axis mesh lacks raise ValueError here, as
shardings refuses them.
capacity is the bytes each device of mesh has. Unset, each of this process’s devices
gives its allocator’s limit (Device.memory_stats’ bytes_limit) where JAX reports one,
which is what an allocation fails against, under whatever memory fraction the process
set. A CPU device reports none, so a CPU mesh standing in for accelerators takes theirs
from the caller.
Raises LayoutRefused, naming the device and what it holds by field, when a device
would hold more than its capacity between steps: the state cannot be placed there,
before any of the step’s temporaries, which the plan checks against the compiled step
(dew.training.planning).
LeafPlacement
Section titled “LeafPlacement”class LeafPlacement(spec: P, shard_shape: tuple[int, ...], memory_kind: str)Where one leaf of a placed state lives (Validation.leaves).
spec: P-
The mesh axes each of its dimensions splits over.
shard_shape: tuple[int, ...]-
The part of it each device holds.
memory_kind: str-
The memory that part sits in:
device, or the host memory a layout keeps it in (pinned_host), which takes none of the device’s.
MeshSpec
Section titled “MeshSpec”class MeshSpec( fsdp: int = 1, expert: int = 1, tensor: int = 1, sequence: int = 1, stage: int = 1, replicas: int = 1, diloco: int = 1, explicit: tuple[str, ...] = (),)Device counts for the mesh’s sharding axes; the data axis takes the devices left over.
expert: int-
Devices the expert dimension of an MoE layer is split over.
tensor: int-
Devices the mlp, head and vocabulary widths are split over, in addition to fsdp. 1 keeps every width on fsdp alone.
sequence: int-
Devices the batch’s sequence dimension is split over; 1 keeps whole sequences. The persistent state splits over them with fsdp (
STATE_AXES). stage: int-
Pipeline stages the layer stack is split into, each on its own devices. 1 runs the whole stack on every device. How many microbatches a step feeds through them is the trainer’s (
Trainer.microbatches). replicas: int-
Groups of hosts the data axis spans, for hybrid sharded data parallelism. Every other axis, fsdp included, stays inside one group. So the parameter gathers and gradient reduce-scatters run over the fast links, and only the gradient all-reduce between replicas crosses the slow network. A group is a whole number of granules. A granule is the devices that share a slice_index (a TPU slice, and on multi-host GPU a host or an NVLink domain), or one process when every device reports the same slice. At 1,
jax.make_meshplaces the devices of a single slice; devices on several slices still take the hybrid layout, with each slice as one replica when the slice count divides the data axis. diloco: int-
DiLoCo’s replica groups (
dew.training.diloco), the mesh’s outermost axis: each trains its own inner copy of the model on its own share of every batch’s rows over the devices of the rest of the mesh, and the groups meet only when the outer model averages them, every few updates. Across several slices, the groups take whole slices where they divide them, so only that average crosses the slow network. 1 is no DiLoCo. explicit: tuple[str, ...]-
The axes whose sharding an array’s type carries (JAX’s Explicit mode), where every other axis is left to the partitioner (Auto). An Explicit axis is placed where an array is made and follows the operations from there; a model’s sharding constraints name only the Auto ones.
MeshSpec.build
Section titled “MeshSpec.build”def build(devices: Sequence | None = None) -> MeshBuild the seven-axis device mesh this spec describes.
Parameters shard over ‘fsdp’, ‘expert’ and ‘tensor’. A batch’s rows shard over ‘data’, ‘expert’ and ‘fsdp’, its sequence dimension over ‘sequence’, and the layer stack over ‘stage’.
Expert parallelism splits an MoE layer’s expert dimension, which no
dense model has. So that dimension gets its own axis, and ‘fsdp’ is
left for the model’s widths. The tensor axis splits the mlp, head and
vocabulary widths alongside fsdp, as DEFAULT_RULES places them. The
sequence axis splits long sequences, and the stage axis splits a
decoder’s layers into pipeline stages, each stage on its own devices.
An axis of size 1 splits nothing, and with every size at 1 the mesh is
plain data parallelism. So the same code path serves every topology
without a flag. The axes are Auto, so GSPMD infers the collectives,
but for those explicit names.
devices lists the devices of one slice in the order to use them, any
devices a mesh holds (a compile-only GPU’s among them). The mesh takes
them row-major over MESH_AXES, so neighbouring entries land on the
last axes. When it is unset, the mesh uses every device, in the order
jax.make_mesh gives the platform’s topology.
When replicas is above 1, or the devices are on several slices, the
mesh is built the way MaxText builds a multislice one, through
mesh_utils.create_hybrid_device_mesh. The data axis takes replicas
groups of granules as its outer factor, and each group lays out the
rest of the mesh over its own devices. A group of more than one granule
splits fsdp across them, because fsdp’s traffic (a gather and a
reduce-scatter per layer) is the only kind that tolerates the slower
link.
Raises LayoutRefused when the expert, fsdp, tensor, sequence and
stage sizes multiply to a number that does not divide the device count,
or when replicas does not divide both the granule count and the data
axis.
RefinesLayout
Section titled “RefinesLayout”class RefinesLayout(Protocol)A model that places its own tree’s leaves: a tier-3 model’s torch-named tree, which no module
declares axes for, placed by its tensor-parallel plan (dew.interop.torchax_fallback.TorchLayout).
A run’s objective (Objective.placement) and a source’s load hand it the run’s layout, and it keeps
every choice the layout makes.
RefinesLayout.placement
Section titled “RefinesLayout.placement”def placement(layout: Layout) -> LayoutValidation
Section titled “Validation”class Validation( leaves: dict[str, LeafPlacement], device_bytes: dict[int, int], capacity: dict[int, int], batch_axes: tuple[str, ...], collectives: dict[str, tuple[str, ...]], unbound: dict[str, tuple[str, ...]],)What a placed state holds on a mesh before the first step (Layout.validate).
leaves: dict[str, LeafPlacement]-
Each leaf’s placement by its path (
variables/params/...), in the state’s order. device_bytes: dict[int, int]-
The bytes each device of the mesh holds of the state in its own memory between steps, by the device’s id.
capacity: dict[int, int]-
The bytes each device has, by its id, as the caller stated them or the device reports them; a device that reports none, such as a CPU, is absent.
batch_axes: tuple[str, ...]-
The mesh axes a batch’s rows split over (
batch_axes). collectives: dict[str, tuple[str, ...]]-
The collectives a training step runs over each axis of the mesh above size one, by the role the placement gives the axis, in HLO’s names (
_implied_collectives). unbound: dict[str, tuple[str, ...]]-
Each logical axis the variables declare that no rule names, with the leaves that declare it. A leaf keeps the dimensions it names whole on every device where a rule places another of its names, and takes the shape heuristic where none does (
shardings).