Skip to content

dew.distributed.placement

Where a run’s arrays live: the device mesh (MeshSpec), the parameter and state layout over it (Layout, from the logical axes every module declares, dew.nn.sharding, and the rule table, dew.mesh), and what a placed tree holds (placed_bytes, placed_shapes). A run’s batches on the devices are dew.distributed.batches’. It sits below the objectives and the trainer, which place their state by it, and imports nothing of theirs.

NameSummary
LayoutDecides how a train state is placed on a mesh.
LeafPlacementWhere one leaf of a placed state lives (Validation.leaves).
MeshSpecDevice counts for the mesh’s sharding axes; the data axis takes the devices left over.
RefinesLayoutA model that places its own tree’s leaves: a tier-3 model’s torch-named tree, which no module declares axes for, placed by its tensor-parallel plan (dew.interop.torchax_fallback.TorchLayout).
ValidationWhat a placed state holds on a mesh before the first step (Layout.validate).

dataclass source

class Layout(
rules: LogicalAxisRules | Mapping[str, MeshAxes] = DEFAULT_RULES,
min_shard: int = 2 ** 16,
tolerance: float = 0.02,
host: tuple[str, ...] = (),
host_parameters: tuple[str, ...] = (),
frozen_whole: bool = False,
stage_local: bool = True,
data_axes: tuple[str, ...] = (),
)

Decides how a train state is placed on a mesh.

rules map the logical axis names that modules declare (dew.nn.sharding) onto mesh axes, in precedence order. They place the parameters, and the trainer sets them in context so they also place the activations a compiled step constrains. An axis of size 1 splits nothing, so the same table serves every topology.

What the rules place on fsdp also splits over the sequence axis, whose devices compute their own positions with the same weights (STATE_AXES). shardings refuses rules that place a parameter on the data or stage axis. The data axis holds the batch’s replicas, each with the whole state. The stage axis holds a pipeline’s stages: the trainer keeps each run a pipeline repeats stacked by stage on it, each row placed as these rules place its layer (dew.training.stages.StageLocal.placement).

Below min_shard elements a parameter costs more in collectives than it saves in memory, so it stays replicated. tolerance is the fraction of shardable parameter elements a layout may leave replicated before check refuses it.

host names the fields of HOST_RESIDENT to keep in pinned host memory between steps. For opt_state and ema, the step fetches the field to the device, updates it as usual and writes it back, so the values are the same and only where they are stored changes. Naming variables instead moves the whole TrainState to the CPU, including the optimizer state, the EMA and gradient accumulation. The optimizer update then runs on a CPU companion of this mesh, and the parameter copies on the accelerators are read-only snapshots for running the step, never a second master copy. Every process needs at least as many CPU devices as accelerators, set before JAX initializes; neither this layout nor the trainer changes the count, and a process’s surplus CPUs, and the processes the mesh leaves out, stay out of the companion (companion_mesh).

host_parameters lists the variables an inference placement keeps in pinned host memory, as globs over their logical paths (params/layers_*). A selected leaf keeps the spec the rules give it, and only its memory kind changes. Only offloaded reads the patterns, because only the layer stack fetches each layer’s parameters as it reaches that layer. When host_parameters is set, check refuses a placement that keeps every parameter on the device, so the patterns are never silently ignored.

frozen_whole keeps the frozen collection (FROZEN, a LoRA run’s base) whole over fsdp and sequence (STATE_AXES), split only over the other axes the rules give it (a tensor split stays). A frozen leaf has no gradient and no optimizer state, so a split over them saves only its memory, and the step gathers it whole forward and again backward. check counts no leaf it keeps whole, and a leaf host_parameters also names is kept so in pinned host memory.

stage_local: bool

Whether a pipelined run’s layers, with their optimizer state and EMA, are kept stacked by stage on their own stage’s devices (dew.training.stages), about 1/S of the run on each; False keeps every layer whole on every stage, which an optax transformation that may read a stacked leaf whole needs. Either trains the same arithmetic, bit for bit.

data_axes: tuple[str, ...]

The axes a batch’s rows split over, ahead of the rules’ own activation_batch entries: a native mesh’s names for its data-parallel axes, which the run’s data shares, its batch placement and its row counts read (dew.mesh.batch_axes). Empty leaves the rules as they are.

axis_rules: LogicalAxisRules

The rules as the (name, axes) pairs flax reads, in precedence order, data_axes first.

def shardings(mesh: Mesh, tree: TreeT) -> Placement[TreeT]

Return a NamedSharding for each leaf of tree, from the axes its module declared.

A parameter its module boxed (nn.with_logical_partitioning) takes the axes it carries, in tree or in the trainer’s boxed table, when the rules place one of them (ruled_boxes). A leaf whose path no module declares falls back to a shape heuristic, which splits its largest dimension that fsdp divides evenly over fsdp. So the axes of one model family can be declared at a time. Either way, what lands on fsdp splits over the sequence axis too where the two divide it (state_spec). A leaf below min_shard elements is replicated either way.

Flax metadata that a caller’s own module attached is removed, because the state the trainer creates from these shardings holds plain arrays. Raises ValueError when the rules place a leaf on the data or stage axis (UNSPLIT_AXES) or on an Explicit axis, or name an axis the mesh lacks.

def offloaded(mesh: Mesh, tree: TreeT) -> Placement[TreeT]

Return shardings, with the memory kind each leaf’s path asks for.

Leaves that host_parameters selects go to pinned host memory. Each leaf keeps the spec the rules give it, so a selected parameter is the same shard in another memory space, and its layer issues the same collectives as when the parameter stays on the device.

Every pattern must match at least one variable, or this raises ValueError. A pattern that matches nothing is a typo or a stale path. Without the error the run would keep those weights on the device, and the only sign would be the memory it did not save. So each pattern is checked on its own.

def check(params: Variables, shardings: Placement[Variables], mesh: Mesh) -> None

Refuse a placement that replicates too much of the model or ignores host_parameters.

This is MaxText’s guardrail (base.yml sharding_tolerance) against a mesh whose parameter axes divide none of the model’s dimensions. On such a mesh the shape heuristic replicates every parameter, and nothing else reports it.

MaxText measures the excess per-chip memory over perfect sharding across every parameter. This check takes the same ratio over the parameters of at least min_shard elements whose rules name a parameter axis this mesh splits. Smaller ones are replicated on purpose, so counting them would fail models that are merely small; one no rule splits over this mesh’s axes is replicated by the mesh’s choice, as a data axis replicates every parameter, so an expert-only mesh keeps a mixture’s dense weights whole as data parallelism does. The sequence axis splits only where it divides what the rules place on fsdp (state_spec), so a leaf it cannot split is the rules’ to count, and a mesh of sequence alone refuses none.

Raises LayoutRefused when the replicated fraction is above tolerance, naming the five largest replicated parameters, and ValueError when host_parameters is set but every parameter in shardings stays on the device. A mesh whose parameter axes all have size 1 passes the replication check.

def validate(mesh: Mesh, state, *, capacity: int | None = None) -> Validation

Report what state holds on mesh before the first step, and refuse a state a device cannot hold.

MaxText prints each parameter’s logical and physical sharding before it trains (print_shardings_params), and Megatron reports the weight and optimizer bytes a GPU holds from a formula over the model’s config (report_theoretical_memory). Here both are read off the placement itself, so they hold for any rules, mesh and optimizer.

state is placed, or is about to be: each leaf an array, or a jax.ShapeDtypeStruct carrying the NamedSharding it is placed with (shardings, Trainer.shardings), so nothing is allocated. The report (Validation) gives each leaf’s split and the part each device holds, the bytes each device of mesh keeps between steps, the axes a batch’s rows split over, the collectives a step runs over each axis (Validation.collectives), and the logical axes the variables declare that no rule names (Validation.unbound). A leaf in host memory (host), or on devices outside mesh (a CPU-owned state’s companion), takes none of a device’s memory. Rules naming an axis mesh lacks raise ValueError here, as shardings refuses them.

capacity is the bytes each device of mesh has. Unset, each of this process’s devices gives its allocator’s limit (Device.memory_stats’ bytes_limit) where JAX reports one, which is what an allocation fails against, under whatever memory fraction the process set. A CPU device reports none, so a CPU mesh standing in for accelerators takes theirs from the caller.

Raises LayoutRefused, naming the device and what it holds by field, when a device would hold more than its capacity between steps: the state cannot be placed there, before any of the step’s temporaries, which the plan checks against the compiled step (dew.training.planning).

dataclass source

class LeafPlacement(spec: P, shard_shape: tuple[int, ...], memory_kind: str)

Where one leaf of a placed state lives (Validation.leaves).

spec: P

The mesh axes each of its dimensions splits over.

shard_shape: tuple[int, ...]

The part of it each device holds.

memory_kind: str

The memory that part sits in: device, or the host memory a layout keeps it in (pinned_host), which takes none of the device’s.

dataclass source

class MeshSpec(
fsdp: int = 1,
expert: int = 1,
tensor: int = 1,
sequence: int = 1,
stage: int = 1,
replicas: int = 1,
diloco: int = 1,
explicit: tuple[str, ...] = (),
)

Device counts for the mesh’s sharding axes; the data axis takes the devices left over.

expert: int

Devices the expert dimension of an MoE layer is split over.

tensor: int

Devices the mlp, head and vocabulary widths are split over, in addition to fsdp. 1 keeps every width on fsdp alone.

sequence: int

Devices the batch’s sequence dimension is split over; 1 keeps whole sequences. The persistent state splits over them with fsdp (STATE_AXES).

stage: int

Pipeline stages the layer stack is split into, each on its own devices. 1 runs the whole stack on every device. How many microbatches a step feeds through them is the trainer’s (Trainer.microbatches).

replicas: int

Groups of hosts the data axis spans, for hybrid sharded data parallelism. Every other axis, fsdp included, stays inside one group. So the parameter gathers and gradient reduce-scatters run over the fast links, and only the gradient all-reduce between replicas crosses the slow network. A group is a whole number of granules. A granule is the devices that share a slice_index (a TPU slice, and on multi-host GPU a host or an NVLink domain), or one process when every device reports the same slice. At 1, jax.make_mesh places the devices of a single slice; devices on several slices still take the hybrid layout, with each slice as one replica when the slice count divides the data axis.

diloco: int

DiLoCo’s replica groups (dew.training.diloco), the mesh’s outermost axis: each trains its own inner copy of the model on its own share of every batch’s rows over the devices of the rest of the mesh, and the groups meet only when the outer model averages them, every few updates. Across several slices, the groups take whole slices where they divide them, so only that average crosses the slow network. 1 is no DiLoCo.

explicit: tuple[str, ...]

The axes whose sharding an array’s type carries (JAX’s Explicit mode), where every other axis is left to the partitioner (Auto). An Explicit axis is placed where an array is made and follows the operations from there; a model’s sharding constraints name only the Auto ones.

def build(devices: Sequence | None = None) -> Mesh

Build the seven-axis device mesh this spec describes.

Parameters shard over ‘fsdp’, ‘expert’ and ‘tensor’. A batch’s rows shard over ‘data’, ‘expert’ and ‘fsdp’, its sequence dimension over ‘sequence’, and the layer stack over ‘stage’.

Expert parallelism splits an MoE layer’s expert dimension, which no dense model has. So that dimension gets its own axis, and ‘fsdp’ is left for the model’s widths. The tensor axis splits the mlp, head and vocabulary widths alongside fsdp, as DEFAULT_RULES places them. The sequence axis splits long sequences, and the stage axis splits a decoder’s layers into pipeline stages, each stage on its own devices.

An axis of size 1 splits nothing, and with every size at 1 the mesh is plain data parallelism. So the same code path serves every topology without a flag. The axes are Auto, so GSPMD infers the collectives, but for those explicit names.

devices lists the devices of one slice in the order to use them, any devices a mesh holds (a compile-only GPU’s among them). The mesh takes them row-major over MESH_AXES, so neighbouring entries land on the last axes. When it is unset, the mesh uses every device, in the order jax.make_mesh gives the platform’s topology.

When replicas is above 1, or the devices are on several slices, the mesh is built the way MaxText builds a multislice one, through mesh_utils.create_hybrid_device_mesh. The data axis takes replicas groups of granules as its outer factor, and each group lays out the rest of the mesh over its own devices. A group of more than one granule splits fsdp across them, because fsdp’s traffic (a gather and a reduce-scatter per layer) is the only kind that tolerates the slower link.

Raises LayoutRefused when the expert, fsdp, tensor, sequence and stage sizes multiply to a number that does not divide the device count, or when replicas does not divide both the granule count and the data axis.

class source

class RefinesLayout(Protocol)

A model that places its own tree’s leaves: a tier-3 model’s torch-named tree, which no module declares axes for, placed by its tensor-parallel plan (dew.interop.torchax_fallback.TorchLayout). A run’s objective (Objective.placement) and a source’s load hand it the run’s layout, and it keeps every choice the layout makes.

def placement(layout: Layout) -> Layout

dataclass source

class Validation(
leaves: dict[str, LeafPlacement],
device_bytes: dict[int, int],
capacity: dict[int, int],
batch_axes: tuple[str, ...],
collectives: dict[str, tuple[str, ...]],
unbound: dict[str, tuple[str, ...]],
)

What a placed state holds on a mesh before the first step (Layout.validate).

leaves: dict[str, LeafPlacement]

Each leaf’s placement by its path (variables/params/...), in the state’s order.

device_bytes: dict[int, int]

The bytes each device of the mesh holds of the state in its own memory between steps, by the device’s id.

capacity: dict[int, int]

The bytes each device has, by its id, as the caller stated them or the device reports them; a device that reports none, such as a CPU, is absent.

batch_axes: tuple[str, ...]

The mesh axes a batch’s rows split over (batch_axes).

collectives: dict[str, tuple[str, ...]]

The collectives a training step runs over each axis of the mesh above size one, by the role the placement gives the axis, in HLO’s names (_implied_collectives).

unbound: dict[str, tuple[str, ...]]

Each logical axis the variables declare that no rule names, with the leaves that declare it. A leaf keeps the dimensions it names whole on every device where a rule places another of its names, and takes the shape heuristic where none does (shardings).