Skip to content

Scale one training run across many devices

When I trained text-to-image models from scratch with FlaxDiff, Dew’s predecessor, it was on 128 TPU v4 chips. Once a model or its batch outgrows one accelerator, there is no way around spreading the work over several. In Dew that is one argument to the Trainer: a MeshSpec that arranges the devices into named axes. The training code itself does not change. In this notebook, we train the language model from notebook 05 on eight devices, in two layouts:

We look at where one weight lives in each layout, and then restore a checkpoint written in one layout into the other.

Most readers don’t have eight accelerators at hand, so this notebook runs on eight simulated devices on the CPU. XLA can split one CPU into several devices with a flag, and everything Dew does with them is the same as on eight GPUs or TPU chips. The flag has to be set before JAX starts, which is why it comes first. The model is tiny, and the whole notebook runs in under a minute on a desktop CPU.

Just Dew from GitHub this time, no extras, since everything here runs on the CPU.

%pip install -q "dewml @ git+https://github.com/AshishKumar4/dew" --extra-index-url https://pypi.dewml.dev/simple

--xla_force_host_platform_device_count=8 gives the CPU backend eight devices, and JAX_PLATFORMS=cpu makes JAX use that backend even on a machine with a GPU.

import os
os.environ["XLA_FLAGS"] = "--xla_force_host_platform_device_count=8"
os.environ["JAX_PLATFORMS"] = "cpu"
import jax
print(jax.devices())
[CpuDevice(id=0), CpuDevice(id=1), CpuDevice(id=2), CpuDevice(id=3), CpuDevice(id=4), CpuDevice(id=5), CpuDevice(id=6), CpuDevice(id=7)]

The batch of 16 has to split evenly over the eight devices.

STEPS = 20
BATCH_SIZE = 16
SEQUENCE_LENGTH = 32
EMB_FEATURES = 64
NUM_LAYERS = 2
NUM_HEADS = 4
DATA_DIR = "data/07-tokens"
RUN_DIR = "runs/07-scaling"

MeshSpec(...).build() arranges the devices into a mesh with six named axes: data, expert, fsdp, tensor, sequence and stage. Axes we don’t set have size 1, and the data axis takes whatever devices are left. A batch splits over the data and fsdp axes together, so both meshes below split the batch eight ways. They differ only in what happens to the weights.

from dew import MeshSpec
mesh_dp = MeshSpec(fsdp=1).build()
mesh_fsdp = MeshSpec(fsdp=8).build()
print("data parallel:", dict(mesh_dp.shape))
print("fully sharded:", dict(mesh_fsdp.shape))
19:45:27 WARNING dew.telemetry.devices: Dew was imported after the JAX backend opened; its numerical policy cannot take effect. Restart with XLA_FLAGS=--xla_allow_excess_precision=false set before importing JAX.
data parallel: {'data': 8, 'expert': 1, 'fsdp': 1, 'tensor': 1, 'sequence': 1, 'stage': 1}
fully sharded: {'data': 1, 'expert': 1, 'fsdp': 8, 'tensor': 1, 'sequence': 1, 'stage': 1}

Layout decides how each weight maps onto the mesh, from the logical axis names the model’s layers declare. Weights smaller than min_shard elements stay whole on every device, because splitting them costs more in communication than it saves in memory. The default of 65,536 would keep every weight of this small model whole, so we lower it to 1 to see the effect.

Here is the token embedding table, 256 rows by 64 features, in each layout. In the data-parallel mesh every device holds all of it. In the sharded mesh each device holds 32 rows.

import jax.numpy as jnp
from dew import Layout
layout = Layout(min_shard=1)
table = jnp.ones((256, EMB_FEATURES))
for name, mesh in (("data parallel", mesh_dp), ("fully sharded", mesh_fsdp)):
sharding = layout.shardings(mesh, {"params": {"embed_tokens": {"embedding": table}}})
placed = jax.device_put(table, sharding["params"]["embed_tokens"]["embedding"])
print(name, placed.sharding.spec)
jax.debug.visualize_array_sharding(placed)
data parallel P()
                   
                   
                   
                   
                   
CPU 0,1,2,3,4,5,6,7
                   
                   
                   
                   
                   
fully sharded P('fsdp',)
  CPU 0  
         
  CPU 1  
         
  CPU 2  
         
  CPU 3  
         
  CPU 4  
         
  CPU 5  
         
  CPU 6  
         
  CPU 7  
         

The data is a small generated corpus of short sentences, written in the token layout from notebook 05. The model is a two-layer version of the notebook 05 decoder.

from pathlib import Path
import numpy as np
from dew.data import ByteTokenizer, Loading, presets
from dew.data.compile import compile as compile_data
from dew.data.sinks import export
from dew.data.sources import TokenFiles
rng = np.random.default_rng(0)
subjects = ["the cat", "a dog", "the bird", "my friend", "the child"]
verbs = ["sees", "likes", "finds", "wants", "hears"]
objects = ["the ball", "a tree", "the river", "some food", "the moon"]
text = "".join(f"{rng.choice(subjects)} {rng.choice(verbs)} {rng.choice(objects)}.\n" for _ in range(4000))
data_dir = Path(DATA_DIR)
ids = np.asarray(ByteTokenizer().encode(text), np.int32)
val_len = len(ids) // 50
metadata = {"token_store": True, "tokenizer": "byte", "vocab_size": 256,
"dtype": "int32", "eos_id": None}
export([{"text": ids[val_len:]}], str(data_dir / "train"), format="array_record",
metadata=metadata, length_field="text")
export([{"text": ids[:val_len]}], str(data_dir / "val"), format="array_record",
metadata=metadata, length_field="text")
train, val = TokenFiles(str(data_dir / "train")), TokenFiles(str(data_dir / "val"))
windows = {"seq_len": SEQUENCE_LENGTH, "batch": BATCH_SIZE}
data = compile_data(train.shuffle(seed=0).repeat().pipe(presets.token_windows, **windows),
validation=val.pipe(presets.token_windows, **windows, records=2 * BATCH_SIZE),
read="memory", loading=Loading(workers=0, threads=1, read_buffer=2))
print("training windows:", data.records)
training windows: 2987
from dew import Checkpoints, Trainer
from dew.config import OptimConfig
from dew.nn.backbones import CausalTransformer
from dew.objectives.lm import LMObjective
model = CausalTransformer(vocab_size=256, emb_features=EMB_FEATURES,
num_layers=NUM_LAYERS, num_heads=NUM_HEADS, max_seq_len=SEQUENCE_LENGTH,
dtype=jnp.float32, attention_impl="xla")
objective = LMObjective(model, SEQUENCE_LENGTH)

The two trainers below differ only in mesh and in the folder they write checkpoints to. The trainer builds the mesh, works out a sharding for every array in the training state, and compiles the step with those shardings. XLA inserts the communication between the devices for us. The state is created directly in its final layout, so a model too large for one device never has to fit on one.

After each run, we read the embedding table’s placement off the returned state. sharding.spec names the mesh axis each dimension is split over, and addressable_shards lists the piece each device holds.

def describe(state):
table = state.variables["params"]["embed_tokens"]["embedding"]
print("spec:", table.sharding.spec)
for shard in table.addressable_shards[:3]:
print(f" {shard.device}: rows {shard.index[0]}, local shape {shard.data.shape}")
replicated = Trainer(objective, OptimConfig(learning_rate=1e-3), key=jax.random.key(0),
mesh=MeshSpec(fsdp=1), layout=Layout(min_shard=1),
checkpoints=Checkpoints(f"{RUN_DIR}/replicated"))
replicated_state = replicated.fit(data, steps=STEPS, log_every=10)
describe(replicated_state)
Training CausalTransformer from step 0 to 20: 147,840 parameters, on 8 × cpu, mesh data 8, batch 16, float32
step 10/20  loss 3.312  ce 3.312  perplexity 27.44  token_accuracy 35.0%  step_time_ms 50.15  samples_per_sec 319.0  accepted 100.0%  0:00:04 left
step 20/20  loss 2.413  ce 2.413  perplexity 11.17  token_accuracy 47.5%  step_time_ms 10.09  samples_per_sec 1,586  accepted 100.0%
Trained 20 steps in 0:00:05: first step after 3.83 s, then 47.7 step/s
7.4% of the wall time in steps, final loss 2.413
spec: P()
  cpu:0: rows slice(None, None, None), local shape (256, 64)
  cpu:1: rows slice(None, None, None), local shape (256, 64)
  cpu:2: rows slice(None, None, None), local shape (256, 64)
sharded = Trainer(objective, OptimConfig(learning_rate=1e-3), key=jax.random.key(0),
mesh=MeshSpec(fsdp=8), layout=Layout(min_shard=1),
checkpoints=Checkpoints(f"{RUN_DIR}/sharded"))
sharded_state = sharded.fit(data, steps=STEPS, log_every=10)
describe(sharded_state)
Training CausalTransformer from step 0 to 20: 147,840 parameters, on 8 × cpu, mesh fsdp 8, 100% of the parameters' bytes split, batch 16, float32
step 10/20  loss 3.312  ce 3.312  perplexity 27.44  token_accuracy 35.0%  step_time_ms 52.71  samples_per_sec 303.5  accepted 100.0%  0:00:04 left
step 20/20  loss 2.413  ce 2.413  perplexity 11.17  token_accuracy 47.5%  step_time_ms 14.07  samples_per_sec 1,137  accepted 100.0%
Trained 20 steps in 0:00:04: first step after 3.87 s, then 38.2 step/s
11.2% of the wall time in steps, final loss 2.413
spec: P('fsdp',)
  cpu:0: rows slice(0, 32, None), local shape (32, 64)
  cpu:1: rows slice(32, 64, None), local shape (32, 64)
  cpu:2: rows slice(64, 96, None), local shape (32, 64)

Both runs start from the same key and read the same batches, so they compute the same thing and the losses match. Only the placement of the weights differs.

Dew can restore a checkpoint into a different layout from the one that wrote it. The trainer below uses the sharded layout, but points at the data-parallel run’s checkpoints. place() reads each array and lays it out the way this trainer’s mesh specifies. The restored weights equal the data-parallel run’s, and training continues from step 20.

crossed = Trainer(objective, OptimConfig(learning_rate=1e-3).build(STEPS), key=jax.random.key(0),
mesh=MeshSpec(fsdp=8), layout=Layout(min_shard=1),
checkpoints=Checkpoints(f"{RUN_DIR}/replicated"))
restored, _, _ = crossed.place()
describe(restored)
same = all(np.array_equal(np.asarray(a), np.asarray(b))
for a, b in zip(jax.tree_util.tree_leaves(restored.variables),
jax.tree_util.tree_leaves(replicated_state.variables)))
print("restored weights equal the data-parallel run's:", same)
continued = crossed.fit(data, steps=STEPS + 10, log_every=10)
print("continued to step", int(continued.step))
Resumed from step 20 in /content/tutorials/runs/07-scaling/replicated
spec: P('fsdp',)
  cpu:0: rows slice(0, 32, None), local shape (32, 64)
  cpu:1: rows slice(32, 64, None), local shape (32, 64)
  cpu:2: rows slice(64, 96, None), local shape (32, 64)
restored weights equal the data-parallel run's: True
Resumed from step 20 in /content/tutorials/runs/07-scaling/replicated
Training CausalTransformer from step 20 to 30: 147,840 parameters, on 8 × cpu, mesh fsdp 8, 100% of the parameters' bytes split, batch 16, float32
step 30/30  loss 1.751  ce 1.751  perplexity 5.762  token_accuracy 57.6%  step_time_ms 53.10  samples_per_sec 301.3  accepted 100.0%
Trained 10 steps in 0:00:03: first step after 2.65 s, then 21.9 step/s
13.2% of the wall time in steps, final loss 1.751
continued to step 30

Everything above ran in one process. On a TPU pod slice, every host runs the same script, and each one first joins the group:

from dew.training.runtime import prepare_process
prepare_process(multi_host=True)

prepare_process calls jax.distributed.initialize(), which finds the coordinator from the environment the TPU pod provides. The token loaders give each host its own share of the records, and the checkpoint folder has to be one every host can write to, such as a gs:// bucket.

The dew tpu command creates a slice, installs Dew on every worker and starts a recipe on all of them; the TPU guide describes it. None of that ran in this notebook.

Layout.min_shard decides which weights are worth splitting; in a decoder with a large vocabulary, the embedding table is usually the first. Trainer(accumulation=k) adds up gradients over k smaller batches when a full batch doesn’t fit in memory. The distributed training guide covers the expert, tensor, sequence and stage axes.