Dew documentation
Dew is a library for training neural networks with JAX and Flax. A training run is a Flax model, an Objective that defines the variables and the loss, a Dataset that yields batches, and a Trainer that runs the optimization on one device or a mesh of devices. Dew ships objectives for language models, image and video diffusion, diffusion language models, JEPA encoders and post-training (SFT, DPO, GRPO, PPO).
import itertools
import jaximport jax.numpy as jnpimport numpy as npimport optaxfrom flax import linen as nn
from dew import Trainerfrom dew.data.dataset import Loaderfrom dew.objectives.base import Aux, Objective
class Regression(Objective): def __init__(self, model): self.model = model
def init(self, key, variables=None): return self.model.init(key, np.zeros((1, 1), np.float32))
def loss(self, variables, batch, step): errors = (self.model.apply(variables, batch["x"]) - batch["y"]) ** 2 return self.row_mean(errors, batch), Aux(metrics={})
x = np.linspace(-1, 1, 32, dtype=np.float32).reshape(32, 1)batch = {"x": x, "y": 2 * x + 1}data = Loader(train=lambda partition: itertools.repeat(batch), val=None, records=32, batch=32)trainer = Trainer(Regression(nn.Dense(1)), optax.sgd(0.1), key=jax.random.key(0))state = trainer.fit(data, steps=100, log_every=50)The pages assume you know Python, NumPy and the basics of training with gradients. Pages that also need Flax or sharding say so at the top.
Getting started
Section titled “Getting started”- Installation: install Dew and check which devices JAX sees.
- Quickstart: the example above, one part at a time.
- Key concepts: the model, the objective, the dataset and the trainer.
- Tutorials: notebooks that train diffusion models, a language model and a JEPA encoder on real data.
Topics
Section titled “Topics”| Topic | Page |
|---|---|
| Batches, dataset specifications, TFDS and Hugging Face data, resuming the data stream | Training data |
Writing an Objective | Custom objectives |
| Validation metrics, previews and experiment trackers | Evaluation and tracking |
| Saving, resuming and exporting | Checkpoints |
| Diffusion processes, presets, solvers and guidance | Diffusion processes and solvers |
| Training an image diffusion model | Diffusion training |
| Language models: training, loading published checkpoints | Language models |
| Diffusion language models: masked diffusion, DiffusionGemma | Diffusion language models |
| Mixture-of-experts layers | Mixture of experts |
| SFT, DPO, GRPO and PPO | Post-training |
| Text generation and serving | Generation and serving |
| JEPA encoders | JEPA |
| Meshes, sharding rules and several hosts | Distributed training, Multiple hosts, Cloud TPUs |
| Checkpoints Dew can load | Supported models |
| Every public class and function | API reference |
Status
Section titled “Status”Dew is research software before version 1.0, so the API and checkpoint formats can change between versions. I have run it on CPU, on pools of local processes, on single GPUs, on one host with four GPUs and on one TPU v6e chip, but not on two physical nodes. Multiple hosts lists what has and has not been run. The roadmap lists work still to complete or qualify.
Papers and attribution lists the papers and upstream code behind the models and methods. How Dew works follows one training run from a model to kernels on the GPU, and holds the shared contracts. The binding training, data, post-training, testing, algebraic optimization and kernel templates specs stay in the repository under docs/design. Research notes stay under docs/research.